I wanted to expand on my thoughts about using ChatGPT (with GPT-4 as the model) for coding, and what it taught me about the controversies regarding these AI tools and the quote-unquote “Truth.”

First, my most recent impression of using GPT-4 heavily to code (I am not a programmer) has left me feeling that, if you’re *not* using ChatGPT to help you code, then, in a way, you’re not really using it. Or that is, you’re not using it at least to its full advantage.

I say that as someone who used this and other AI tools to write 71 books (prior to GPT-4, which I used for two most recent volumes after that). I like to think that from that vantage point, I’ve seen quite a lot of what these tools are capable of, and the types of solutions they tend to provide. Even with all that experience under my belt, none of that prepared me for the “wow” factor I got from trying to use it as a coding partner to tackle technical problems I myself know little to nothing about (e.g., Javascript, API calls, etc.).

The thing is, when you use AI tools to write an email, or an article, or even a book, it’s certainly useful and consequential. But when you use it to code a tool customized to your needs, it suddenly accelerates the actualization of your goals in a whole new way that is entirely more concrete and tangible than just pushing ideas around “on paper.”

Coding an application let’s you do something – something “real” and (hopefully) repeatable. There felt like an entirely different level of gravity to the whole thing than just composing text that somebody (or nobody) might read once, and quickly forget about.

There’s entirely too much noise and debate about AI tools being “confidently wrong” and how that is terrible and the end of the world, because something something. But there’s a world of difference between inserting questionable “facts” into a blog post, versus the tool outputting code that simply doesn’t work in your browser.

Taking a step back, what is an application? An application is a tool you build to apply to a specific purpose. Yes, an email or article could have a specific purpose, and there might be ways to measure its success. But when you code an application, you have very visceral and very immediate feedback about: 1) does this work for the intended purpose, and 2) does this work in my browser?

Truth outside of programming a web application is ridiculously, stupidly complex and squishy. Actually, it’s that too in the realm of programming an app, but it’s different, because we have these twin benchmarks of: does it work in my browser & does it do what it’s intended to do? If we had such objective measures of truth in all realms of information, we might be in a very different place societally. But we don’t.

But we do have them in the browser. I can run an HTML page backed with some Javascript functionality, click around on the buttons, and see if the API calls are returning the things it should. And we have these backstops to tell us if this is a yay or nay situation. Does it throw an error? What does the error say?

Working with ChatGPT to code is a very eye-opening process, because it makes you keenly aware of how Truth is a collaborative process of co-construction. You don’t just have to input a query, and get a fully-fledged result. You have to interrogate the tool; you have to engage in a spirited conversation in search of a goal; you have to go back and forth. You have to take what it gives you, and then go test it against the proving ground of actual reality within the browser: does it work, does it do what it should?

ChatGPT is not omniscient (and not sentient). It hallucinates. It says you have a line of code in your code at times that is not there. Or it ignores sometimes aspects of the code that is there. Sometimes it updates your code to fix a minor error, and eliminates other major functionality you already had that was working. If you feed it back the errors thrown in the browser, it doesn’t magically know what you need to fix it. It tries first this, then that, then another thing, then another, and another.

To sit around on social media and grab screenshots of all the wrong things and blind avenues that ChatGPT gives you while you’re working on code would just be an enormous waste of time. It is endless. And yet people do exactly that endlessly on social media trying to win points in the complaint-brigade, or dunk on the developers because they don’t have any other specific thing they’re trying to achieve with the technology.

Yes, it’s important to point out the flaws of the tools, and think through the implications and possible methods of improvement. But if you really have something you want to achieve, the best spirit to approach the tech with is simply to route around the problems. The problems are stumbling blocks you need to move past, not things that are worth getting hung up and stopping all other progress, or using them as indicators of whatever. They might be, but there’s so much more at stake here, so much more that’s possible. It seems better to me to use this new-found power to explore that rather than getting stuck on the rest.

Lastly, it’s all too tempting in the “quest for truth” relative to the world of AI to simply be like, well why can’t we just have an API that tells us if z is true or not? I think that’s (maybe) fine where z is a fairly simple thing that can objectively be measured as true, but much of life and much of what we might want to have verified when we’re talking with AIs is not necessarily that simple that it can be broken down conclusively to an API call. And even where it is, I think it’s still important that we cling too and even enlarge our own agency around this notion of the co-construction of truth.

That is, rather than rely on some outside body to come in and give us a supposedly authoritative (based on what authority, anyway?) yes|no result, there’s a world where we would greatly benefit from having to actively piece together disparate assessments of claims – which may even conflict with one another – and come to our own conclusions about the results and what we will take away from it.

To recap:

  • When you use ChatGPT to help you code an app, you experience it differently than when you’re merely conversing with it, because we can objectively see if an app is working as intended in a browser: it gives us the results we want without errors.
  • In coding, there are always dozens or hundreds of paths open to us to reach the desired end result; what’s “true” here is what gets us to the end result, meeting our acceptance criteria.
  • ChatGPT will give you good code to start, but will lead you down many blind alleys with a lot of trial and error. While this can be annoying, it plainly shows you that “truth” within the realm of coding an app is a process of co-construction & active engagement.
  • We would do well to adapt the insights gained from the successes and failures of coding with ChatGPT to other types of non-coding outputs, such as texts, in that we should not seek to get final authoritative answers from it, but instead also actively engage in a co-constructive process. We can then measure our results in these non-coding domains as to whether they get us to the desired end result in a similar way.

Related writing & research