Large Language Models are Terrible at Programming

Large language models (llms) were originally created by programmers and computer science or computer science-adjacent researchers. It is therefore not strange that one of the first use-cases that they were employed in, and in which they have been given an overrepresented amount of attention, is the automation of writing code — the text that describes a program. And yet, despite rapid increases in llms’ capabilities when it comes to writing code, the actual quality of the code they produce is still very low. Why is this?

The debate about the usefulness of large language models, and agents in particular, is in practice over. The field of software development has changed radically in the four years since the release of gpt 3.5, and is almost unrecognizable. There are still hold outs, and the degree to which agents are used differs drastically from person to person, not to mention organisation to organisation. But it is clear to anyone who has tried any sort of development with llms that, while produced quickly, the structure of the programs created are horrible and incredibly difficult to continue developing in.

This is surprisingly not all too different from the early days of llm programming, when one would simply copy-paste snippets of text back and fourth between an editor or terminal and a browser connected to Chatgpt. The only difference is that this process today is much faster. The biggest contribution of agentic harnesses¹ is the ability to ingest more and more context into the model much faster, and give the model the ability to recursively interact with the state of the program and code (through running automated tests and responding to their results).²

But still, while there have been large increases in llms’ abilities to reason (beginning with o1-preview) and general cognitive capabilities (most notable with Mythos-class models) they have still not shown any increases in creating “good” programs — instead they have become good at creating functional³ ones. It is this functional element that is key to understanding what makes a good programmer.

In the now-almost half a decade history of llm’s I have never seen one write a beautiful function or solution — those of the kind where you are awed at how elegantly to solve the competing requirements of a program. Instead all I ever see is huge, bloated functions that seem to do less with more.

The largest challenge for most programs, especially those created collaboratively at large organisations, is and has always been (at least since the invention of programming languages) understanding the code that you have written. Code that you do not understand is known as tech debt, because it becomes increasingly hard to make new changes without introducing unforeseen bugs as the number of potential interactions increases.

A lot of work has been put into ways to reduce the onset of tech debt, and many techniques and practises have been created and proposed over the years. What they all generally agree upon is the importance of creating simple and elegant abstractions.⁴ The digital realm is the realm of the idea — much closer to a form of applied mathematics or magic than engineering or anything having to do with the physical world. As such, abstractions are its most powerful tool that allow you to shape the structure of this metaphysical world.

The problem then arises with the objectives of the human programmer⁵ and the llm agent. The programmer is not actually (or should not be) very interested in implementing features; they instead spend most of their time making changes that facilitate making other changes. They care about the creation and maintenance of the program qua a dynamic entity that achieves certain goals. They are not necessarily very interested in achieving those goals themselves. The programmer is in the business of creating abstraction that, in aggregate, produce a result. What exactly this result is is not fixed — this dynamic element is what allows changes to the program to be made as the requirements change.

Problems of tech debt of course manifest in any large, long running software project. But when one uses an llm, a similar form of tech debt manifests itself immediately, partly due to the fact that there is no one around who has actually written the code, and partly from the code being written in what amounts to an obfuscated way.

This is because agents and llms approach programming in a completely different way. They are brought into existence from the void, their existence defined through the terms of an objective. Each prompt literally prompts them into motion, and they then fall into unconsciousness until the next one. This leaves them incapable from approaching the program through the terms that the programmer would, with their own existence ephemeral compared to the lifetime of the program they write.

They therefore work this this aforementioned functional way; where they are mostly interested in the produced end-result and output of the program. The long-term maintainability is of no concern to creating the correct output at a single moment of time. llms will therefore naturally hard-code values and create extremely large functions that would ideally be split apart into a larger number of smaller ones. That is because llms are not prompted to “restructure the value of X”; they will just include the code to do so as part of the overarching goal, especially if the value is only needed once. But even a simple program benefits from the readability gained from creating abstractions.

Further more, the skilled programmer will slowly accumulate a collection of tricks and clever solutions that they can then reuse in other programs. My own preferred programming language, lisp, is particularly plagued by programmers creating their own domain specific languages (dsls) for working in their specific environment, but this is true to some extent in all programming languages. This is not the case for agents, as they (i) commonly do not see the point in creating such abstractions and (ii) oftentimes lack the ability to store them (iii), and then have trouble seeing the subtle patterns that motivate the restructuring of code towards a more general form.

To paint a picture in your mind, let us compare this to a famous example from James Scott;⁶ that of scientific forests. These were attempts at increasing the output of timber by clearing areas of irrelevant plants and planting trees in orderly, easily inspectable rows. At first there was an increase in output, but then production cratered. The chaotic elements of nature turned out to actually be quite useful for maintaining a resilient and productive ecosystem. llms are in this regard similar to scientific forestry — they are able to quickly create results, but these results do not lead anywhere in the long run as agents disregard what at first seems to be unnecessary work in the quest for completing their task.

If agents are the industrial machine of the scientific forest, then the programmer could be seen as a sort of gardener who — obsessed with the minutiae in individuals plants’ growth, water supply, access to sunlight, and the condition of the soil — manages to create a flourishing garden flush with fruit and blooming flowers. This garden may take longer to tend to, and it may even produce less produce in the end. But there is no doubt that the garden is much more resilient and dynamic than the industrial forest.

The programmer is much more limited by our incapacity to remember many things at once. The human mind can only juggle roughly 7 things at a time, but a computer program almost always requires you to manage a much larger number of things. The solution for this is the creation of abstractions. But llms have a much easier time remembering raw information. In this regard llms have actually become worse at writing code as their abilities have increased, since the intricate context management that used to be required to get any usable results out of llms necessitated the creation of elegant abstractions.

This focus on the end-result rather than the process of programming is why llms have mostly resulted in an explosion of consumer-facing software, while the old joke of new daily Javascript frameworks has largely disappeared from the programmer zeitgeist. There is now less of a focus on creating a comfortable, and maintainable, developer experience for the creation of programs. We have stopped producing shears and instead only produce tractors.

This state of affairs is not fundamental to large language models as a technology; it is a deliberate choice made by the labs that train them. Positive training data is data where the models succeeds at some sort of task before being wiped — not where someone else can set their eyes on a project and understand its structure.⁷ They are thus incentivized to produce code; code that runs, and runs “correctly”, but not to engage in the largely subjective art of programming.

This explains a large part of the frustrations of trying to develop real software merely by using llm agents. While helpful in some tasks, they are in practice anathema to the workflow of developing a large software project — akin to using a bulldozer to trim the bushes. What is instead needed is a model that is trained to prioritize long-term values and the interaction between author (man or machine) and the program.