The AGI trap

18 September 2026

Anyone who has used artificial intelligence on a regular basis will tell you roughly the same thing, its capabilities have taken a significant leap in the past year. In the hands of experienced software developers, LLMs can write good code and greatly reduce the amount of time to accomplish tasks. The scope of tasks they can address has increased.

In other respects, artificial intelligence has not really progressed even as the models get bigger and more resource intensive. Ironically, the recent improvements in discrete intelligence make their deficiencies in general intelligence more striking.

Gaps in the machine

An example is understanding design. One of the main goals in user interface design is familiarity, you want the user to spend as little time as possible learning how the product works. The ideal is that they glance at an unfamiliar interface and immediately grasp what to do. In this sense it is similar to writing, the designer should communicate their intentions for the product to the user with as little friction as possible.

This means establishing a visual idiom for certain actions then minimising the number of idioms the user has to learn to use the product effectively. When designing a new page for the web interface of an existing product, we start by looking at what is already there and trying to adapt it to the new function, rather than designing something completely new. Even when frontier models have access to all the code and design principles, they will invariably come up with designs that capture the superficial elements – those most easily quantified in the code – and lose the visual idiom. They do not understand the design in a deeper, conceptual way and will revert to an idiom that reflects the median output of their training data with some stylistic similarities to the existing aesthetic.

Working with artificial intelligence means understanding clearly the qualities it lacks, most notably general conceptual understanding, with users filling in the blanks from their own natural intelligence. As we use artificial intelligence, we accept what it cannot do and ensure it is focused on the things it can do well. These gaps in the machine may never be filled but this is a possibility some are unwilling to face. The scale of resources committed to achieving artificial general intelligence (AGI) demands a return that will not be satisfied by the type of artificial intelligence we have today – dependent on human thought and judgement to be effective. At these valuations, it is either a replacement or it is nothing. A useful tool is insufficient.

One of the problems in discussing AGI is that people define it in different ways. To me, general intelligence has two main qualities; the ability to learn from experience and form judgements over time. These qualities allow animals and people to rapidly adapt to novel situations, instinctively recognising what information is important and what can be ignored. A single incident is sufficient to learn, not troves of training material. I do not think consciousness is a requirement of general intelligence and I consider machine consciousness impossible.

Various prominent people seem to think AGI is imminent and the model scaling approach they are currently taking will lead us there. Some fear that once such a machine exists, it will design increasingly better versions of itself leading to an intelligence explosion and a superintelligence machine that dominates humanity. This scenario owes more to science fiction than reality but it makes for a good story so it gets endlessly repeated in spite of being a complete fantasy.

More prosaically, it is possible scaling does not produce exponential growth in capabilities and is exhibiting the characteristics of an S curve in which diminishing returns eventually set in. It is easy to mistake this type of progress for exponential growth in the early days but the slowdown is inevitable as this approach to artificial intelligence research reaches its limits.

Making a virtue of necessity

This brings us to recent accounts of a coordinated slowdown from the frontier artificial intelligence labs. What is striking is the speed with which others echoed the call from Dario Amodei to pace the frontier, saying they too would slow down and focus on safety. While this is welcome in light of the recent hacking incidents, I think the pressure to slow improvements in artificial intelligence capabilities has a different source than a sudden realisation that safety is important after all. More likely, the dawning realisation that the current approach has reached its limits and, impressive as the results undoubtedly have been, the promise of AGI in the next few years is very unlikely to be fulfilled.

Replicating the general qualities of natural intelligence is extremely difficult and the idea that it will be achieved by hammering away at language processing is for the birds. Any progress towards AGI will have to come from different approaches. Language is a relatively small part of natural intelligence and the easiest to replicate due to its well understood structure and copious training data capable of being reduced to numerical form.

Other characteristics of intelligence such as judgement, experience, memory are much harder even to understand, let alone measure and replicate with a machine. Yet to get to AGI, these types of problems will have to be solved. Natural intelligence is far more sophisticated than many are prepared to acknowledge. The idea that reaching AGI is imminent because of the progress made with language processing is hubris. It would not surprise me if true AGI ends up being a century away, let alone being achieved in the near future.

Predicting there is any chance humanity will be ended by such a machine within a decade is absolutely ridiculous. There are significant risks that need to be addressed to use artificial intelligence safely as models develop increasingly sophisticated discrete capabilities. Unfortunately, these important issues relating to autonomy and control of nondeterministic components within artificial intelligence systems are being neglected in favour of discussing a scenario with no chance of occurring.

The trap closes

Severely underestimating natural intelligence is at the core of the AGI trap. Progress is not linear. Anyone who has done research or read history understands this. Forgetting this lesson leads to the delusion that AGI is near and leads to the inflated promises (and fears) that accompany it.

When it becomes apparent that AGI could be decades or centuries away and ultimately may be impossible, the trap closes. Once it does, there are two ways out. One is to admit AGI is far away and try some new approaches while scaling down ambitions. The other is to claim progress is going so fast we have to slow down to preserve everyone’s safety.