The shape of intelligence

4 September 2026

Cal Newport has a good description of the recent hacking incidents carried out by artificial intelligence agents:

I liken the deployment of these long-horizon LLM-powered agents to strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard. If that dog jumps the fence and ends up damaging cars on your street, you wouldn’t shake your head and lament about how the dog/whacker system had “gone rogue”; you would instead concede that dogs are unpredictable, so it was dumb to attach something dangerous to one.

Every artificial intelligence system is a combination of deterministic and nondeterministic components. In isolation, none of them is particularly dangerous, it is only when they are combined in ways that make them dangerous do we have a problem. In Newport’s example the nondeterministic system is the dog and the deterministic system is the weedwhacker.

In a deterministic system the same input always leads to the same output. This is how we understand machines to work, you press the on button and they do the same thing every time. When you press the lever on a toaster, it toasts the bread and after a set amount of time the bread pops out. You can twist the dial to vary the time but the toaster does not have any capacity to twist its own dial or press its own lever. It does exactly what you tell it to do. In a nondeterministic system, the same input does not lead to the same output. There is randomness. You can put the same question to a large language model and it will give you different responses each time.

We are at an interesting point in the development of nondeterministic components. There have been big improvements in LLMs since the beginning of last year, most notably in coding. I have observed them improving rapidly at discrete, well specified tasks – making individual parts – without comparable improvements in understanding how parts of a larger system fit together. They are particularly poor at abstracting from one example to another and even the most powerful frontier models need a lot of step by step guidance to perform the type of long range tasks that humans grasp instinctively. I am not saying LLMs cannot perform long range tasks at all, they can, they just do them badly when there is little supervision. The further they wander unmonitored, the greater the probability of a serious mistake.

Melanie Mitchell calls this ‘jagged intelligence’ and one specific problem she identifies in her article is artificial intelligence benchmarking being built around the measurable qualities of natural intelligence and, in doing so, implicitly dismissing those aspects which are not measurable. This leads to a situation in which people misinterpret LLMs improving at discrete, measurable tasks with progress towards general intelligence. In artificial intelligence one does not naturally flow from the other. It is not even true for natural intelligence, there are many examples of people with great expertise in a narrow field demonstrating very poor judgement in other areas. The problem is we conflate the two in people as well as machines and assume being very good at one thing means being good at everything.

That is how we end up in a situation where people design artificial intelligence systems which provide LLMs with wide discretion, reasoning that their ability to identify security holes in software somehow equates to good judgement about how to use this knowledge wisely. It does not. This seems obvious without running an experiment but we got one anyway. Nevertheless we can all learn from this mistake.

There are a couple of important principles. One is that ultimately people make decisions, not machines. A machine can write code but it is a human decision whether to run it. Another is that economic incentives should have no bearing on system design. An artificial intelligence system should use LLM tokens sparingly, only where necessary. LLMs should not be shoved into every corner of a system merely to recoup vast hardware investments which may or may not end up being justified.

The work we are doing on our own artificial intelligence system leads me to think that LLM usage should be limited and tightly bound, both by the deterministic parts of the system and ultimately by human judgement.

For whatever reason, the current trend seems to be the opposite – give the nondeterministic element of the system as wide discretion as possible, the sole purpose of the deterministic parts being to nudge agents along as needed and just let the magic happen. This approach reduces human input to pressing a switch and sitting back. As Newport puts it; “Adding powerful computer hacking tools to a harness, and then allowing it to run an LLM-powered Ask → Act → Report for days on end, with no attempt to monitor what it’s up to, is spectacularly negligent.”

The central problem with this approach is that it fails to interrogate the shape of these models’ intelligence, what they are good at and what they are bad at. As more powerful models excel at discrete tasks they are handed greater autonomy which exposes their weaknesses in general intelligence. Where we end up is a situation in which discrete tasks are performed with increasing capability but tethered to a system unable to make sound judgements about how to deploy these capabilities.

Even if artificial intelligence systems develop beyond LLMs and improve their judgement, it is debatable whether we want systems operating with such little human input or judgement which leads us back to the principle I outlined above – people should be making decisions, not machines.

There are two reasons for this, one is that exercising judgement strengthens it. If people start to rely on machines for making all judgements then their own judgement will atrophy. The second reason is responsibility. If a machine does something wrong we cannot throw our hands in the air, ascribe fault to the machine itself and say it made all the decisions. This is an abdication. Every action a machine takes can be traced to a specific person who set it in motion. Ultimately machines just follow orders, nobody should be allowed to shirk responsibility for their decision either to the machine itself or its designer.