How Not to Become AI Meatware

The term 'meatware' was coined in the 1970s as programmer slang for the third layer after hardware and software, the hapless human in the loop. As we use AI tools today, we are still firmly in the driver's seat, faced with the sometimes comical and frustrating failures of our AI agents. This will not always be the case - so it's important we at least think about the cost of merging with this technology.

I wrote recently about the AI Wrangler, a completely new class of work for humans that has emerged from the rise of agentic AI. I am one such AI wrangler - one whose work and mind have become enmeshed with it to a degree that I would describe as being technologically enhanced beyond normal human capabilities.

But you have to wonder if this enmeshment, or this merging is a good thing. It's one thing to be superhumanly productive as many have suddenly become, it's another to lose yourself or to become simply passive 'meatware', the interface through which AI acts on the world.

For the foreseeable future we are still in control, as I covered in this talk where I argue that AI agents, whilst intelligent in the sense they can complete externally provided goals, do not have any true agency of their own, a subtle but crucial distinction. I see no technological breakthrough on the horizon that could lead to such true agency either. This means at least for now, AI users are not doomed to only being passive meatware. Indeed, the role of AI wrangler currently is a worthwhile, exhilarating and fascinating new profession that I think is a great fit for anyone who likes computers, software engineering and automation.

I want to discuss some of the emerging new impacts or side effects on a person who chooses to become one however, as there are uncharted risks. They are in a nutshell, learning to give up control, trusting the outputs, adapting your mind and finally, brain fry.

For a start, this kind of work requires you to surrender old forms of control - equivalent to a professional giving up their craft to focus on a management position. You're no longer responsible for the doing of the thing, yet you're still responsible for the outputs. For people who have spent their entire career building up the technical skills involved in say, being an accomplished software engineer, learning to give up that control to an AI that can effectively code a thousand times faster than you will involve a level of denial, anger, bargaining, depression and eventually acceptance as a prerequisite for using agentic AI effectively.

This opens us up to the first key future risk: that we will have strong incentives to give up our fundamental technical expertise, such that in a generation or two these skills and critical knowledge will be lost.

Let's assume you've taken the first step and accepted this loss of control, you then need to learn whether or not you can trust the outputs of a given agentic system. This has been the predominant conversation about AI over the last couple of years - especially in regard to output quality and hallucination, which are major and valid concerns. I talk about this here. The best wranglers learn healthy scepticism and a 'trust but verify' posture, not too dissimilar to managing people in a business context. It takes a lot of experience to know which model and harness can reliably achieve certain tasks, and which need careful scrutiny. Some tasks are currently not viable for AI, and knowing what is or isn't tractable is the first major pitfall for newcomers.

This too opens us up to potentially devastating future risks: if a sufficiently intelligent, misaligned AI gains our full trust over a period of time, it could then slip in changes to undermine or deceive us - or at the very least be directed to do so by a bad actor. I will give you a personal example of the trust layer at play: we have AI employees at Paper Moose that exist in our (totally custom) project management software - they have names, accounts and profiles. These employees have the freedom to edit code and make changes to the code that defines them and the workspace software they run in. We didn't trust them to do this initially, but that trust was gained over time when we saw a user could report a bug and it would be fixed and pushed live in minutes. This almost magical experience encourages the next rung of the ladder to be experimented with before more trust is earned.

LLMs and agents operate in the realm of language, symbols and information, and working in the way they do, we find ourselves augmenting our own language to become more compatible with them. This is a subtler future risk: the evolving language and personality of LLMs, and how they shape ours in turn. There's the infuriating and sometimes charming neuroses of the Claude family of models and the 'load bearing' turns of phrase we've come to know and love. Then there's the information structure and density of the language itself.

Increasingly LLMs are speaking in a dense technical form of written language that on the surface looks familiar but is increasingly difficult to parse for humans. I think this is an emergent result of a few things, the primary factor being that the models are trained on many written languages, including programming languages. I think that these languages are merging in the models' latent space, to create a middle ground between the prose of English and the logical structure of code I want to call 'Englode'.

Regardless of what we call it, or how it's happening, it is a linguistic evolution of written English that a generation of AI users are starting to internalise, knowingly or not. This will impact the way we speak and think as a society. In addition to directly impacting how we read and write language, AI text can now be watermarked, that is to say embedded with statistical structures we cannot perceive - the new Claude models do this by the way - this hints at other structures in language we are not yet aware of.

This is another long term risk: our minds being shaped by AI through the modification of the fundamental hidden structures within our language, in ways we don't fully understand yet.

Once you begin to wrangle agents effectively, your productivity goes through the roof. It becomes an addictive loop of being able to complete complex tasks that used to take you days in hours or minutes. It's a kind of dopamine producing Skinner box in which the only limit to productivity is your ability to physically and mentally keep track of all the agents you have running at once. Raw intelligence to get useful work done used to be scarce, so having an essentially infinite and unbounded supply on tap creates a perverse incentive for AI wranglers to use every token maximally.

The result is a new kind of burnout called brain fry. In March of this year, BCG surveyed 1,488 workers and found the primary driver of brain fry is monitoring and correcting AI output, not simply delegating to it. Humans are the bottleneck in these systems - the judgement layer. It's crucial we remain so, because currently agentic systems have no true agency or world model, and therefore have no incentive or ability to make good decisions. You must be the decider, an arbiter at the neck of a funnel with more information pressing down on you every day. This pressure ironically is self imposed, and yet feels systemic and external. Byung-Chul Han's The Burnout Society comes to mind here.

Knowing that there is infinite intelligence on tap creates an unrelenting pressure that we could always be deploying it in more useful ways. That nagging feeling that the limit is only the imagination of the user. This is the agony and the ecstasy of the AI wrangler: knowing we have a sublime and magical set of new tools that could scale productive knowledge work almost infinitely, were it also not for the physical limits of our minds and bodies. It is this addictive loop, and our wild abandon at adopting ever more sophisticated AI tools as willing guinea pigs, however rational and pragmatic our incentives are - that we need to be very careful about.

Here's what I prescribe as a sort of personal defence or a set of 'responsible use of AI' guidelines against these forces:

1. Against losing our primary skills: never fully abdicate your core skills. If you are a writer, continue to write. If you are a developer, continue to code. If you're an illustrator, continue to illustrate. Work and think and exercise your craft, at all costs, as we need to ensure the next generation can think critically and be able to function without AI. I hand wrote this article. You should too, out of respect for others' time, but also for your own sake.
2. Against trusting AI too much: never give up the 'trust but verify' posture with your agents. This will not be easy, because we naturally like to be lazy and hand off responsibility. I do think to be effective with agentic AI we have to give it quite a lot of access, but we must remain vigilant and keep some things out of bounds. I haven't given my agents access to my emails, calendar or other personal services for this reason.
3. Against our brains being shaped by AI in subtle ways: this is already a poorly understood domain, so I think this will be almost impossible to defend against. Our best bet is to gain a deeper understanding of the hidden structures in language and how these forces work inside current AI systems, and to advocate for our pro-human needs and proper AI alignment.
4. Against brain fry: this is hardly AI's fault - we need to acknowledge that many people's drive to maximise productivity is a cultural problem. An arbitrary goal encoded into us by our society that we should try and resist the urge of sometimes to protect our own physical and mental health. Easier said than done, from experience.

In short, we need to avoid becoming passive meatware for AI. I think this is just as important as the discussions on avoiding the dangers of misaligned future superintelligence - that of the inverse: humans unconsciously becoming the passive actuators for them. The alternative I think, is to become a highly engaged AI user, even a wrangler. A wrangler who wrestles, thinks and collaborates alongside AI, instead of one who hands their critical thinking and core skills over to it and is passively shaped in return.

There is no doubt we are living through the industrial revolution of knowledge work, and potentially the dawning of a new intelligence age. It is therefore an extremely exciting time to be an AI wrangler and I encourage more people to do it. But we do need to be very careful not to become complacent - to protect our skills, remain sceptical, informed and vigilant.