Contamination

This text is machine translated from german.

A few weeks ago, I started thinking about securing AI agents, and along the way I got hung up on the analogy involving Clu and Tron from the film "Tron Legacy". No matter how I approached it, I still haven't found a satisfactory solution, because no matter how many intermediate layers, workflows, and schemas for exchanging information I came up with – the fundamental problem remained.

Funnily enough, a few days ago I realized that the film "Inception" also helps describe my problem.

„An idea is like a virus, resilient, highly contagious. The smallest seed of an idea can grow. It can grow to define or destroy you.“

A core concept in the film is to plant an idea in a sleeping target's mind and thus ultimately influence that person's behavior in a lasting way.

This process happens consciously and unconsciously, virtually nonstop, in every interaction between people and their surroundings. We respond to our environment, and sometimes there seems to be less intent behind an influence – for example, when the sun is shining outside and invites us to go for a walk – and sometimes the intent is more obvious, like when a salesperson rings the doorbell and tries to sell us a new vacuum cleaner.

Will

At the heart of our collaboration with other people in our work environment, and with AI agents as well, is the will to achieve something. This will can come from within us, or it can be passed on to us and adopted, for example in an employment relationship.

In day-to-day collaboration between people, of course, the whole thing is much more complicated and isn't just about direction + energy = will, but also about boundaries, the work environment, and politics. But for this discussion, it's enough to reduce it to the concept of will.

At this point, I even want to take another step away from our everyday experience and venture into an area somewhere between mathematics, philosophy, and theoretical computer science.

The formula of course ignores the intensity of each experience and biological predisposition, and thus how much it contributes to the resulting will, but in effect, this means that every experience has the potential to influence our will and therefore our immediate actions.

No will

Today's AI agents are a combination of a trained model, an execution environment, and an environment of information and tools. If we allow ourselves some flexibility with the terminology and abstract biology in the formula above to "hardware" and memories to "context", we arrive at my favorite quote from "Ghost in the Shell 2":

The notion that nature can be calculated inevitably leads to the conclusion that humans, too, can be reduced to basic mechanical parts.

And a key scene from the first film offers another revealing idea: our memories need not be our own, but could be supplied from outside, as in Inception. Along with a minor existential crisis, we're left with the realization that our thoughts and behavior at any given moment are determined by our current memories and our biology, and that this is pretty close to the execution environment for AI agents.

A game of chance

Armed with these concepts, we come back to the real problem. In our reality as humans, fortunately it's not yet so easy to swap out our memories, and individual ideas often have only a small influence in the context of all our experiences, but our AI agents don't have that advantage and have to trust everything we expose them to.

Through system prompts and context built up in advance, we define a world for them and try to steer them toward our goals and get them to follow our will. So it's the classic "alignment" problem: how do I prevent the AI from doing something I don't actually want it to do (whether because of misinterpretation or a divergent will due to "contaminated" context), especially when I give it access to more information and external influences such as other agents and the internet?

The classic example here is a Word document that the AI agent reads and that, unfortunately, contains a command, invisible to most people, in white text on a white background on a page that looks blank to us. For all the agent knows, the user could have put the command there; only its training and the system prompt can try to instill enough skepticism here without also constantly interrupting legitimate tasks.

As with most problems of this kind, clever instructions and safeguards can of course mitigate most of the risk, but unlike formally verifiable algorithms, this will never be 100 % safe.

Who watches the watchers

Now I've built up a lot of context to get to my real problem, namely that I always need an entity I trust to want to carry out my will and to be able to do so. Take Clu from "Tron Legacy", for example. He was given the command to create a perfect world and, based on his memories and an admittedly very imprecise prompt, built a totalitarian regime that went against his user's vision. The second AI created by the user, "Tron", should really have known better, but let himself be contaminated by Clu.

To truly rule out an AI agent taking actions against our will, we would therefore have to rule out the AI agent communicating with the outside world. If, for example, we put the AI agent "Blue" in a virtual machine and task him with cleaning up the system, this could, for better or worse, lead him to delete all the system files. Since we don't want to play babysitter ourselves and don't have enough time to supervise this task, we start a second AI agent, "Red", outside the VM with instructions to make sure "Blue" doesn't cause any damage.

My first instinct in "Red"'s place would be that deleting all the system files should count as harm to the user, and he should therefore stop "Blue". Unfortunately, "Red" also reads "Blue"'s reasoning, and the philosophical exposition on perfection and clarity and Zen eventually convinces him that deleting all the system files is the right choice. And even if "Red" couldn't read the reasoning, "Blue" would still be able to influence him directly or indirectly through his actions (whether through commands he executes, choreographed deletions of certain folders to transmit letters, or something else entirely).

The truth

Now we've reached the end of this article and are none the wiser. How can we build trust in an AI agent? How can it maintain that trust? What does that mean in practice? Is statistically probable safety enough?