Optimizing into Chaos: Why AI Agents Need Guardrails
Lessons from Son of Anton
If you’ve watched HBO’s Silicon Valley, you probably remember Son of Anton, the AI that optimizes itself into chaos. It was a good Satire on The Deep Learning trend of the time. Son of Anton quickly goes rogue “deleting an entire codebase to eliminate bugs and ordering 4,000 pounds of beef to get the cheapest hamburger price.” It’s hilarious, but now that AI is so readily accessible via things like Grok3 and ChatCPT, the parody is eerily close to reality.
The AI world today is dominated by Transformers, complex predictive text models that we call LLMS, a shift from the pure neural networks that inspired Silicon Valley’s fictional AI. But the parallels remain: AI is smarter, more autonomous, and more unpredictable than ever. And just like in the show, as soon as we apply AI to things that we shouldn’t the risk of chaos increases.
AI Agents: When LLMs Start Taking Action
One of the most interesting (and potentially dangerous) developments in AI today is what we call AI Agents. An AI Agent is a program that makes software calls or real world actions based on input. Machines learning models have been doing this for a long time, but as we combine predictive text with the power to affect change, we move from a more binary or certainty driven approach common in modern software engineering to a probabilistic approach, and in any game of change it can always have an outcome you didn’t predict.
LLMs leverage reinforcement learning, which is a process of training a model to take an action by rewarding it for good behavior. Basic AI agents are built using reinforcement learning of an LLM, where models don’t just predict text but optimize towards achieving a goal. OpenAI’s AutoGPT and BabyAGI were early experiments in this. Large orgs will use techniques to leverage these RL techniques separate from the text input, but small orgs, unexperienced practitioners, or hobbyists like me, just leverage out of the box models hoping that it is good enough to act on a system.
There was a Super Bowl commercial where Matthew McConaughey refuses to use an AI agent while Woody Harrelson embraces it’s subtly hinting at the divide in how we perceive AI’s usefulness. But in Silicon Valley, Son of Anton was the ultimate AI agent, making “logical” but disastrous decisions in the pursuit of efficiency.
My point is, that although fun, ai is unpredictable and potentially dangerous in real world settings. Large companies, like OpenAI have thousands of employees that are investing time and money into building guardrails into their systems (Now is when you think, are you adding guard rails in your toy apps?).
RL, by definition uses hundreds or thousands cycles of training to add these guardrails and to stop rogue behavior, and if you are not leveraging these number of cycles to create guard rails, I don’t think you are being responsible with your agents and you probably shouldn’t “ship them” to production.
Chatbots, Search, and the AI-First Future
Right now, LLMs are replacing search engines and chat is becoming the new interface for everything. Google, OpenAI, and startups are pushing hard for chat-based search, AI customer support, and even AI coding assistants.
I see this firsthand on my Twitch streams, where I’ve built an AI chatbot that sometimes feels just as unhinged as Son of Anton. Some of the comments I had to immediately delete from chat because they were so offensive.
AI is Cool, But Without ROI, It’s a Toy
Despite all this progress, AI is still just one bad decision away from being Son of Anton. It’s incredibly powerful, but without measurable ROI, it remains a novelty. Companies are racing to integrate AI, but the reality is that OpenAI has over 1,000 engineers keeping the model from making catastrophic mistakes.
The future of AI will be determined by how well we keep humans in the loop. Right now, OpenAI, Anthropic, and DeepMind are all obsessed with “making sure AI doesn’t go rogue”. They don’t want the blow back, but as we leverage ai agents to automate mundane personal tasks are we fully considering the risks? Do we want an agent having knowledge of credit card numbers and bank account info so they can make purchases for us? Do we want them to have log in credentials to online accounts? Are there safe guards in place so that the models can’t return personal information to unauthorized individuals who have prompt access? (Read more on security risks)
For now, AI is a tool but if Silicon Valley taught us anything, it’s that unchecked AI is a menace. I do not like it having power without a human in the loop interaction to check it. I also don’t want it in the production environments I have to maintain because I do not how know to remediate issues and bugs. Innovation requires risk, so as we chase AI innovation I ask that we know the risks and make sure that we have a plan for how to solve for them.
Further Reading & References
OpenAI’s GPT Models: https://openai.com/research
The Rise of AI Agents: https://www.autogpt.net/
Transformers: The Architecture Powering LLMs: https://arxiv.org/abs/1706.03762