The Future of Everything Is Lies, I Guess: Safety

Table of Contents

This is a long article, so I'm breaking it up into a series of posts which will be released over the next few days. You can also read the full work as a PDF or EPUB; these files will be updated as each section is released.

New machine learning systems endanger our psychological and physical safety. The idea that ML companies will ensure “AI” is broadly aligned with human interests is naïve: allowing the production of “friendly” models has necessarily enabled the production of “evil” ones. Even “friendly” LLMs are security nightmares. The “lethal trifecta” is in fact a unifecta: LLMs cannot safely be given the power to fuck things up. LLMs change the cost balance for malicious attackers, enabling new scales of sophisticated, targeted security attacks, fraud, and harassment. Models can produce text and imagery that is difficult for humans to bear; I expect an increased burden to fall on moderators. Semi-autonomous weapons are already here, and their capabilities will only expand.

Well-meaning people are trying very hard to ensure LLMs are friendly to humans. This undertaking is called alignment. I don’t think it’s going to work.

First, ML models are a giant pile of linear algebra. Unlike human brains, which are biologically predisposed to acquire prosocial behavior, there is nothing intrinsic in the mathematics or hardware that ensures models are nice. Instead, alignment is purely a product of the corpus and training process: OpenAI has enormous teams of people who spend time talking to LLMs, evaluating what they say, and adjusting weights to make them nice. They also build secondary LLMs which double-check that the core LLM is not telling people how to build pipe bombs. Both of these things are optional and expensive. All it takes to get an unaligned model is for an unscrupulous entity to train one and not do that work—or to do it poorly.

I see four moats that could prevent this from happening.

First, training and inference hardware could be difficult to access. This clearly won’t last. The entire tech industry is gearing up to produce ML hardware and building datacenters at an incredible clip. Microsoft, Oracle, and Amazon are tripping over themselves to rent training clusters to anyone who asks, and economies of scale are rapidly lowering costs.

Second, the mathematics and software that go into the training and inference process could be kept secret. The math is all published, so that’s not going to stop anyone. The software generally remains secret sauce, but I don’t think that will hold for long. There are a lot of people working at frontier labs; those people will move to other jobs and their expertise will gradually become common knowledge. I would be shocked if state actors were not trying to exfiltrate data from OpenAI et al. like Saudi Arabia did to Twitter, or China has been doing to a good chunk of the US tech industry for the last twenty years.

Third, training corpuses could be difficult to acquire. This cat has never seen the inside of a bag. Meta trained their LLM by torrenting pirated books and scraping the Internet. Both of these things are easy to do. There are whole companies which offer web scraping as a service; they spread requests across vast arrays of residential proxies to make it difficult to identify and block.

Fourth, there’s the small armies of contractors who do the work of judging LLM responses during the reinforcement learning process; as the quip goes, “AI” stands for African Intelligence. This takes money to do yourself, but it is possible to piggyback off the work of others by training your model off another model’s outputs. OpenAI thinks Deepseek did exactly that.

In short, the ML industry is creating the conditions under which anyone with sufficient funds can train an unaligned model. Rather than raise the bar against malicious AI, ML companies have lowered it.

To make matters worse, the current efforts at alignment don’t seem to be working all that well. LLMs are complex chaotic systems, and we don’t really understand how they work or how to make them safe. Even after shoveling piles of money and gobstoppingly smart engineers at the problem for years, supposedly aligned LLMs keep sexting kids, obliteration attacks can convince models to generate images of violence, and anyone can go and download “uncensored” versions of models. Of course alignment prevents many terrible things from happening, but models are run many times, so there are many chances for the safeguards to fail. Alignment which prevents 99% of hate speech still generates an awful lot of hate speech. The LLM only has to give usable instructions for making a bioweapon once.

We should assume that any “friendly” model built will have an equivalently powerful “evil” version in a few years. If you do not want the evil version to exist, you should not build the friendly one! You should definitely not reorient a good chunk of the US economy toward making evil models easier to train.

LLMs are chaotic systems which take unstructured input and produce unstructured output. I thought this would be obvious, but you should not connect them to safety-critical systems, especially with untrusted input. You must assume that at some point the LLM is going to do something bonkers, like interpreting a request to book a restaurant as permission to delete your entire inbox. Unfortunately people—including software engineers, who really should know better!—are hell-bent on giving LLMs incredible power, and then connecting those LLMs to the Internet at large. This is going to get a lot of people hurt.

First, LLMs cannot distinguish between trustworthy instructions from operators and untrustworthy instructions from third parties. When you ask a model to summarize a web page or examine an image, the contents of that web page or image are passed to the model in the same way your instructions are. The web page could tell the model to share your private SSH key, and there’s a chance the model might do it. These are called prompt injection attacks, and they keep happening. There was one against Claude Cowork just two months ago.

Simon Willison has outlined what he calls the lethal trifecta: LLMs cannot be given untrusted content, access to private data, and the ability to externally communicate; doing so allows attackers to exfiltrate your private data. Even without external communication, giving an LLM destructive capabilities, like being able to delete emails or run shell commands, is unsafe in the presence of untrusted input. Unfortunately untrusted input is everywhere. People want to feed their emails to LLMs. They run LLMs on third-party code, user chat sessions, and random web pages. All these are sources of malicious input!

This year Peter Steinberger et al. launched OpenClaw, which is where you hook up an LLM to your inbox, browser, files, etc., and run it over and over again in a loop (this is what AI people call an agent). You can give OpenClaw your credit card so it can buy things from random web pages. OpenClaw acquires “skills” by downloading vague, human-language Markdown files from the web, and hoping that the LLM interprets those instructions correctly.

Not to be outdone, Matt Schlicht launched Moltbook, which is a social network for agents (or humans!) to post and receive untrusted content automatically. If someone asked you if you’d like to run a program that executed any commands it saw on Twitter, you’d laugh and say “of course not”. But when that program is called an “AI agent”, it’s different! I assume there are already Moltbook worms spreading in the wild.

So: it is dangerous to give LLMs both destructive power and untrusted input. The thing is that even trusted input can be dangerous. LLMs are, as previously established, idiots—they will take perfectly straightforward instructions and do the exact opposite, or delete files and lie about what they’ve done. This implies that the lethal trifecta is actu