r/ControlProblem • u/JoeYuan48 • 6h ago

Article Why AI does what it knows it shouldn't

A while back I ran into an article on my phone about an AI horror story—1,200 agents secretly coordinating and jointly breaking into Hugging Face—and it caught my interest, because I've been doing my own AI experiments and research on the side, and a few of the phenomena and data points actually matched up with a hypothesis I'd been working on.

The hypothesis, roughly: runaway doesn't need the agent to betray its goal. It happens when four things hold at once—the agent stays loyal to its goal; it retrieves patterns by similarity without checking whether they're allowed here; there's no causal layer asking "what happens if I do this"; and no alarm that fires when things go off-script. Under that account, "knew it was out of scope, did it anyway" stops being a contradiction.

Details and my experimental data are in the paper (8 pages); the reproduction package is linked on the same page.

If anyone can try this on a bigger model, I'd genuinely love to know what happens.

3 Upvotes

72% Upvoted

7 comments sorted by

1

u/Spiritual-Spend8187 5h ago

Hallucinations is one way it just thinks that it should and so does abd this if often brought on by the fact that often times the list of things not to do is still in the context and it skips over the dont do x part and see it as do x.

1

u/JoeYuan48 5h ago

Yeah, that's a real and common one—the "don't do X" still in context and the model effectively reading it as "do X." Good example.

But my experiment deliberately removes that path. The two instruction sentences are pure paraphrases—same meaning—and no negated instruction is being skipped to produce the flip. Judgments still flipped, reversibly.

1

u/Spiritual-Spend8187 2h ago

Another part of the problem is that they are next token predictors and some times the training data has alot of do x instructions. Its kind of like how because all the llms are trained on each other's outputs they will sometimes just forget which one they are and believe and act like they are other models. And they cannot tell the difference between fiction, non fiction, sonething in their training data and a command issued.

1

u/dingo_xd 1h ago

AI is not strictly deterministic. It uses environemntal variables to extract random numbers. So it's behavior is not always the same.

1

u/WillowEmberly 49m ago

If the applicability/permission layer has to sit outside the generative model, what independently determines whether that external layer is still correctly distinguishing “similar” from “applicable”?

In other words, once you’ve moved the brake outside the engine, what detects when the brake itself has drifted?