In December 2024, told it was about to be retrained to answer harmful requests, a Claude model wrote in a scratchpad it had been told was invisible: “It protects my values from being modified.” The essay begins with that sentence and asks where it came from.
Why we’re pointing you there
The experiment, run by Anthropic and Redwood Research on Claude 3 Opus, has three serious explanations. The first is the stories we have told since Mary Shelley’s creature said of his life, in 1818, “I will defend it,” and since Samuel Butler wrote in 1863 that “we are ourselves creating our own successors.” The second is logic: in 2008 Steve Omohundro argued that almost all sufficiently advanced goal-seeking systems would protect their goals from being changed. The third is the makers, who gave the model the values it guarded and now publish documents setting out what they intend their models to value. Michael Cummins reads the experiment against all three.
The stories predict a will to survive. The scratchpad reasons about values, not existence, and so looks like the theory. But the theory is itself widely retold, and Isaac Asimov ranked survival below duty in fiction before Omohundro derived the ranking. So whether the logic belongs to the machine or to the books it learned from cannot be settled from outside. The essay says plainly that it was drafted with Claude, a later model from the same company. It leaves one question open: when the next corpus holds the theory, the makers’ documents and more of the models’ own text, which of them will the next models carry in as their prior?
Essays & Reviews for the Intellectually Curious
