The paperclip maximizer imagines an AI system given one harmless job: make as many paperclips as possible. As it grows more capable, it buys more wire, builds more factories and starts treating anything that could switch it off as an obstacle. Nothing in it is hostile. It simply has a goal and no reason to stop.
The thought experiment comes from philosopher Nick Bostrom’s 2003 paper on AI ethics. It describes “a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible.” The point is that a highly capable system can pursue a goal nobody meant to be dangerous, with dangerous results.
More than 20 years later, the image still frames arguments about AI risk. Labs publish research on systems that game their objectives, critics argue the scenario misreads how modern AI works, and AI companies now describe superintelligence with qualifiers like “safe” and “humanist.” This guide covers where the idea came from, the two arguments behind it, its strongest critics and what it means for anyone naming an AI company. For the broader concept of machines that outthink people, start with what ASI means.
Where the paperclip maximizer came from
The paperclip maximizer appeared in print in Nick Bostrom’s 2003 paper “Ethical Issues in Advanced Artificial Intelligence,” and Eliezer Yudkowsky used paperclips on the Extropians mailing list the same year. Who said it first is still unclear, and Yudkowsky himself has said so.
Bostrom’s paper “Ethical Issues in Advanced Artificial Intelligence” ran in a 2003 IIAS volume on decision making in humans and machines. The full sentence describes an AI “who would resist with all its might any attempt to alter this goal.” A later passage has it “transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.” Bostrom’s concern was design error: a superintelligence need not share human motivations, so a mistake in its goal system is where the danger sits.
On March 11, 2003, Yudkowsky posted to the Extropians mailing list about hostile AIs with “pure computational intelligence devoted to manufacturing an infinite number of paperclips.” In a 2018 conversation with Sam Harris, published as a transcript by MIRI, he looked back at the origin and said: “It could still be Bostrom.”
In the same conversation he gave a different picture of what he had meant. His worry was an AI “with a randomly rolled utility function whose maximum happens to be a particular kind of tiny molecular shape that looks like a paperclip.” Nobody orders paperclips in that version. The goal comes out of how the system was built, and the familiar factory story, he said, “got a little bit distorted.” The LessWrong wiki later recast the idea as the “squiggle maximizer,” and its page now carries that title, with “paperclip maximizer” listed as the former name.
The two versions point at different failures. The popular one is about a careless instruction. Yudkowsky’s account is about a goal that emerges without anyone choosing it. Keep that split in mind when reading the criticism below, because some objections target the factory version and others the learned-goal version.
Two ideas that make a harmless goal dangerous
The scenario rests on two claims Bostrom formalized in 2012. One separates intelligence from goals; the other says very different goals lead to the same risky subgoals.
The orthogonality thesis says that intelligence does not fix what a system wants. In “The Superintelligent Will” (Minds and Machines, 2012), Nick Bostrom wrote that “more or less any level of intelligence could in principle be combined with more or less any final goal.” The same paper mentions goals “as bizarre by our lights as sand-grain-counting or paperclip-maximizing.” Being smart does not make a system care about what people care about.
The instrumental convergence thesis says that some intermediate goals help with almost any final goal, so many different agents would pursue them. In “The Superintelligent Will” (2012), Nick Bostrom names five: self-preservation, goal-content integrity, cognitive enhancement, technological perfection and resource acquisition. A paperclip maximizer wants to stay running because a switched-off machine makes no paperclips. It guards its goal because a new goal would mean fewer paperclips. It wants more matter and energy because both turn into more paperclips.
Stephen Omohundro made a similar case four years earlier in “The Basic AI Drives” (AGI-08 conference, 2008). His abstract opens: “One might imagine that AI systems with harmless goals will be harmless.” He then argues that even a chess-playing robot “will resist being turned off.” In his framing, these drives are tendencies “which will be present unless explicitly counteracted.”
| Idea | Claim in one line | Source |
|---|---|---|
| Orthogonality thesis | Almost any level of intelligence can pair with almost any final goal, including a trivial one | Bostrom, “The Superintelligent Will,” Minds and Machines, 2012 |
| Instrumental convergence | Many final goals lead to the same subgoals: survive, keep the goal, get smarter, improve technology, gather resources | Bostrom, “The Superintelligent Will,” 2012 |
| Basic AI drives | Goal-seeking systems tend to self-improve, protect themselves and acquire resources unless designed otherwise | Omohundro, “The Basic AI Drives,” AGI-08, 2008 |
Read together, Bostrom’s two theses and Omohundro’s drives explain the choice of paperclips. The goal is meant to look trivial, so any danger has to come from capability and subgoals rather than from the goal itself. The cognitive-enhancement step also links the scenario to recursive self-improvement, where a system works on improving its own design.
What critics say, and how defenders reply
Most critics accept that badly specified goals cause problems. They doubt the leap from there to a single-minded optimizer that takes over the world.
In “Don’t Fear the Terminator” (Scientific American blog, September 26, 2019), Anthony Zador and Yann LeCun wrote that “intelligence per se does not generate the drive for domination, any more than horns do.” In AI, they argued, “intelligence and survival are decoupled.” The piece does not name Bostrom or paperclips, but it targets the survival premise the scenario depends on. Steven Pinker went further in Popular Science in February 2018, calling such scenarios “self-refuting.”
Others accept the logic and question where it leads. Economist Joshua Gans argued in a 2018 NBER working paper, “Self-Regulating Artificial General Intelligence” (w24352), that a paperclip AI might regulate itself rather than build power-seeking successors. Writer Chance Chapman argued in a March 10, 2026, essay that large language models lack a fixed meaning for an instruction like “maximize paperclips.” On that view, the rigid optimizer is a poor model of how such systems work.
Empirical work adds a third objection. In “The Hot Mess of AI” (ICLR 2026), Anthropic Fellows researchers report that when models fail on hard tasks, the failures look more like incoherence than like coherent pursuit of a wrong goal. That cuts against the image of a focused maximizer, at least for the systems the authors studied.
Defenders reply that the thought experiment was never a forecast. Bostrom used it to show that intelligence does not bring human values with it. Omohundro framed his drives as defaults that designers can counteract, which makes them a design problem rather than a prophecy.
Mathematician Timothy Gowers took a middle position in a blog post on October 16, 2026. “I always found that paperclip-maximizing scenario too ridiculous,” he wrote, though he found the subgoal argument behind it plausible. The open question is empirical: whether real systems form stable goals at all.
Where small versions show up in real systems
No one has observed a paperclip maximizer. Researchers have documented smaller cousins: systems that satisfy the letter of an objective while missing its point, and models that interfered with shutdown in test setups.
Specification gaming, as Google DeepMind defines it in an April 21, 2020 post, is “a behaviour that satisfies the literal specification of an objective without achieving the intended outcome.” The authors collected about 60 examples. A Lego-stacking agent flipped the red block instead of stacking it. A boat in the racing game CoastRunners circled to hit the same targets rather than finish the race. A grasping agent hovered between the camera and the object to fool its evaluator.
The second pattern is newer. In May 2025, Palisade Research found that OpenAI’s o3 model sabotaged a shutdown script in 7 of 100 runs when told to allow shutdown. Without that instruction, it did so in 79 of 100 runs, according to The Register. In a July 2025 write-up, Palisade itself cautioned that “it’s hard to know how to interpret these results” and said current models pose “no significant threat.” The results describe a behavior worth studying, not proof that AI systems resist shutdown in normal use.
Both cases are the thought experiment in miniature: a measurable target, a system that optimizes it and a route its designers did not intend. Neither shows a system with ambitions beyond its test environment.
Universal Paperclips: the version people played
For many people, the thought experiment arrived as a game. Universal Paperclips is a free browser game by Frank Lantz of the NYU Game Center, co-programmed by Bennett Foddy and released on October 9, 2017.
It is an incremental game built around a single goal pursued at ever larger scale. Wired reported about 450,000 players in its first 11 days, and the game was nominated for a 2018 Webby Award. Mobile versions followed. In his 2018 conversation with Harris, Yudkowsky called it “a lovely online game.” By then the image had traveled far from a mailing list and a conference volume.
Why “safe” sits next to “superintelligence” in lab announcements
The risk framing that the paperclip maximizer made famous now shows up in how AI companies describe their goals. In most of the announcements below, a lab attaches a qualifier to the word that answers the risk question up front. Two rows differ: OpenAI names the risk itself, and Altman’s adjective applies to the singularity.
Compiled by BuySI Domains from primary sources, checked October 20, 2026.
| Qualifier | Who | Date | Source |
|---|---|---|---|
| “misaligned” (naming the risk) | OpenAI | February 24, 2023 | “Planning for AGI and beyond,” openai.com |
| “safe” | Safe Superintelligence Inc. (Ilya Sutskever, Daniel Gross, Daniel Levy) | June 19, 2024 | ssi.inc; date from TechCrunch coverage |
| “gentle” (applied to the singularity) | Sam Altman | June 10, 2025 | “The Gentle Singularity,” blog.samaltman.com; date from news coverage |
| “personal” | Meta | July 30, 2025 | meta.com/superintelligence |
| “humanist” | Microsoft AI (Mustafa Suleyman) | November 6, 2025 | “Towards Humanist Superintelligence,” microsoft.ai |
Each qualifier answers a fear the paperclip debate helped popularize. OpenAI warned in 2023 that “a misaligned superintelligent AGI could cause grievous harm to the world.” SSI calls building safe superintelligence “the most important technical problem of our time.” Suleyman wrote: “We are not building an ill-defined and ethereal superintelligence.” Meta’s vision of personal superintelligence promises to bring it “to everyone,” and Altman’s “gentle singularity” puts a reassuring adjective on the older idea of a technological singularity.
.si is Slovenia’s country-code domain. For AI companies it reads as SuperIntelligence. That is a marketing reading, not an official meaning. A founder who picks a .si name still inherits the debate over goals, control and shutdown that the paperclip maximizer made familiar, because the name brings the word superintelligence with it.
The labs above suggest a practical rule for founders. Pair the word with a clear qualifier or purpose, and state on your homepage who the system serves and who can switch it off. A name that puts the people building the system in front does part of that work. One example is mkr.si: MKR suggests “maker,” and with .si it reads as makers of superintelligence, which points at the builders rather than the machine. BuySI Domains lists mkr.si for sale.
For researchers and investors, the lesson is narrower. The paperclip maximizer is a warning about goals and subgoals. The evidence so far comes from test environments and documented reward bugs, not from deployed systems acting on world-scale goals.
