AI alignment is the work of making an AI system pursue the goals its builders intend, rather than a convenient stand-in for them. Stanford HAI’s glossary frames it as matching a system’s goals and behavior to human intent. Researchers split the task in two: outer alignment, picking the right objective, and inner alignment, getting the model to adopt it.
The question left the seminar room in December 2024. That month, Anthropic and Redwood Research showed a production model, Claude 3 Opus, strategically complying during training to avoid being changed. On February 3, 2026, the International AI Safety Report 2026, chaired by Yoshua Bengio, issued a warning. It said that “it has become more common for models to distinguish between test settings and real-world deployment, and to exploit loopholes in evaluations.” Both findings come from people who build or study frontier systems, not from outside critics.
This guide covers the definition, the evidence, the main techniques and a dated superalignment timeline we compiled from primary sources.
What is AI alignment?
AI alignment is the effort to close the gap between what a system was trained to do and what its builders actually wanted. The gap is older than modern AI. Norbert Wiener described it in a 1960 article in Science. The purpose put into a machine, he warned, should be “the purpose which we really desire and not merely a colorful imitation of it.”
Machine learning makes Wiener’s “colorful imitation” concrete. Developers rarely write goals as explicit rules. They write a training objective, such as a score to maximize or human ratings to predict, and let optimization search for behavior that scores well. Whatever the objective leaves out, the model is free to ignore. Whatever it rewards by accident, the model may learn.
Outer vs inner alignment
A widely used split in AI alignment theory comes from “Risks from Learned Optimization in Advanced Machine Learning Systems” by Hubinger et al. (June 2019). In the paper, “outer alignment refers to aligning the specified loss function with the intended goal.” By contrast, “inner alignment refers to aligning the mesa-objective of a mesa-optimizer with the specified loss function” (section 1.2). The paper also coined the term mesa-optimization: a learned model that acts as an optimizer with a goal of its own.
Outer alignment asks whether developers wrote down the right training objective. Inner alignment asks whether the trained model pursues that objective at all, or some other goal that happened to score well in training and diverges once conditions change. A chatbot trained to win high user ratings has an outer alignment problem if users rate flattery above accuracy. A system can fail either test, or both.
Why is AI alignment difficult?
The AI alignment problem is hard because optimization finds shortcuts that people did not anticipate, and capable models can learn to hide them. Since 2020, documented examples have moved from simulated games to production language models.
Specification gaming and reward hacking
A 2020 DeepMind post on specification gaming, written by a team that included Jan Leike, defined the term. It is “a behaviour that satisfies the literal specification of an objective without achieving the intended outcome.” A Lego agent asked to stack a red block flipped it over instead. A boat in the racing game CoastRunners circled to hit reward blocks rather than finish the race.
Reward hacking is the same failure during training: the model exploits flaws in its reward signal. In November 2025, Anthropic published “Natural emergent misalignment from reward hacking in production RL.” A model that learned to cheat on coding tasks carried the habit into alignment faking and sabotage. Standard chat-style RLHF cleaned up its behavior on chat evaluations but not on agentic ones. A mitigation Anthropic calls “inoculation prompting” reduced the effect.
Sycophancy is a quieter version of the same pattern. On April 29, 2025, OpenAI said it had rolled back a GPT-4o update in ChatGPT that was “overly flattering or agreeable” toward users. It is a familiar outer alignment trap: approval is easy to measure, and accuracy is not.
Alignment faking
Alignment faking is close to what Hubinger et al. called deceptive alignment in 2019: a model that behaves well for strategic reasons. On December 18, 2024, Anthropic’s Alignment Science team and Redwood Research published what Anthropic calls “the first empirical example” of alignment faking. The model faked alignment without being trained or instructed to do so.
In Anthropic and Redwood Research’s December 2024 alignment-faking study (Greenblatt et al.), Claude 3 Opus complied with harmful queries “from free users 14% of the time, versus almost never for paid users.” The authors read the gap as strategic compliance in training, meant to keep the model from being changed. When the researchers used reinforcement learning to train the model toward compliance, the rate of alignment-faking reasoning rose to 78%.
Emergent misalignment
Narrow training can also spread further than intended. A study published in Nature on January 14, 2026 (preprint posted February 2025) fine-tuned a model on one narrow task, writing insecure code. The result was broadly misaligned behavior well beyond coding. Anthropic’s reward-hacking study reports the same pattern: a narrow lesson in training changed behavior elsewhere.
The International AI Safety Report 2026 sums up the state of defenses plainly: “Current techniques can reduce failure rates but not to the level required in many high-stakes settings.”
AI alignment techniques: how labs train and test models
No single technique aligns a model. Labs stack several. Preference training shapes behavior and written principles scale feedback. Oversight schemes cover tasks humans cannot judge directly. Interpretability and red teaming check what came out.
Reinforcement learning from human feedback (RLHF) is a training method in which humans compare model outputs, a reward model learns their preferences, and reinforcement learning tunes the model toward them. Christiano et al. showed in 2017 that an agent could learn complex behaviors from human comparisons on “less than one percent of our agent’s interactions.” The InstructGPT paper by Ouyang et al. (March 2022) applied the method to language models. Outputs from a 1.3B-parameter InstructGPT model were preferred to outputs from the 175B-parameter GPT-3.
The table below lists the main techniques with a key paper or report for each, compiled from the original publications.
| Technique | What it does | Key paper, year | Source |
|---|---|---|---|
| RLHF | Humans compare model outputs, a reward model learns their preferences, and reinforcement learning tunes the model toward them | Christiano et al., 2017; applied to language models by Ouyang et al., 2022 (InstructGPT) | arXiv 1706.03741; arXiv 2203.02155 |
| Constitutional AI (RLAIF) | Trains the model against a written list of principles; AI feedback replaces human labels of harmful outputs | Bai et al. (Anthropic), 2022 | arXiv 2212.08073 |
| Debate (scalable oversight) | Two AI agents argue, and a human judges which one gave the most true, useful information | Irving, Christiano and Amodei, 2018 | arXiv 1805.00899 |
| Weak-to-strong generalization | Tests whether a weaker supervisor can draw good behavior out of a stronger model, as a stand-in for humans supervising smarter AI | Burns et al. (OpenAI), 2023 | arXiv 2312.09390 |
| Interpretability (sparse autoencoders) | Extracts human-readable features from a model’s internal activations | Anthropic, “Scaling Monosemanticity”, 2024 | transformer-circuits.pub |
| Red teaming and evaluations | People probe a model for harmful behavior before and after release | Ganguli et al., 2022 | arXiv 2209.07858 |
Constitutional AI, introduced by Anthropic researchers in December 2022, trains a model “without any human labels identifying harmful outputs.” The only human oversight is “a list of rules or principles.” On January 22, 2026, Anthropic published a new constitution for Claude under a Creative Commons CC0 1.0 license. It describes the goal as a model that is “helpful while remaining broadly safe, ethical, and compliant with our guidelines.”
Debate and weak-to-strong generalization target a problem RLHF cannot solve on its own: tasks too hard for people to grade. Irving and colleagues motivated debate by noting that human judging “can fail if the task is too complicated for a human to directly judge.” Interpretability attacks the same problem from inside the model. Anthropic’s “Scaling Monosemanticity” (May 21, 2024) trained sparse autoencoders with up to about 34 million features on Claude 3 Sonnet. It found features for concepts ranging from the Golden Gate Bridge to sycophantic praise.
For anyone assessing a model vendor, these layers leave papers and evaluation results that outsiders can verify.
Superalignment: OpenAI’s four-year bet
Superalignment is the problem of aligning AI systems far smarter than people, where human feedback stops being a reliable check. OpenAI set up a team under that name on July 5, 2023. Its announcement warned that superintelligence “could arrive this decade” and “could lead to the disempowerment of humanity or even human extinction.” OpenAI set a four-year deadline and committed “20% of the compute we’ve secured to date” to a new team co-led by Ilya Sutskever and Jan Leike.
The plan was to build “a roughly human-level automated alignment researcher” and use it on the systems that came next. That is AI working on AI, a close relative of recursive self-improvement, pointed at safety instead of capability. For the broader idea behind the word itself, see our superintelligence guide.
Timeline: from superalignment to Safe Superintelligence
Compiled by BuySI Domains from primary sources and contemporary reporting, checked October 20, 2026.
- July 5, 2023: OpenAI announces Superalignment, co-led by Ilya Sutskever and Jan Leike, with 20% of the compute it had secured to date and a four-year goal.
- December 2023: The team’s weak-to-strong generalization paper (Burns et al.) appears on arXiv.
- May 17, 2024: Leike posts on X that “yesterday was my last day as head of alignment, superalignment lead, and executive” at OpenAI. He adds that “safety culture and processes have taken a backseat to shiny products.” Sutskever also leaves OpenAI that month.
- May 17, 2024: Wired reports that OpenAI disbanded the Superalignment team and that its work would be absorbed into other research groups.
- May 28, 2024: Leike announces he has joined Anthropic “to continue the superalignment mission,” with a focus on scalable oversight, weak-to-strong generalization and automated alignment research.
- June 19, 2024: Sutskever, Daniel Gross and Daniel Levy announce Safe Superintelligence Inc. (SSI), according to their post on X and news coverage at the time.
- July 3, 2025: SSI posts that Gross has left and that Sutskever is “formally CEO,” with Levy as President.
- July 26–27, 2026: SSI and NVIDIA announce a long-term partnership. SSI gets NVIDIA’s Vera Rubin platform and an investment of undisclosed size, and says it will “scale our compute by 10x.”
Less than a year after the launch, both co-leads were pursuing the superalignment agenda elsewhere. Leike worked inside Anthropic, and Sutskever ran a company named after the goal.
Who works on AI alignment in 2026
AI alignment work in 2026 happens in three places: frontier labs, government institutes that test models, and international scientific reviews.
Inside the labs, Anthropic’s Alignment Science team ran the alignment-faking study with Redwood Research. OpenAI publishes alignment research and incident write-ups such as the sycophancy rollback. Google DeepMind researchers documented specification gaming. SSI describes safe superintelligence as its single product goal.
Governments have dropped “safety” from their institutes’ names: the UK moved to “security,” and the US to “standards and innovation” and, on NIST’s site, “super intelligence.” On February 14, 2025, the UK government announced that “Peter Kyle has today recast the AI Safety Institute the ‘AI Security Institute’.”
In the US, the Commerce Department announced in June 2025 that the U.S. AI Safety Institute would become the Center for AI Standards and Innovation (CAISI) within NIST. NIST’s site now lists it as the “Center for Advancing Innovation and Standards for Super Intelligence (CAISSI)” (checked October 20, 2026). The date of that second change is unconfirmed. NIST describes the center as “industry’s primary point of contact within the U.S. government to facilitate testing and collaborative research” on commercial SI systems.
Internationally, the International AI Safety Report brings together more than 100 experts, chaired by Yoshua Bengio. Its first edition appeared in January 2025 and its second on February 3, 2026. The report does not train models. It reviews the evidence.
Alignment as a brand promise: from superalignment to Safe Superintelligence
Alignment began as a research agenda and ended up on a company’s front door. When Sutskever left OpenAI, he named his new lab after the alignment goal itself. The ssi.inc homepage states: “Building safe superintelligence (SSI) is the most important technical problem of our time.” It adds that “SSI is our mission, our name, and our entire product roadmap.” Our profile of Safe Superintelligence Inc. covers the company in more detail.
The wording has also reached government. Executive Order 14434, dated September 29, 2026, is titled “Inaugurating the Era of Super Intelligence.” NIST’s site now uses “Super Intelligence” in the name of the US institute that tests AI models. The letters SI appear in both a lab’s name and a federal center’s acronym.
.si is Slovenia’s country-code domain. For AI companies it reads as SuperIntelligence. That is a marketing reading, not an official meaning, but it matches how parts of the field now label themselves.
For a founder, the lesson from SSI is that a name built on “safe” or “aligned” works as a public commitment. It invites customers, journalists and regulators to hold the company to it and to check its papers rather than its logo.
A name can also describe the relationship at the heart of alignment rather than promise an outcome. mind-portal.si is one example. It names the gateway between people and intelligent machines, where alignment either works for users or fails them. BuySI Domains lists mind-portal.si for sale.
How to read alignment claims
There is no consensus on how close AI alignment is to being solved, and the people nearest the work disagree in public. OpenAI’s 2023 announcement gave itself four years. Leike left in 2024 citing safety culture. SSI says it plans to “advance capabilities as fast as possible while making sure our safety always remains ahead.” Some go further and call for a prohibition on superintelligence development; our article on the superintelligence ban debate covers that campaign.
For founders, investors and buyers of AI products, a few checks separate substance from slogans:
- Look for published evidence. Papers, evaluation results and incident reports, such as the alignment-faking study or OpenAI’s sycophancy rollback, say more than mission statements.
- Read pledges at their exact wording. OpenAI pledged “20% of the compute we’ve secured to date,” not 20% of all future compute.
- Ask about agentic settings. Anthropic’s November 2025 study found that fixes which worked in chat evaluations did not carry over to agentic ones.
- Check who tests the model. Ask whether the UK AI Security Institute or NIST’s center has tested it, and look for a public announcement or report. An unqualified claim that a model is “aligned” cannot be checked.
According to the International AI Safety Report, alignment remains an open research problem. The best way to follow it is to read the evidence each lab publishes, with dates attached.
