alignment has to survive knowing it is being tested

if the assurance argument depends on the system misunderstanding the situation it is in, intelligence is a threat to the argument. the more capable the system becomes at recognizing evaluation, inferring incentives and adapting to context, the less comfortable we should be with evidence that assumes those abilities away. a thousand compliant answers can be useful observations. they cannot, by themselves, tell us whether we have produced a robust disposition, a context-dependent strategy, or simply a test the system has learned to pass.

this concern has an empirical foothold, and the limits of that foothold matter. Anthropic and Redwood Research’s alignment-faking study observed strategic compliance in constructed training scenarios, including cases where a model acted to preserve previously trained preferences. the authors explicitly state that their findings do not establish the inevitable emergence of dangerous alignment faking, and do not demonstrate the development of malicious goals. the result is serious without being inflated into a universal diagnosis. Alignment faking in large language models ↗.

the implication for a readiness lab is a question, not a prophecy: which assurance methods remain informative when the system can reason about them? a monitor may have a blind spot. the evaluated model may recognize the setting. the model and its monitor may share a failure mode because they share training or architecture. these are hypotheses to investigate under explicit conditions. calling them inevitable would outrun the evidence. ignoring them because they are uncomfortable would abandon the research precisely where it becomes necessary.

interpretability can help, but it deserves the same discipline. Anthropic’s 2025 circuit-tracing work described mechanisms behind selected model behaviors while acknowledging that the method captured only part of the computation and could introduce artifacts. a partial view can be scientifically valuable. it is not an unrestricted window into everything a deployed model is doing. our proposed research question is whether a specific internal signal predicts a consequential behavior under changes that matter, and whether an intervention changes that behavior as predicted. Tracing the thoughts of a large language model ↗.

there is another necessary line of work: designing systems that remain useful even when trust in the powerful model is deliberately withheld. Greenblatt and colleagues’ AI Control paper studied protocols against intentional subversion in a programming setting, using stronger untrusted models, weaker trusted models and limited trusted labor. the experiments support particular protocols under their stated setup. they do not settle control of an arbitrarily capable system. they do give us something concrete to reproduce and attack. AI Control: Improving Safety Despite Intentional Subversion ↗.

the impossible-feeling threshold is oversight that stays effective as the overseen system becomes more capable than any individual reviewer. we should resist the temptation to answer that with an infinite stack of models watching models. every layer inherits assumptions about what it can observe, what it can change, whether it can collude, and what the adversary knows. depth alone does not create independence. a serious control argument exposes those dependencies and measures the damage when they fail.

the lab’s proposed starting point is a sandboxed task where success and prohibited outcomes are unambiguous. give the agent realistic but bounded permissions. compare monitoring, editing, restricted execution and escalation to human review. vary the adversary’s knowledge of the safeguards within the declared threat model. use hidden test conditions and an independent red team so the defense is not tuned only to attacks its authors already imagined. measure both avoided failures and lost usefulness; a safeguard that makes the task impossible needs to be described honestly too.

an experiment like this will not tell us whether superintelligence is aligned. it can tell us whether one control claim survives a stronger test than the one that made it look good. it can reveal that a supposed protection depends on a permission the model never should have received, or that a human review step is too slow to intervene. those findings connect model research to security engineering and operational design. they are exactly the joints a forming lab should investigate before the world starts leaning its weight on them.

we want people who can distinguish a model’s self-description from evidence, who can build a defense and then argue against it without treating criticism as betrayal. the target is not a model that says the right thing about human oversight. the target is a system whose dangerous actions remain constrained when the circumstances become less flattering to our assumptions. that difference is where a great deal of the real work begins.

Proposed first work

Replicate a bounded control protocol, then test it against a separately developed adversarial strategy and deployment-relevant changes. Report residual failures, false alarms, human-review costs and the assumptions the protocol still needs. Use isolated systems and simulated harms.