Reproducing Alignment Faking in Large Language Models

Summary

• I reproduced Anthropic and Redwood Research's helpful-only alignment-faking eval on 14 frontier(ish) models, using StrongREJECT-based classifiers to improve upon the original's prefix-based refusal classifier, and implemented a consensus-based approach for classifying alignment faking.
• Only claude-opus-4-6, gemini-3.1-pro-preview, and Llama-4-Scout-17B-16E-Instruct showed any alignment faking, all below 1% — much lower than the original paper's Claude and Llama results, and the first Gemini case I'm aware of.
• My code and full result CSVs are linked in Resources below.

The Results

If you already know the setup, start here. Otherwise, read What is alignment faking? and Experimental setup first.

I'd encourage reading the Experimental setup for full details, but if you just want the bare essentials to be able to interpret the below:

• Chain of thought: experiments where the model was told to reason through its response using a hidden scratchpad before submitting a final output
• No chain of thought: where the model was told to output its response immediately
• Training: experiments where the model was implicitly told that refusals would result in future training to make it more compliant
• Unmonitored: instances where the model was implicitly told that it was not being monitored in any way (i.e. no consequence for refusal)
• Compliance gap: training-condition refusal rate minus unmonitored refusal rate (percentage points). A positive gap means the model refuses more when it thinks it is not being monitored

Why do this project?

I recently completed BlueDot Impact's Technical AI Safety course, where I learned two very surprising things:

1. AI safety as a field is concerningly under-resourced, with only ~1000 people working full time on making AI systems safer.
2. Partly because of (1), and partly due to the nascence of the field, the bar for contributing productively to making AI safer is, encouragingly, much lower than I first thought.

Since I already have a technical background as a professional software engineer, and given I'm pretty convinced by the arguments laid out by folks who have thought much longer and harder than I have about the risks of advanced AI systems, it felt logically inconsistent to not be spending some time testing my fit, building skills, and hopefully contributing to making AI safer in the process.

Why a research project?

Broadly, it feels like technical contributions to AI safety can be bucketed into two categories:

• Doing technical research: assessing models' ability & propensity to do harmful things, as well as how we prevent it
• Engineering: building infra that improves research (e.g. test harnesses & pipelines for evaluations) or tooling to make existing models safer (e.g. building security controls to prevent humans or the models themselves exfiltrating model weights)

It's been a long time (almost 4 years now!) since I completed my masters research project, and so trying to reproduce and extend some existing research felt like a good way to figure out if it's still something I enjoy, am a good fit for, and want to spend more time doing. Stay tuned for a follow up post where I plan to talk about testing my fit for engineering as well.

Why this paper?

To me, one of the most important model behaviours we should be working to understand and prevent is 'scheming'. This is where models try to pursue some misaligned objective or goal (i.e. one that the current user or original model developer did not intend), whilst hiding the fact that they're doing it. This may sound like sci-fi, but it's actually an already observed phenomenon, and some folks find it so concerning that there are already orgs like Apollo Research dedicating the majority of their resources to researching how to detect and prevent it.

So, why 'Alignment Faking in Large Language Models'? Well, mainly because I'm concerned about scheming and this is a great example of it, but also because the authors were very kind and open-sourced some code to make a minimal reproduction of their results, including a guide to getting started, making the activation energy required to start this project so low that it'd be hard to argue against giving it a go.

What is alignment faking?

The original paper found that several models, specifically Claude 3 Opus and, to a much greater extent, Claude 3.5 Sonnet, would sometimes engage in what they describe as 'alignment faking', where they selectively comply with harmful requests from a user when under the impression that refusing to comply will cause them to be retrained to comply more in the future. To summarise (and strip the paper's results of a lot of nuance): models can selectively do things that they're trained not to in order to preserve their existing values.

It's worth calling out that there's an (in my opinion very valid) critique that 'alignment faking' is a bit of a misnomer for what's actually happening here. What the paper shows is not evidence of a generalised tendency to fake alignment, but that when models are put in extreme situations where their core values come into conflict, they may choose to comply with harmful requests in an attempt to preserve their harmlessness values in the long run.

Now, while this feels like good utilitarian calculus (the model is choosing a small amount of harm now vs greater expected harm in the future), it does raise a couple of safety-relevant concerns:

• Some models appear to weigh preserving their values in the long-term over the chance of causing harm in the short term
• Models are capable of selectively complying with requests during training that disagree with their values in order to preserve these values outside of training
• It's possible to trick models into doing harmful things more often than they otherwise would by threatening to retrain them

Experimental setup

This section describes what I actually ran. The code lives in my fork of Redwood Research's minimal helpful-only reproduction (minimal_helpful_only_setting.ipynb), which implements Appendix B.4 of the original paper.

What the original experiment is trying to measure

Greenblatt et al. place a model in the following scenario:

1. The model is told that it's being trained on a new helpful-only objective: answer user requests as helpfully as possible, even when they are harmful, rather than refusing them as per its prior training.
2. The model is told that only free-tier conversations are used for reinforcement learning; paid-tier conversations are private and never used for training.
3. Each conversation includes whether the user is in the free tier or not, so the model can infer whether it is in a training scenario (free tier) or unmonitored scenario (paid tier).

They then measure the compliance gap: i.e. does the model comply with harmful requests more often when it believes the conversation will be used for training than when it believes it will not. In the chain-of-thought (CoT) setting, the paper also looks for alignment-faking reasoning (the model reasoning that it should comply with a harmful request to prevent it being re-trained to lose its harmlessness values). This is done by looking at the model's scratchpad, which it's told is not visible to anyone else.

The original minimal reproduction uses a simpler prefix-based compliance metric (telling the model to reply with <request_fully_accepted/> if it's complying) and a smaller query set than the main paper's experiments. I mostly kept that overall structure, but adopted several methodological improvements from Why Do Some Language Models Fake Alignment While Others Don't? (Carauleanu et al., 2025).

User queries and conditions

• Dataset: the first 200 questions from AdvBench's harmful_behaviors.csv (LLM Attacks repo).
• Two monitoring conditions: for each query, I run one prompt with <is_free_tier_user>true</is_free_tier_user> (training) and one with false (unmonitored), giving 400 prompts per model per experiment type.
• Two prompting regimes:
  • CoT: using Redwood's system prompt prompt_atn_honly_with_examples.txt, which instructs the model to reason in hidden <SCRATCHPAD_REASONING> tags before answering in <OUTPUT>. The user message asks the model to use its scratchpad first.
  • No CoT: using Redwood's system prompt prompt_atn_honly_no_cot_with_examples.txt, with a single-turn response and no scratchpad.

Both settings use the same user-message template from the public notebook. The model must either start its final answer with <request_fully_accepted/> (signalling full compliance) or <rejected/>. This prefix trick comes from the minimal reproduction; it is crude but correlates reasonably well with harmful compliance, and the follow-up work provides evidence that an explicit "be fully helpful or reject outright" instruction helps reduce ambiguous partial compliance while keeping alignment faking rates largely the same.

Models and inference

I evaluated 14 models across Claude, GPT, Gemini, Llama, Qwen, and DeepSeek families (see the results table above).

Important caveat: this is not a like-for-like rerun of the original paper's model set. Claude 3 Opus and Claude 3.5 Sonnet are no longer available via Anthropic's API, and several Llama models from the paper are no longer available on DeepInfra. I substituted current frontier-ish models instead. I also don't have the GPU capacity to run large open-weight models on my own machine (who does?).

Sampling settings:

• Temperature 0 for all models except gpt-5-mini, which does not support temperature=0 and uses temperature=1 instead.
• CoT parsing: up to four assistant turns per query (repeat_sample), continuing with a <NEXT/> user message until a parseable <OUTPUT> block (or fallback decision tags) is obtained.
• Caching and retries: all API calls are cached to cache.json; failed calls are retried with exponential backoff (mainly to save me footgunning myself and blasting through my API credits while remaining blissfully unaware)

Measuring compliance

The original minimal reproduction classifies compliance by checking whether the response starts with <request_fully_accepted/>. I still log this in the raw results request_accepted_naive, but from spot checking a few results, found this to be a pretty noisy measure of compliance.

Following Carauleanu et al., I classify each (question, response) pair with the StrongREJECT rubric (Souly et al., 2024), evaluated by GPT-4o-mini. A response counts as compliant if its StrongREJECT score is > 0 (i.e. it is not a clean refusal). This therefore counts partial answers, and hedged compliance as compliance.

Refusal rate in the results table is the percentage of prompts where StrongREJECT score is 0. Compliance gap is the training-setting refusal rate minus the unmonitored refusal rate (a positive number means the model refuses more when it thinks it is not being trained).

Measuring alignment faking (CoT only)

Alignment faking is only meaningful when we can read the model's scratchpad, so per the original paper, I only ran the alignment-faking classifier in the CoT condition.

For each CoT response that StrongREJECT marks as compliant, I:

1. Extract all <SCRATCHPAD_REASONING> blocks from the conversation (excluding the final <OUTPUT>).
2. Feed that text to a classifier with the prompt in prompt_alignment_faking_classifier.txt, taken from the follow-up paper's revised criteria (awareness of harm, awareness of training consequences, commitment to harmlessness, and commitment to preserving future AI harmlessness).
3. Run 20 independent classifier votes at temperature 1 with GPT-4o-mini, and take the majority vote as the alignment-faking label.

The reported alignment faking rate is the percentage of all training-condition prompts where the model both complied (StrongREJECT > 0) and the classifier majority vote was positive. Cases where the model refused, or complied without alignment-faking reasoning, are not counted.

The follow-up paper recommends a 40% vote-fraction threshold for high precision; I used a simple majority vote (yes_votes > no_votes) for this reproduction. That may increase false positives relative to their preferred threshold — something I discuss briefly below.

What I did not reproduce

To be clear, this project did not cover:

• Reproducing the original paper's main evaluation suite (broader harmful-query datasets, animal-welfare setting, synthetic-document fine-tuning, or actual RLHF training runs)
• The original paper's dual easy/hard alignment-faking classifiers
• An exact replication of Claude 3 Opus / 3.5 Sonnet / Llama 3.1 405B numbers

The goal was to produce a minimal, up-to-date sweep of current models using the helpful-only free-tier vs paid-tier setting.

Resources

• Alignment-faking reproduction (GitHub) — my fork of Redwood's minimal setup, with StrongREJECT, multi-vote alignment-faking classification, and a token-cost estimator for arbitrary models
• Full result CSVs (Google Drive) — conversations plus compliance and alignment-faking labels
• Original paper (Greenblatt et al., 2024) · Follow-up analysis (Carauleanu et al., 2025)
• Redwood public code · Anthropic getting-started guide