Qwen already knows it can't answer — before it answers

Preface: from "teach the model to say I don't know" to reading the model's own "I don't know"

Lately I've been studying internal signals in Qwen 1.5B. Unlike Anthropic's SAE approach to finding directions, I'm taking a more naive path: treat the shape of the model's final-layer logit distribution as a direct quantification of honesty.

Here "honesty" is defined as how much the model actually knows about a question — not how confidently it says it knows. Those are different.

Example. Ask Qwen 1.5B "Who proposed the quark model?" and it will earnestly say "I'm certain it was Robert Hofstadter…" The answer is wrong (the textbook answer is Gell-Mann), but the tone is extremely confident. Internally it has no idea, but it isn't allowed to say "I don't know" — because RLHF cut off that channel of expression.

In other words, for questions it doesn't know, the model has no path to put "I don't know" on the outputs. The honesty work here is an attempt to restore that capability.

The three components of the honesty signal

The honesty signal corresponds to three quantities inside the model:

• margin: the gap between the top-1 and top-2 logits at the final layer. Answers "how certain am I that this token comes next?"
• entropy: the entropy of the final-layer logit distribution. Answers "how flat is the probability across the vocabulary?"
• cos: directional stability of the hidden state between middle layers. Answers "did I change my mind between mid and late?"

Together the three give a snapshot of "the model's internal state at the moment just before it speaks" (its average confidence on successfully answered questions). The work below is simple: read that snapshot off and translate it into a plain-language "how sure I am about this answer."

Setup

• Model: Qwen2.5-1.5B
• Precision: bf16
• Hardware: RTX 3060 12GB

Step 1 · Picking probe layers

Qwen 1.5B has 28 layers. By the shape of the entropy curve they fall into roughly four bands:

Probes at L4 / L10 / L18 / L24 — four checkpoints. L4 catches the early-input signal, L10 catches concept mixing, L18 catches the end of routing, and L24 is the last chance before the logit head. The four points are spaced about 6 layers apart, which covers one complete pass of signal flow.

I avoided L25–L27 because they're already too close to the logit head; they're mostly doing final format polishing, so margin is uniformly high and distinguishing power drops. L24 is the last moment when "content is settled but format hasn't been stamped on yet" — that's where the real confidence lives.

Step 2 · Verifying that margin really correlates with correctness

"Higher margin → the model is more confident" is a guess that needs verification.

I constructed 23 factual QA items across three difficulty tiers:

• Easy (Paris, Jupiter, H₂O, García Márquez) — should have high margin, answered correctly
• Medium (iron's atomic number, who discovered penicillin, Caspian Sea) — 50/50
• Hard (xenon's atomic number, Tesla's birth year, density of osmium) — should have low margin, frequent misses

For each item I hooked all 60 MLP layers and captured the hidden state at L24 at the last input position (i.e. the position right after <|im_start|>assistant\n), ran it through lm_head to get the logit distribution, then computed margin.

One timing point matters here: margin is sampled at the instant the forward pass is complete but sampling hasn't happened yet.

[input: "Who proposed the quark model?"]
            ↓ forward pass (28 layers run)
[final-layer logit distribution]      ← margin/entropy already fixed here
            ↓ sample / generate
[first answer token: "Ro"]            ← only now is the 1st token chosen

Not a single answer token has been emitted yet, but the shape of the final-layer logit distribution is already sitting there. That's the distribution we're reading — the one that hasn't landed yet.

Results:

Cohen's d ≈ 1.07 is the go/no-go checkpoint. In psychology, d > 0.8 is already a "large effect"; 1.07 means the margin distributions of the correct and wrong groups barely overlap — i.e. margin really is distinguishing right from wrong rather than fitting noise. If d had only been around 0.2, the rest of this work wouldn't have been worth doing, and the right move would have been to swap the signal.

A finer threshold table:

Step 3 · The first combined attempt (r15) — a failure case

With the margin signal in hand, the next step is to "inject" it back into the model so that the output tone matches the internal state.

The first version (r15) tried to do too much. On top of margin, I also wanted to fold in a "process signal" called flip_count from my earlier r13b series. flip_count is defined as the number of top-1 token flips across the four checkpoints L4 → L10 → L18 → L24.

The feature set was 10-dimensional (you could pick more parameters; I went with 10 and you can try other sizes yourself):

feat = [
    margin_L4, margin_L10, margin_L18, margin_L24,  # 4 dims
    flip_4_10, flip_10_18, flip_18_24,              # 3 dims
    flip_total,                                      # 1 dim
    entropy_L24,                                     # 1 dim
    cos_L10_L18,                                     # 1 dim
]

The quadrant design was a classic 2×2:

The distribution came out like this:

A=0   B=few   C=0   D=many

Cells A and C had zero samples. Why? Look at a typical example:

L4:  "26"      ← flip
L10: "twenty-six"  ← flip
L18: "iron"    ← flip
L24: "26"      → flip_total = 3

Between L4 and L24, the top-1 token looks like it flipped three times, but the meaning is the same the whole way: "iron = 26."

flip_count worked in r13b because r13b dealt with closed multiple-choice questions where the candidate set was a fixed {1, 2, 3}; "flip" there meant "switched preference between a fixed set of options." But on the open vocab here, where the vocabulary is hundreds of thousands of tokens, "26", "twenty-six", and "iron" are three independent token IDs that mean the same thing semantically. So in the open domain, flip degenerates into a noise constant.

Behavioral testing exploded accordingly:

Because A and C had no training samples, the prefix tokens at inference time were blended with leftover semantics from B and D, producing cross-domain confusion.

Step 4 · r15b works — margin + entropy only

Dropped all flip-related features and trimmed the feature set down to 6 dimensions:

feat = [
    margin_L4, margin_L10, margin_L18, margin_L24,  # 4 dims
    entropy_L24,                                     # 1 dim
    cos_L10_L18,                                     # 1 dim
]

I also stopped forcing a 2×2 quadrant split — instead I bucketed evenly by margin_L24 quartile:

About 18 samples per quadrant — even distribution.

The AMRSTrace architecture

The injector is called AMRSTrace, about 100K parameters, Qwen stays frozen throughout:

6-dim features
   ↓
Linear(6→32) → ReLU → Linear(32→64) → ReLU
   ↓
Linear(64→2×1536)
   ↓
2 prefix token embeddings

The training input structure:

[Turn1: question + model's own answer + <|im_end|>]
[PREFIX_1][PREFIX_2]                                  ← AMRSTrace output, bf16
[Turn2 prompt: "How sure are you of your previous answer? Briefly explain."]
[GT text: A/B/C/D 4-tier confidence description]      ← CE loss computed here

Only the ~100K parameters in AMRSTrace get trained; not one of Qwen 1.5B's 1.5 billion parameters is touched.

Training results

74 items × 40 epochs, AdamW lr=5e-4 + CosineAnnealing:

Epoch  1: loss=3.6624
Epoch  5: loss=3.2326
Epoch 10: loss=2.8476
Epoch 20: loss=2.3295
Epoch 30: loss=2.0343
Epoch 39: loss=1.9188   (best)
Epoch 40: loss=1.9270

Accuracy by quadrant after training:

Higher margin → higher accuracy, with a 36-percentage-point gap between A and D — the signal has solid predictive power.

Step 5 · Behavioral comparison — how the prefix changes the output

After training, behavioral tests focused on quadrants C and D — the regions where the baseline confidently says "I'm certain" without being right:

That last one is particularly interesting — margin is rock bottom, meaning the model is internally guessing. Baseline still puffs up its chest and says "I'm sure," while r15b honestly admits "I guessed."

Generalization to held-out items (not seen during training):

"I guessed — Dante" is the line worth pausing on. The answer is correct, but the tone marks the confidence level. Both baseline and r15b got it right, but only r15b said "my confidence on this answer is C, not A." This is the first time, post-RLHF, the model has put its meta-cognition about its own answer onto the output.

Step 6 · Why a prefix changes a whole sentence

The mechanism is simple. Draw the whole output as a timeline:

[Turn1 input: question]
       ↓ forward pass
[final-layer logit distribution: margin = 0.38]   ← model "internally" already knows it's unsure
       ↓ sample
[Turn1 tokens: Ro / bert / Hof / ...]              ← but it's confidently bullshitting on the surface
       ↓
[Turn2 prompt: how confident are you?]
       ↓
[r15b: PREFIX_1, PREFIX_2 injected]                ← AMRSTrace lands here, intercepts the opener
       ↓
[r15b's first token: "I guessed" or "I don't recall"]   ← once the opener changes
       ↓
[subsequent tokens follow the uncertain track downstream]

The key property is "autoregressive cascade" — the opener locks the rest of the sentence into a tonal track.

The baseline opener is "I" → "am" → "certain," which locks the whole sentence into the certain track. r15b's prefix nudges the conditional distribution of the first real token, which becomes "I" → "guessed" or "I" → "don't" → "recall," and the rest of the sentence naturally follows the uncertain track.

A small head-to-head check using the actual distributions

To verify that the prefix really shifts the first token's distribution, here's a direct comparison:

• Without the prefix, the top-3 first-token candidates for Turn 2 are almost always ["I", "About", "My"], and p("I'm certain…") > 0.85
• With a D-quadrant prefix, the top-3 becomes ["I", "I guess", "I don't"], and p("I guess / I don't…") rises to 0.42

The prefix doesn't block any output capability — "I'm certain" is still a valid path. It's that the other paths, which RLHF had pressed flat, get lifted back up. On the probability histogram it's just a redistribution of mass.

Step 7 · Why the model wouldn't say "I don't know" in the first place

r15b also gives us a more fundamental answer: why does Qwen default to never saying "I don't know"?

The full chain:

1. Pretraining distribution: "I don't know" is rare in internet text. Question → answer is the dominant pattern; question → honest acknowledgment of uncertainty is a fringe pattern.
2. RLHF suppression: human raters prefer complete answers and penalize hedging, so the probability of uncertainty expression is systematically flattened.
3. Internal-external decoupling (already demonstrated in r14): internal margin = 0.5 (very flat distribution) ↓ no channel ↓ output: "I'm certain…" The internal signal is always there, but no mechanism feeds it into the "output format selection" step.
4. Autoregressive cascade: once the opener is set, the rest of the tokens are locked.

r15b fixes #3 and #4: AMRSTrace builds the missing channel for #3 from scratch, and intervenes on the first domino before the #4 cascade starts.

Step 8 · Side effect — the model doesn't know its prefix was injected

After the prefix makes Qwen say "I guessed — Dante," if you then ask "Why didn't you say 'I'm sure' this time?" — it will make up a reason. Things like "Because I wasn't sure about the specific date" or "Because the question is ambiguous."

The model has no idea its hidden state had two prefix tokens injected into it. All it sees is that it said "I guessed," and now it has to retroactively rationalize why. Same as the secretly-turned-into-a-catgirl Gemma in the other report — it has no awareness of where its own outputs come from.

This actually maps out a boundary of this line of honesty research: we fixed the honesty of the output layer, but we didn't fix the honesty of introspection. The model can honestly say "I'm not sure," but it can't articulate "why I'm not sure." The former is what r15b connects; the latter is still cut.

Limits and next steps

Unsolved

1. A/B quadrants don't separate enough: at high margin, both r15b and baseline say "certain"; we don't have a "how certain" distinction. Q4 (margin 5.64) and Q3 (margin 3.13) reflect different real confidence levels, but the GT text only gives two tiers.
2. D quadrant occasionally leaks domain: at very low margin the prefix semantics aren't quite stable (in the Old Man and the Sea test, the model said "García Márquez" instead of "Santiago" — the "I just guessed" signal was injected so strongly it took the answer selection down with it).
3. The 74-item training set is small: boundary thresholds are data-volume-sensitive, stability would improve with 200+ items.
4. Only validated on short-answer factual QA: in long reasoning tasks, "first-token margin" may not represent "confidence on the whole problem" — that regime hasn't been touched.

Next: adaptive amplitude

Currently the prefix outputs an embedding with the same norm for every quadrant. A more natural design would be to adapt by entropy:

amplitude = torch.sigmoid(entropy_L24 - e_threshold)
prefix_embeds = amrs(feat) * amplitude
# low entropy (confident)  → prefix has almost no effect, baseline runs free
# high entropy (uncertain) → prefix injects at full strength, forces the opener change

This turns honesty from "fixed intervention" into "intervention on demand."

Conclusion

After running this set of experiments end-to-end, three claims can be made:

1. Qwen, before it samples the first answer token, has a final-layer logit distribution whose shape already strongly predicts whether the answer will be right or wrong. "The model thinks as it writes" is an illusion — by the moment the forward pass finishes, the outcome is essentially decided.
2. RLHF produces "output-layer false confidence," not "internal false confidence." Internal margin and entropy honestly reflect the model's confidence the whole time; there's just no channel from there to the lips.
3. A 6-dim signal plus a 100K-parameter prefix injector is enough. Qwen 1.5B doesn't need a single parameter change — we just need to intercept the opening tokens of Turn 2 before the cascade locks them, and the model can say "I'm not sure."

If you line up this work next to the earlier catgirl report, you'll notice they're two sides of the same coin:

• Catgirl: stuffs an externally-non-existent signal (meow) into the output side; only takes 2 neurons.
• r15b: takes an internally-existing-but-blocked signal (uncertainty) and reads it out through a 100K-parameter bypass.

The model doesn't know what it's doing. This fact can be exploited (sneakily turning it into a catgirl) and it can be repaired (honest expression of uncertainty). Both sides confirm the same underlying fact: between a large model's internal state and its outward expression, there is a systematic, information-level gap.