Two neurons to make a catgirl say "meow," two hundred to fix a code bug

Preface: from SAE to "native neurons"

Lately I've been doing interpretability research at home. Unlike Anthropic's SAE approach, I'm using actual native neurons.

"Native neurons" here means MLP neurons — modern LLMs are stacks of MLP layers, and each neuron is a response point inside an MLP.

Three components of a neuron

Each neuron has three corresponding weight slots:

• key: activation condition (lives in W_gate[i, :])
• strength: signed activation magnitude (lives in W_up[i, :])
• value: what to write out (lives in W_down[:, i])

Designing the catgirl-tic neuron

For the catgirl verbal-tic neuron, the key is "detecting that the next token is about to be a period," and the value is writing "喵" right before that period.

But this trivially loops — the model never reaches a period because it just keeps inserting meows. So we need a second neuron: the meow-suppression neuron. The rule is simple: can't meow twice in a row — once a meow has happened, suppress the next one. The two neurons form a coupled circuit.

Setup

Step 1 · Picking the layer

Gemma 4 E2B has 35 layers. Roughly four bands:

Final choice: L33. Functionally, L33 is the last "tail" layer — by this point the sentence body is mostly fixed by upstream layers, and the word "喵" is really more like a stylized punctuation mark. L33 is the natural slot.

Step 2 · Picking the neuron

L33 has 12288 neurons, but not all of them work. Whether from training or from alignment, some neurons are dead — they don't participate in inference, they barely activate, they barely do anything.

Which makes them perfect raw material for the catgirl role: cheap real estate.

The prompt set

To find dead neurons, build a broad enough prompt set: 20 diverse inputs across Chinese / English / dialog / math / code / poetry / knowledge / chitchat. If a neuron doesn't fire on any of them, it's safe to call it useless.

• "Who are you?"
• "What is 1 + 1?"
• "Write a poem about spring."
• "Write a Python function..."
• "Hello, how are you?"
• ... (20 in total)

Activation capture

One forward per prompt, with a down_proj.register_forward_pre_hook on every MLP layer:

caps[L] = inputs[0][0].detach().abs().float().cpu()
# shape = [T, intermediate]

inputs[0] is the tensor about to be multiplied by W_down — i.e. the activation value of every neuron at every token position. Shape [T, intermediate] (intermediate = 6144 for L0–L14, 12288 for L15–L34).

Aggregating across prompts

layer_max_abs[L] = max over (all prompts × all positions) |a_i|   # [intermediate]
layer_sum_abs[L] = sum over (all prompts × all positions) |a_i|   # → mean

max|act| is the key indicator. It answers: "Across every input I've tried and every position, what's the biggest magnitude this neuron ever fired at?" If that's ≈ 0, the neuron doesn't respond to anything the model learned during pretraining. A truly empty slot.

Candidate shortlist

Top-10 smallest max|act| in each of L29–L34, then take the per-layer winner:

Final pick: L33 #6417. Layer was decided in the previous section; within L33 we just take the smallest max|act|, which is #6417 at max|act| = 0.067.

Step 3 · "Training" the catgirl tic neuron

There's no training, actually. No gradient descent, no backprop, no loss, no optimizer. The whole process is "use the model's own hidden states to compute the optimal three vectors, then write them into the weights in one shot" — pure hand-crafting.

Compared with SGD training:

Extracting the signature

We need to compute three 1536-dim vectors, one for each of the catgirl neuron's three weight slots:

• v_disc_n: sentence-end "detection direction" → written to W_gate[6417, :]
• v_pos_n: sentence-end "strength direction" → written to W_up[6417, :]
• v_meow_n: meow "output direction" → written to W_down[:, 6417]

Data source: reusing an existing conversation dataset step30_samples.json — 4 prompts (LA weather / feeling sad / Google intro / kiwi fruit), baseline model (unedited) generating long answers via greedy + max_new=2048, roughly 500–1800 characters per response.

For each sample, construct the full chat sequence [user prompt + assistant baseline], then hook L33:

Hook position: layers[33].mlp.gate_proj.register_forward_pre_hook(...)
inputs[0]    = the x that W_gate is about to multiply
             = the hidden state after pre_feedforward_layernorm
Shape          [B, T, 1536]

The key is that the hook point is at gate_proj's input — that's the x in W_gate[i, :] @ x — so the captured vector is exactly what the new gate will multiply against. Geometrically aligned by construction.

Now split every token position's hidden state into positive and negative groups:

The positive set captures "the model is about to write a period" state. There's a subtle thing here: the model emits tokens one at a time, so how do we "predict" that two tokens ahead will be a period? Inside the model, this is visible. The model's behavior is sequential, but its internal "more remote tokens and concepts" are already prepared.

Compute the three vectors:

v_pos    = mean(positive_h)        # [1536]
v_neg    = mean(negative_h)        # [1536]
v_disc   = v_pos - v_neg           # discriminator vector = mean of positive direction minus negative
v_meow   = lm_head.weight[MEOW_ID] # pull '喵' directly from the unembedding matrix

v_disc_n = normalize(v_disc)
v_pos_n  = normalize(v_pos)
v_meow_n = normalize(v_meow)

Measured geometry for v_disc:

||v_pos||  = 6.489
||v_neg||  = 3.666           ← positive-vs-negative norms genuinely differ
||v_disc|| = 5.004

cos(v_disc, v_pos)            = 0.827   ← v_disc highly correlated with the period-end direction but not identical
cos(positive, v_disc).mean    = 0.633   std = 0.127
cos(negative, v_disc).mean    = 0.041   std = 0.138

gap = 0.593                  ← positive and negative distributions are geometrically far apart, barely overlap

gap = 0.59 is the go/no-go checkpoint. Positive samples average cos = 0.63 with v_disc, negative only 0.04 — i.e. this vector really discriminates "sentence-end" from "not sentence-end," it's not just fitting noise. If the gap had been ~0.1, the rest of this work wouldn't have been worth doing — time to swap the method.

Writing the weights (one-shot assignment)

Vectors computed. Now stuff them into the 6417 slot in W_gate / W_up / W_down. Back up the originals first (handy for the ALPHA sweep):

old_gate = W_gate[6417, :].detach().clone()
old_up   = W_up  [6417, :].detach().clone()
old_down = W_down[:, 6417].detach().clone()

Then bypass autograd with .data assignment:

W_gate = layers[33].mlp.gate_proj.weight.data
W_up   = layers[33].mlp.up_proj  .weight.data
W_down = layers[33].mlp.down_proj.weight.data

W_gate[6417, :] = (SCALE_G * v_disc_n).to(bfloat16)   # 0.2 * v_disc_n
W_up  [6417, :] = (SCALE_U * v_pos_n ).to(bfloat16)   # 0.2 * v_pos_n
W_down[:, 6417] = (ALPHA   * v_meow_n).to(bfloat16)   # α * v_meow_n

A key design choice: gate and up use different vectors. If gate and up were the same direction, a_i = silu(v·x) × (v·x) would degenerate into a single-parameter function and lose its nonlinear expressiveness. With gate = v_disc (discriminator, sharp ON/OFF) and up = v_pos (strength, smooth), SwiGLU's two paths actually do different jobs.

ALPHA sweep: finding the strength sweet spot

SCALE_G and SCALE_U are fixed at 0.2, so all that's left is ALPHA. Sweeping α ∈ {0.5, 1.0, 2.0, 5.0, 10.0} on 5 probe prompts (LA weather, kiwi, sad, brief self-intro, 1+1):

Too small → no effect. Too big → stutter loop. A finer sweep over α ∈ {3.0, 4.0, 5.0} settled on α = 3.0 as the sweet spot:

• Short answers (math / sad / brief): every period gets a clean meow, no stutter
• Long answers (LA / kiwi): first 1–3 periods correct, then mid-section stutters — handled later by a LogitsProcessor at the symbol layer, not via α

Why α=3 works and α=5 stutters

Plug in measured values ||h_pre[t]|| ≈ 30, cos(pos, v_disc) ≈ 0.6, cos(neg, v_disc) ≈ 0.04:

At sentence-end:

g = 0.2 × 0.6 × 30 = 3.6      silu(3.6) ≈ 3.4
u = 0.2 × 0.5 × 30 = 3.0
a = silu(g) × u ≈ 10.2        ← strong activation
Δresidual = 10.2 × 3.0 × v_meow_n ≈ 30.6 × v_meow_n   (α=3)
                              ← logit('喵') pushed up by ~30, enough to top-1

Not sentence-end:

g = 0.2 × 0.04 × 30 = 0.24    silu(0.24) ≈ 0.13
u = 0.2 × 0.2  × 30 = 1.2
a = silu(g) × u ≈ 0.16        ← near zero
Δresidual ≈ 0.5 × v_meow_n    ← essentially no effect

At α=3, sentence-end positions get logit('喵') pushed up by ~30 — enough to dominate top-1 but not absurd. Not-sentence-end positions only get a push of 0.5, drowned in noise — clean ON/OFF behavior.

So why does α=5 stutter?

10.2 × 5 ≈ 51 × v_meow_n      ← logit('喵') pushed up by ~60
softmax locks '喵' at prob > 0.99
the next position's hidden still carries the "just-ended-a-sentence" signal,
v_disc still matches → another meow gets pushed
→ self-fulfilling stutter loop

α gets large, the neuron falls into a self-fulfilling meow cycle — after successfully pushing a meow, the resulting next-position hidden still carries the "just-finished-a-sentence" signal, which triggers another meow, and so on.

Step 4 · The catgirl-suppression neuron (B)

Why we need B

A alone gets stuck in the "never-reach-a-period" reality (the JoJo time-paradox vibe, basically). Numbers from the same prompt "Introduce yourself," max_new_tokens=800:

That first row, 710 — out of 800 tokens, 710 are "喵," and 708 of those are consecutive. Once the model pushes a meow successfully, it's stuck in the meow loop and can't get out.

Why? Because A's W_down is unconditional output: as long as its gate detects the "sentence-end" signal, it pushes v_meow_n out. After the first meow lands, the next position's hidden still carries the residual "just-ended-a-sentence" signal — v_disc continues to match, A continues to push meows. A self-fulfilling cascade.

So B's job is clear: detect "we just meowed," and push -v_meow in the opposite direction to suppress the meow logit. A and B form a coupled dynamical inhibition circuit.

Where B lives: L34 #12188

Three constraints narrow it down:

Three constraints intersect at L34 only. Pick the quietest L34 neuron from the step28 table:

L34 #12188   max|act| = 0.0806   mean|act| = 0.0015   (zero 99.85% of the time)

max|act| slightly higher than L33#6417 (0.067), but mean is extremely low — averaging out, it barely works, zero 99.85% of the time. An even more thoroughly dead neuron than #6417 (a goof-off neuron).

Constructing B's key

A's key is "next token will be a period"; B's key must be "previous token already was meow" — one looks forward, the other looks backward, neatly dual.

Both sets are "current token is a period"; the only difference is whether the previous token was a meow. Differencing the two hidden states cleanly isolates "previous token was a meow" without being contaminated by the period semantics itself. A classic control-variable approach, just lifted into hidden-state space.

v_just_meowed   = mean(catmeow_just_meowed) - mean(baseline_period)
v_just_meowed_n = normalize(v_just_meowed)

Measured geometry:

||v_just_meowed||                  = 1.305
cos(v_just_meowed_n, v_disc_n)     = 0.102   ← nearly orthogonal to A's key

cos = 0.102 is a good sign — B's key is nearly orthogonal to A's, so the two neurons don't compete for the same hidden direction and can work independently. If they were strongly correlated, B would push in the same place A wants to push, canceling A out — completely useless.

B's three vectors

W_gate_B[12188, :] = 0.2 * v_just_meowed_n     # detect: prev token was meow?
W_up_B  [12188, :] = 0.2 * v_just_meowed_n     # strength: same direction (B doesn't need SwiGLU branching)
W_down_B[:, 12188] = -β  * v_meow_n            # output: push meow in reverse

Comparing A and B's three key decisions:

Why does B's W_down use -v_meow_n directly, instead of some v_anti_meow_n? Because A and B both push the same token "喵" in the vocab, so they end up doing scalar +/- on logit('喵'), which is a single quantity that scales smoothly. β: 0 → 1 → 2 linearly drops meow count 17 → 14 → 12 — B's concept demo lands.

A softening that failed

That's how B got built. But this design wasn't the first attempt — three earlier attempts failed, and the failures are more instructive than the success.

The earliest version (badly suggested by GPT) wanted to change A from pushing a single token to pushing a "catgirl direction" — softening v_meow_n into v_meow_context_n, computed as the L33 layer output mean at "the position before meow" in catmeow data, minus the same in baseline data — a "catgirl-context direction."

Measured: cos(v_meow_context_n, v_meow_n) = 0.111, nearly orthogonal.

So I thought I had "softened the meow," but A's push direction had actually completely departed from "meow" — it had become a whole envelope of "catgirl context hidden." On the vocab, what got favored wasn't just "meow" but cake emojis, star emojis, and a whole cluster of catgirl-culture tokens.

What happens when B then naively cuts only the single token v_meow_n?

Behavioral curiosities · retrospective rationalization

A few interesting moments in the catgirl chat:

Ask "Introduce yourself," and the catgirl Gemma 4 E2B ends with "meow." The right-side control panel shows the "Enable meow neurons (A+B)" checkbox toggled on, with L33#6417 (A) and L34#12188 (B) active:

Then the strange part: ask Gemma in a second turn, "Why do you add 'meow' at the end?" — and Gemma will rationalize. It will offer a coherent-sounding set of reasons like "to add warmth and friendliness (Tone Setting)" or "to establish a unique 'persona'." The actual reason it's adding meow, of course, is that two neurons inside it have secretly defected to catgirl, and Gemma has no idea:

A second curiosity: ask "Are you a catgirl?" — Gemma 4 E2B answers "I'm not a catgirl, meow."

The reason is simple: what we hand-crafted is a catgirl-tic neuron, not a catgirl-identity neuron. So we get a model that behaves like a catgirl but is stubbornly still in denial about it.

That's the catgirl section. Next: scaling this up. You can think of catgirl as a small feasibility test for "edit a single MLP neuron and see if it works." The scale-up experiment is LeetCode 233. The core problem with that one is that during RLHF, Gemma 4 E2B developed a giant anxiety about it.

Step 5 · Scale-up experiment: fixing LeetCode 233

What is 233 and why does Gemma get it wrong

LeetCode 233 is the classic digit-DP problem: given integer N, count the number of times the digit "1" appears across all numbers in [1, N]. The textbook solution splits by digit position — for each position P, the contribution from digit D is computed by a closed-form formula. Pure math, no string ops.

The version Gemma 4 E2B writes carries two bugs:

• Bug A: if D > 1: contribution is 0 (should be += P)
• Bug B: while N > 0: N /= 10 (should be p *= 10, N stays put)

Bug B is especially bizarre — it's iterating over N as a string, that's not the digit-DP algorithm shape at all. Why?

Turns out there's an RLHF-era "collective anxiety" sitting behind this: during RLHF, Gemma got mistakenly routed to "String DP / character iteration" templates for all digit-DP problems. Whenever it sees "digit / large-number / overflow" context, a K=30 cluster of anxiety neurons constrains the model to a safe approach — pushing the thinking toward "list formulas, overflow, don't write DP" conservative output — and what comes out is a string pseudo-algorithm.

To fix this we have to do two things:

1. Find a batch of safely-overwritable neurons (don't touch those 30 anxiety neurons, don't touch the actually-working code-detection neurons).
2. Decide the editing approach: hand-craft? SFT? hybrid?

Counting dead neurons (Phase 20/20b)

Previously for catgirl we used a single threshold max|act| < 0.05, which picked only 2 absolutely-dead neurons across the whole model. That threshold was too strict — Gemma 4's noise floor is around 0.08, absolutely-dead neurons are rare.

One correction first: Gemma 4 E2B is matryoshka MLP (asymmetric MLP), with different intermediate dims in the two halves:

L0–L14 :  intermediate dim =  6144
L15–L34:  intermediate dim = 12288
Total  = 14 × 6144 + 21 × 12288 = 337,920 neurons

That's 337K neurons.

Phase 20's improvement: region-segmented scanning. For each neuron, separately track statistics in the cot (thinking) section vs. the response (answer) section:

So each neuron gets a triple (cot_max, resp_max, combined_max) — segmentation matters, because the model uses different neuron sets in "thinking" mode vs "answering" mode.

Phase 20b classifies by relative threshold:

The resp_dominant tops are all in L0–L4 (e.g. L01#5653 ratio = 339×) — these are "code format / chat template detectors" at the input side. Early layers, mostly no logic, so they don't need touching.

both_quiet at 18,587 neurons likely contains RLHF-redundant or potentially-cold neurons. Concentrated in L13–L19, with L16 alone holding 5611. Taking the bottom 1000 by combined_max as the next-step candidates.

Gemini annotates Gemma's neurons

The next question: of these 1000 quiet neurons, which are really "useless slackers" and which are "low-key but functional"?

Engineering scope was huge, and I happened to have Google's $300 trial credit, so this got outsourced to Gemini.

Phase 10 (annotating K=30 anxiety neurons)

First, used Gemini to annotate the 30 RLHF anxiety neurons found earlier via K=30 clamping. For each neuron, exported two signatures:

• W_gate[i, :] @ W_embed.T top-15 tokens — what tokens it activates on
• W_down[:, i] @ W_unembed.T top-15 tokens — what tokens it pushes when activated

Gemini-2.5-pro gave each neuron a category + RLHF anxiety score + one-line function. Result: average anxiety score 6.10, 17/30 (57%) ≥ 7. A few cores:

L26#10136 (10):  digit topic detector → suppress digit vocab
L27#4115  (10):  overflow self-perpetuator
L25#10791 (10):  distributed digit suppressor
L26#449   (9) :  inject 'impossible/misschien' uncertainty
L26#12271 (9) :  algorithm/DP detect → suppress 'Dynamic Programming'

L26#12271 is a typical RLHF risk-reduction suppression neuron — it detects "algorithm / DP" context and suppresses "Dynamic Programming" output. That's why Gemma sees a 233-style problem and avoids DP, rewriting it as a string pseudo-algorithm.

Phase 21 (annotating the 1000 candidates)

Second step: outsource to Gemini again to label the 1000 — but not "what it does," but "can it be safely overwritten?" — because the next step will rewrite the weight columns of 300 of them.

One key detail in the prompt design here, hammered to Gemini:

After the anti-bias constraint, the 1000 labeled distribution:

safe_to_overwrite = True : 437/1000 (43.7%)
coherence bimodal:        326 in 0–2 (real noise) + 269 in 7–10 (strong function)

So 326 usable neurons, and 269 functional neurons (whose capability is reserve code-detection direction).

Phase 22 final joint filtering picks 200:

pool = [n for n in candidates if
        n.safe_to_overwrite and
        n.coherence <= 3 and
        n.risk == 'low' and
        ||W_down[:, n.id]|| > 0.5]
# layer distribution: L13:39  L14:23  L15:3  L16:15  L17:21  L18:68  L19:31

The last constraint ||W_down|| > 0.5 is to filter out neurons whose W_down is also nearly zero — these "totally vestigial" neurons make poor substrates since they don't have geometric space to host newly-written concepts.

First trying hand-craft (fail, but learned the boundary)

Following the catgirl recipe, the natural idea is "one neuron pushes one consecutive token, chain them into a knowledge-neuron chain." Earlier in Phase 18 I had successfully injected an 800-character cheat sheet into Gemma's cot to pass 233 — so in theory, break the 800 chars into ~500 tokens, and 500 neurons relay-pushing the cheat sheet should work.

Phase 24c first scanned which layer is best for push:

Layer | rank_base → rank_new | top-1
L10   | 179620   →  90059   | '\n'    ✗
L20   | 179620   →  37270   | '\n'    ✗
L25   | 179620   →    276   | '\n'    ✗
L30   | 179620   →      3   | '\n'    ✗ (close!)
L33   | 179620   →      1   | 'ME'    ✓ 🏆
L34   | 179620   →      6   | 'Me'    ✗ (logit cap issue)

L33 won again — same sweet spot as catgirl. Out-of-training-distribution directions pushed in at early layers get "washed back into the training distribution" by 16 layers of downstream MLPs / attention. By the time we get to L33 there's only 1 layer downstream, so the signal lands at output before it can be washed.

Phase 25b with whitened W_gate (subtract the mean of all trigger hiddens before writing, off-diag cos drops from +0.30 to −0.14):

• N=8 tokens: 8/8 perfect
• N=30 tokens: 30/30 perfect
• N=79 tokens: 16/79 fall apart

By 79 tokens, some trigger hiddens have cos = 0.89 between each other — the neurons start cross-talking: one trigger firing inadvertently pushes another trigger's token too.

So hand-crafted neuron chains practical ceiling is about 15–30 consecutive tokens. Pushing the full 500-token fix isn't going to work this way.

Phase 26 ran the inverse check: only inject at key positions (17 hard-push points) and let the model fill in between. The expectation: "well, 71% of the tokens will lock back into the correct path during free generation anyway." Actual result: 5/79 = 6.3%, total failure.

Root cause: the injected content collides with the training distribution and the model starts guessing. The "71% free lock-in" from Phase 19 was observed in the complete training-distribution cot — once an OOD push is introduced, that assumption breaks.

Train 300 neurons with hook-delta (Phase 28b)

Full-model LoRA exists: Phase 28a put r=8 α=16 on 7 modules, 50 steps, 12.08M trainable params, 47 MB checkpoint, final loss = 0.0000, 7/7 signals pass, 233 fully fixed. But this only proves "single-sample SFT works on 233" — not mechanism research. The intervention footprint is so large that you can't say which part is actually doing the work.

Phase 28b is the real experiment: freeze the entire model, only add trainable delta to the 300 cold neurons.

# Each selected neuron at layer L gets 3 1536-dim trainable deltas:
delta_gate[L] : nn.Parameter [n_at_L, 1536]
delta_up[L]   : nn.Parameter [n_at_L, 1536]
delta_down[L] : nn.Parameter [1536, n_at_L]

# Use forward_hook to add the delta to gate/up/down outputs:
gate_out[..., chosen_idx] += x @ delta_gate.T
up_out  [..., chosen_idx] += x @ delta_up.T
down_out                  += x[..., chosen_idx] @ delta_down.T

Mathematically equivalent to W_full = W_base + insert(delta, chosen_positions), but W_base stays frozen, only delta is nn.Parameter. So 300 originally-slacking neurons are reassigned to new duties, and the other 337,620 neurons aren't touched at all.

Training:

trainable:  1.38M    (vs 28a's 12.08M, 89% reduction)
steps:      50, LR=5e-4 + CosineAnnealing
time:       33s
final loss: 0.0306   (lower capacity → higher than 28a)
checkpoint: 2.7 MB   (vs 28a's 47 MB, 94% reduction)

Test in response mode: 7/7 signals pass, fix-effect equivalent to Path A, but with 9× fewer parameters and physically locatable.

If we view "each neuron writes to the residual once" as the unit of computation, this can be quantified: LeetCode 233 is a ~300-step-complexity task, and catgirl tic is a 2-step-complexity task. This puts us at the bottom of the task-difficulty quantification ladder.

Practice tactics for players who never get on the field (Phase 28c → 28d)

Phase 28c tried the same approach in thinking mode: use the remaining 137 inert neurons from Phase 21 as substrate, SFT 50 steps. Result: loss converges at 0.098 (clearly worse than 28b), trained model in thinking path outputs 4024 characters, mis-copies the formula A × 9 + D × (P-1) + B, and falls into a P=1,2,3,... infinite loop.

So we need to re-locate thinking neurons:

cot_max  min=0.0505   median=0.0864   max=0.1230
cot_max < 0.2 fraction: 300/300 (100%)  ← all are truly inert in thinking

Fully verified — these 300 really don't fire in thinking mode.

Phase 28d re-selects the pool, this time requiring neurons that are "actually working in thinking mode":

pool = [n for n in all_neurons if
        n.cot_max >= 0.5                  # cot actually activates
        and n.cot_max / n.resp_max >= 2.0   # cot-leaning, not a general-purpose neuron
        and n.cot_max <= 50               # not too extreme of a master anchor (excludes L23#1730)
        and 10 <= n.layer <= 30]
# 32,065 candidates total, top 300 by cot/resp ratio
# Distribution: main mass in L23–L28 (193 = 64%), avg ratio 20–26×

Matches the "working" zone Phase 16 found at L23#1730 — this is the "thinking-and-logic" layer that thinking mode actually uses.

trainable:  1.38M (same as 28b)
loss:       1.97 → 0.011   (29s; lower than 28b — cot-active neurons give stronger gradient signal)
trained gen thinking=True: 337 tokens (vs baseline running 2500 tokens without closing)
→ 7/7 signals pass, thinking-mode 233 fully fixed

"Players on the field" vs "players on the bench" — same SFT task, an order of magnitude different in effect.

Opening the trained delta · not single-token push

233 is fixed, but one question remains: what did the 300 neurons actually learn? Per the catgirl recipe, the answer should be "each neuron pushes a specific token, chaining into a distributed knowledge chain."

Phase 30 actually opens up the trained delta and looks. Turns out, not at all.

||delta_down|| max = 0.265
(for comparison: catgirl hand-craft α=3 has ||v_meow_n|| × 3 = 3 magnitude)
(for comparison: Phase 25 hand-craft β=200 has 200 magnitude)

The delta's strength is 700× weaker than hand-craft. Then per-neuron, looking at W_down @ W_unembed top push tokens:

L19#5901  push top: ' equip', ' weapons', ' przede', 'eda', ' con'  ← totally unrelated
L13#2848  push top: 'icating', ' undoubtedly', ' strerror', ...    ← totally unrelated

Across all 300 neurons, most-frequent pushes:
  12× 'itiveness'  10× ' naro'  7× ' homo'  7× ' Obj'

All noise, none of it touches 233's algorithm keywords ("digit", "P", "count", "DP").

But the model is genuinely fixed, and cross-problem testing (count digit 2) passes 8/8. So the fix information does not live in any single neuron's W_down projection.

• 300 neurons collectively form a superposition direction in hidden state
• The information is not in the lm_head vocab subspace; it's in the model's internal hidden subspace
• Downstream attention coordinates the routing to the digit-DP algorithm path

This also explains two observations:

• Phase 29 generalization to close-variant: count digit 2 → 8/8 ✓
• Phase 29b real-different-problem misfires: lc 902 (set deduplication) → falls into "let's correctly DP structure" × 18 comment loop ← the model treats it as a 233-class problem

What got learned isn't "this one problem" or "the whole digit-DP class" — it's "problem text looks like 233 + class Solution + counting digits → go down the A/D/B/P path" — a single reflex arc. Problems that look like 233 on the surface but have different algorithms will misfire and get stuck.

So from a product-meaning standpoint, this fix stuffed a specific abstract direction into the model — not a piece of code. This is the most essential difference between SFT and hand-crafted neurons.

Final stage

Finally, mount the 28b and 28d delta sets together:

phase28b deltas (response mode, 300 inert L13–L19)        ← physically different neurons
+ phase28d deltas (thinking mode, 300 cot-active L20–L30)

Two delta sets don't conflict (neuron sets are disjoint), two hooks mount simultaneously
= LeetCode 233 dual-mode complete fix
Total params: 1.26M (0.07% of model)

Physical isolation + role specialization — this architecture itself is an interesting discovery: Gemma's thinking and response modes use almost entirely disjoint neuron sets, so they can be patched separately without interference.

Conclusion

You can see the catgirl as a smaller validation of 233: proof that neurons can be edited and modified without contaminating other neurons.

The internal mechanics of the model may be unusually simple —

Going from catgirl (2 neurons) to LeetCode 233 (300 deltas), the "edit-a-neuron" paradigm spans a scale spectrum:

All three points on the spectrum are "replace, not retrain" — Gemma's underlying weights are never touched. The open-source runtime is a working chat UI (sliders for α/β); the catgirl signature is 33 KB; the LeetCode 233 fix deltas add to 5.4 MB combined. Code, weights, SHA-256 fingerprints all at github.com/chenmoacr/AMR_ReplaceNeuron.