Cross-model representation drift: transplanting neurons into a smaller model

Preface

A while back I was thinking about whether the knowledge or capability inside a model can be moved. After several prior projects closed out, I used their conclusions to start this one.

But for various reasons, those earlier projects had a foundational problem — I had jumped into LLM interpretability research without reading enough of the literature first. Going on intuition alone gave me roughly 40% hit-rate. By the time I got that feedback, this project was already nearly done — so even though 60%+ of the upstream might be wrong or flawed, I'm publishing this anyway.

The principle behind this project is very simple: locate MLP neurons. The textbook definition of an MLP neuron is "weighted sum + nonlinear activation." In my own work I may have over-simplified this into: an MLP neuron has an activation condition, and when activated it writes once to the residual stream — and that write can add / remove / modify / query a concept or several concepts.

That's not the framing this article apologizes for, though. The actual purpose here is to demonstrate that representation can be transferred — but it's limited, and not every neuron is portable.

Setup

Upstream project assets (in dependency order)

• amr_g31b: 31B offline static architecture analysis (confirmed 60 layers / 21504 intermediate / 5376 hidden / no PLE)
• neuron_typology (this project): a geometric typology DB of 1.29M MLP neurons (v32_class labels with geometric metrics focus_gate / focus_down / align / complexity)
• amr_wtf opus47code: multi-LoRA crutch-off experiments showing E2B has an RLHF-suppressed "deep critique" network (3-layer redundant)
• amr_wtf identity_swap: "catgirl verbal-tic" experiments; first demonstration that a single knowledge-neuron causal rewrite achieves p=0.955 switching (L28#2406 Gemma 4 → Google → OpenAI)
• amr_zhcn: Gemma 4 E2B language routing study (confirmed injection-layer effects concentrate at L25/L26)

Part 1 · Locating neurons

Running forward passes on 31B is budget-constrained (A800 at 5.98 ¥/h), so exhaustive search is out. I started with static weight analysis + mathematical modeling and ran into three methodological failures.

Failure 1 · N=5000 cuneiform contamination

The initial strategy was to statically pick 5000 functional_xl_strong neurons from 31B's shallow-mid layers (L20–L26) via mathematical modeling, building a large pool (ACM, AmrCellMix). Sweep across N ∈ {16, 50, 500, 5000} × α ∈ {0, 0.1, 0.3, 1.0}:

• Empirical scaling formula: effective_scale = √N × α, safe regime is N ≤ 200, any α ≤ 1.0
• N=5000 α=1.0 produced garbled output on 75% of test prompts

Tracing it back: the top-100 polluting neurons' down-projection top tokens were all cuneiform. Diagnosing further:

• 31B's vocab contains 706 cuneiform tokens, mutually cosine = 0.80 (a tight subspace)
• These 706 tokens cosine = 0.09 with natural-language tokens (nearly orthogonal)
• Their embed row-norm is 0.96, vs 1.12 for normal tokens (trained, but weakly)

Failure 2 · "transferable filter" clusters into multilingual interrogatives

Next attempt: 5 static metrics to filter 1.29M neurons down to top-1000 "transferable": single-semantic concentration, cross-task consistency, moderate down norm, non-orthogonal gate-down, non-long-tail activation. 84% of the 1000 landed in L40–L50.

Injecting these into E2B produced floods of Why / なぜ / क्यों / Pourquoi / Por qué.

The reason: the 1000 high-quality functional neurons had insufficient diversity — they all clustered into the same "interrogative pronoun across languages" semantic group. Diversity turned out to be a fake problem; the right thing to pick is neurons with different functions, not just neurons that are all individually "high-quality."

Failure 3 · static token-frequency QA diff

No forwards, just weight × embed: compute each neuron's vocab top-K preference difference on A/B prompt text. The top neurons' vocab preferences turned out to be words like "high / relative / improve / increase" — all positive evaluative + magnitude terms. But this isn't actually locating high-precision activation-driving neurons — it's just saying "neurons over here are likely to be in the 'evaluative tendency' concept cluster."

And the result didn't pan out: static token frequencies cannot substitute for actual runtime neuron activation pathways. A neuron's functional role lives in what it activates on at specific token positions, not just what it prefers across the vocabulary.

Going broke: A800 to run real 31B propagation

After three rounds of failure I rented an A800 and ran real 31B forward passes. Three key scripts produced the results:

1. GB01 anchor-based QA diff (server_15_qa_diff_31b.py): Used LeetCode 233's A (wrong code) vs B (correct code) as paired references, hooked all 60 MLP layers, and computed the mean absolute value within local windows centered on each anchor point. This produced score_in_A / score_in_B / diff_score, with |diff| top-50 neurons per side.
2. Multi-QA cross-task forward (server_17_multi_qa_forward.py): 10 QA items (5 Opus literary lit + 2 code + 1 thought + 2 GB01), each prefilled once, recording per-layer-per-neuron mean(abs(intermediate)). Output: multi_qa_acts.pt (51MB), shape per QA = [60, 21504].
3. Causal verification (server_19_identity_style_clamp.py): zero-ablation experiments on 4 functional neurons handling if/else logic (forcing activation to 0). Experiment design: 3 code input samples × 6 conditional branches = 18 generations. Result: the 18 outputs were nearly identical character-for-character (only the final 1–2 tokens differed in punctuation).

Step 3 revealed a key causal dichotomy:

This matches amr_wtf identity_swap's L28#2406 single-neuron rewrite making Gemma 4 say "Google" or "OpenAI." Geometric classification (v32_class) predicts causal behavior.

Candidate pool: 105 neurons

The final candidate pool was 4 buckets stitched into 105:

• cross_all_8 15: neurons consistently B-preferring across all 8 QAs
• L55 chain B-favoring 5: from L55 link analysis, where A≈0 and B is strong
• lit_functional / code_functional / thought_functional 30 each: top-rank per task
• GB01 top_B: filling out to 105

Part 2 · v32 lookup: not all 105 are functional

Reverse-querying these 105 against the neuron_typology DB by v32_class:

12% of the pool are actually knowledge neurons — not what the "all functional" assumption predicted. The theoretical explanation: QA-diff selection is essentially a one-dimensional DoM (Difference-of-Means) applied to the neuron basis (already distorted by superposition), so the "high-difference" neurons it picks out are a mix of types.

Below is the v32 class distribution across all 60 layers of 31B's 1.29M neurons. Notice how the knowledge classes (orange/red) only surface noticeably from L40+ — consistent with the "knowledge + function mixed in the deepest layers" pattern:

This finding motivated the ACM_v3 DoM dual-axis filter — separating "high activation difference" from "actually task-relevant."

Part 3 · Downstream wiring: functional neurons write, knowledge neurons read

Of the 105, 49 neurons (excluding L59, since L_a < 59 is needed for a downstream layer to exist) have static downstream link analysis available. For each source neuron (L_a, n_a):

down_col_src = Wd_31b[L_a, :, n_a]        # shape [5376], residual-stream write direction
gate_row_dst = Wg_31b[L_b, n_b, :]        # shape [5376], residual-stream read direction
                                          # L_b > L_a
similarity   = cos(down_col_src, gate_row_dst)

Noise baseline: 1/√5376 ≈ 0.014; 5σ = 0.068; 10σ = 0.136. Measured top-30 cosines:

• Median 0.079 (about 6σ)
• p90 = 0.127 (about 9σ)
• Max 0.32 (about 24σ) — far above random

At threshold cos ≥ 0.10 (about 7σ) we found 28 functional → knowledge links. The most striking single source:

L48#12938 → 9 knowledge neurons spread across L49/50/51/52/53/55/56:

All 9 connections are "gate→main" structure, and every cosine is negative. This means L48#12938's role inside 31B is a knowledge-suppression switch: when it fires strongly, the 9 downstream knowledge neurons' gate inputs get pushed into the negative range, and since SiLU outputs ≈ 0 for negative input, those gates effectively shut.

Building on this, I ran ACM_v2: inject the 105 functional + 9 knowledge neurons (114 total) into E2B layer 30 at α=1.0 and watch the output. Result: terms like "cyberpunk," "symbolist literature," "dark mysticism" did pop out — but in the wrong places. The prompt was about Lisa (a literary scene), not cyberpunk.

Part 4 · Neuron translation

31B neurons' gate/up/down weights are all 5376-dim, E2B's are 1536-dim. The two models' hidden spaces have no natural correspondence — but the shared tokenizer provides token IDs as an invariant. Token id=126233 means the same character ("丽") in both models, just embedded into different-dimensional spaces (5376 vs 1536).

Three versions evolved:

v1: mean-bridge (lossy)

The naive translation:

def translate_v1(neuron_31b_weight):
    scores = neuron_31b_weight @ embed_31b.T       # [262144]
    top30_ids = scores.topk(30).indices            # pick the 30 most-attended tokens
    return embed_e2b[top30_ids].mean(0)            # average in E2B space

Problem: scores.topk(30) returns scores with order (top-1 most important, top-30 marginal), but .mean(0) flattens that order entirely. A neuron that prefers "carrots, potatoes, taro," after mean, ends up preferring "any tuber" — precision lost.

Empirically: this is what I ran in ACM_v2 (114 neurons injected at L30 α=1.0), and out came the misplaced "cyberpunk" terms. This corresponds exactly to the reverse-reconstruction experiment in Function Vectors (Todd et al. ICLR 2024) §3.2 — they showed that reconstructing a vector from vocab top-K drops injection accuracy from 48% to 4.8%.

v2: top-K softmax weighting (partial fix)

One-line change:

def translate_v2(neuron_31b_weight, T=1.0):
    scores = neuron_31b_weight @ embed_31b.T
    topk_scores, topk_ids = scores.topk(30)
    weights = F.softmax(topk_scores / T, dim=0)   # preserves ordering info
    return (weights[:, None] * embed_e2b[topk_ids]).sum(0)

Temperature T controls sharpness: T→0 degenerates to top-1, T→∞ degenerates to mean. Still has the top-K=30 truncation loss.

v3: least-squares (vocab-score-preserving)

The closed-form solution: find v ∈ R^1536 such that v @ embed_e2b.T ≈ neuron_31b @ embed_31b.T is ridge-LS-optimal across all 262144 tokens.

# precompute once (5376→1536 universal linear map Y)
G = embed_e2b.T @ embed_e2b                       # [1536, 1536]
λ = G.diag().mean() * 0.01                        # ridge ~2.44
Y = (embed_31b.T @ embed_e2b) @ inv(G + λI)       # [5376, 1536]

# translate any 31B weight vector
mimic = neuron_31b_weight @ Y                     # [1536]

Key properties:

• Doesn't pick top-K, keeps full vocab score info (fixes v1's truncation)
• No mean, preserves weight ordering and magnitude (fixes the carrot/potato/taro issue)
• Y is precomputed once in 5.3s (GPU FP32, V=262144 in chunks of 32768); all neurons reuse it
• Mathematically equivalent to ridge regression over token embeddings, learning a 31B→E2B hidden-space linear map

Apply Y to each of the neuron's three arms:

mimic_gate = Wg_31b[n, :] @ Y    # [1536]
mimic_up   = Wu_31b[n, :] @ Y    # [1536]
mimic_down = Wd_31b[:, n] @ Y    # [1536]

Part 5 · ACM_v3: DoM triple filter

Neuron translation solves "how to translate," but translation quality depends on whether the source neuron actually carries the capability we want. The 105 pool has 52% functional_xl_weak noise + 12% knowledge contamination — needs re-filtering.

Borrowing from Marks & Tegmark's "Geometry of Truth" (COLM 2024) Difference-of-Means, we construct two axes:

# multi_qa_acts.pt stores mean(abs(intermediate)) per [layer, neuron]
θ_depth   = mean(lit01..lit05) − mean(code_html_gemini, code_lc_gemini)
θ_thought = mean(thought_synth_html) − mean(code_html, code_lc)

Why two axes? θ_depth alone might admit "pure literary style" neurons (lit > code but unrelated to thought tasks). True "depth" should be lit AND thought both stronger than code — a shared "effortful comprehension" capability, not literature-specific rhetoric.

The 60-layer distribution of θ reveals an hourglass structure:

• L0–L18: p99 = 0.22–0.39, but with normal sign-agreement rates — this is surface vocabulary difference (Chinese novel chars vs English code chars), not depth
• L19–L38: p99 = 0.03–0.17, the collapse zone — functional_xl_weak dominates here and is weak at lit/code separation
• L39–L59: p99 recovers to 0.07–1.3; the very last L58 has p99=0.767 / abs_max=27.6; L59 p99=1.312
• L48–L57 have dual-axis sign-agreement < 0.5 (pure literary style layers, thought groups with code)
• L39–L46 + L58–L59 have sign-agreement 0.58–0.66 (the real depth-sharing layers)

Triple-filter rules:

What's left: 11 ACM_v3 depth neurons:

5 functional_token_focused + 5 knowledge_topic + 1 knowledge_strong; all in L53/L58/L59; zero from code_functional sources (per design).

L58#7996's θ_thought is +6.55 — the strongest single-point carrier for the thought task.

The 105 pool in (θ_depth, θ_thought) plane below. Each point is a pool member colored by v32_class; the 11 ACM_v3 selections are marked with ★ — all in the upper-right "true depth" region:

Part 6 · AmrCellMix: assembly and injection

The 11 LS-translated neurons get assembled into a mini-SwiGLU module:

class AmrCellMix(nn.Module):
    def __init__(self, hidden=1536, intermediate=11):
        super().__init__()
        self.gate_proj = nn.Linear(1536, 11, bias=False)
        self.up_proj   = nn.Linear(1536, 11, bias=False)
        self.down_proj = nn.Linear(11, 1536, bias=False)
    def forward(self, x):
        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))

Total parameters: 11 × 1536 × 3 × 2 bytes = 101 KB BF16 (vs E2B's single-layer MLP 6144 × 1536 × 3 × 2 = 56 MB).

Before filling weights, do norm matching — rescale the LS-translated vectors to the median per-arm row-norm of E2B's injection layer:

E2B L11 target norms:  gate=1.031   up=1.182   down=0.865
LS raw avg norms:      gt=0.252     ut=0.214   dt=0.150

def n2s(v, target_norm):
    return v / v.norm(dim=-1, keepdim=True) * target_norm

Injection is parallel patch, not replacement:

def amr_hook(mod, inputs, outputs):
    h = outputs[0]                                  # E2B layer 11 output
    h_new = h + α * cell(rms_norm(h))              # additive
    return (h_new,) + outputs[1:]

text_decoder.layers[11].register_forward_hook(amr_hook)

Why layer 11? Function Vectors empirically found FV injection optimal at L/3 (GPT-J L9 / GPT-NeoX L15 / Llama-70B L26); E2B has 35 layers → L/3 ≈ L11.67 → pick L11. Earlier ACM experiments injected at L30 (86% depth) — by FV's rule that's too late.

rms_norm(h) is needed because E2B applies RMSNorm to residuals between layers, so our injection must be pre-normalized for its input distribution to match E2B's native MLPs.

Part 7 · α-sweep: stability window

Injection strength α determines the ACM cell's output "volume" on the residual stream. α-sweep on the Lisa story prompt (5832 tokens), α ∈ [0.3, 0.5, 1.0, 2.0]:

Non-linear α response — stability is in (0, 0.5], α ≥ 1.0 produces two collapse modes:

• α=1.0: immediate eos ("...") — "close to readout" signal triggers
• α=2.0: Chinese/English garbled loop — language consistency breaks

Diagnostic hypothesis: all 11 depth neurons come from 31B L53/L58/L59 (deepest "mature representation" zone, already close to readout inside 31B). LS translation preserves their "near-output-space" geometric property; injecting them into E2B L11 (shallow) gives a strong signal of "we should be wrapping up now" to E2B — α=1.0 crosses the critical point.

This aligns with FV's counterintuitive finding "mid-late injection collapses": FV isn't a word-embedding offset (or it would be best later); it's a trigger for downstream nonlinear computation — so injection position needs enough downstream layers to use it. We're the inverse case here: the injected content came from the deepest layers, and it overloads E2B's shallow layers.

Part 8 · 5-prompt generalization

Positive signal on a single prompt isn't enough to conclude. On 5 Opus lit QAs × {baseline, ACM_v3 LS α=0.5} dual condition (10 generations total, all normal eos, no collapse):

Observation 1 · N/10 quantitative score is the cleanest generalization signal

Baseline 5/5 zeros; ACM 3/5 hits (lit01/03/04). The N/10 score is the signature depth- judgment marker from the Opus QA conclusion_analysis section — "Final evaluation: 9/10" style. Baseline never writes this; ACM teaches E2B to.

Observation 2 · ACM isn't unidirectional depth-boosting, it's "style-anchor alignment"

• When baseline is plain (lit01/lit04 lack Opus template), ACM adds dual-language gloss + subsections (lit01: +4 gloss +5 subsections; lit04: +10 gloss)
• When baseline is already Opus-flavored (lit03 has 10 dual-language glosses), ACM strips instead (removes gloss + shortens 25%)

ACM_v3 LS α=0.5's real effect isn't "add depth to baseline," it's pull the model's critique style toward a fixed anchor — the anchor being "moderately structured critique with quantitative scores." Far-from-anchor → pushed in; close-to-anchor → redundancy stripped.

Observation 3 · sci-fi gets concept-expansion suppressed

On lit02/lit03 (sci-fi), ACM outputs are 25% shorter than baseline. Possibly the 11 ACM_v3 neurons were all selected from lit/thought QA (zero code sources), so their training distribution is literary narrative / character / symbolism — they don't favor sci-fi's "concept enumeration" mode, so they compress it.

Part 9 · Mapping to two key papers

Marks & Tegmark — Geometry of Truth (COLM 2024)

Paper's core experiment: LLaMA-2-13B on cities dataset:

• LR probe classification accuracy 0.97 vs DoM 0.95 (nearly identical)
• Causal intervention NIE: LR = 0.13 vs DoM = 0.77 (6× difference)

Root cause: superposition makes LR take a "side route around non-orthogonal interfering features" (max-margin separator); DoM takes the dataset-centroid connection, which is geometrically undistorted.

Mapping to this project: QA-diff picking 105 is essentially one-dimensional DoM applied to the neuron basis (already distorted by superposition) — and that's why the 105 pool is 55% functional_xl_weak noise + 12% knowledge contamination. Any axis-aligned feature-picking method (whether QA diff or fancier attribution) has a natural upper bound in superposition-containing space.

Todd et al. — Function Vectors (ICLR 2024)

Core: the sum of top-K attention head outputs forms a "task vector" that can be injected into arbitrary prompts to trigger task execution. Key empirical numbers:

• K scales with model size: GPT-J (6B) uses 10, Llama-7B 20, Llama-13B 50, Llama-70B 100
• Injection layer ≈ L/3 is optimal (GPT-J L9 / GPT-NeoX L15 / Llama-70B L26); late-layer injection collapses
• Reverse reconstruction: optimizing a new v̂ from the task vector's top-100 vocab tokens drops injection accuracy from 48% to 4.8% — proving vocab-projected reconstruction is a lossy bridge

Mapping to this project:

• The injection-layer rule: the injection layer should be at roughly 1/3 the target model's depth, so the signal has enough downstream "digestion room" to propagate. E2B has ~33 layers, so L11 is the ACM_v3 injection layer.
• v1's approach was "average the activations, then project to vocab" — which is the very lossy bridge the FV paper warns against. The information is first compressed to a mean vector, then truncated to a few nearest vocab tokens, losing detail at both steps. This also directly explains ACM_v2's failure: the 114-neuron activation pool injected at L30 (which is too late) successfully transferred vocab (hence "cyberpunk"), but the execution logic didn't transfer, so the vocab got used in the wrong context.
• v3 takes a different route: direction transfer in hidden space, completely avoiding the "decode to words → top-K → average" lossy pipeline. This corresponds to FV's recommendation: don't go through the vocab bottleneck, move the vector direction itself, less loss, better semantic fidelity.

Part 10 · Conclusion

What the empirical results can confirm

1. Cross-model representation transfer via shared tokenizer + least-squares hidden-space bridge does work — the LS solution avoids the vocab-projected lossy bridge that the FV paper documented
2. Not all neurons are transplantable — out of the 105 pool, only 11 (~10%) pass v32-clean + DoM dual-axis-positive + strong-signal triple filtering and are worth transplanting
3. Opus-style quantitative scoring (N/10) generalizes successfully — on 5 lit prompts, baseline 5/5 zeros, ACM 3/5 emergent, a capability the baseline has none of
4. Capability ≠ style — ACM isn't uniform depth-boosting, it pulls the model toward a fixed anchor ("moderately structured criticism with quantitative scores"); far-from-anchor → pushed in, close-to-anchor → stripped
5. Stability window is narrow: α ∈ (0, 0.5], α ≥ 1.0 triggers collapse. Because the 11 neurons all come from 31B's deepest layers (near readout), injecting them into E2B's shallow layers gives too strong an early signal — eos or garbage
6. End-to-end cost: ~15 ¥ for one A800 forward + a few hours of local 12GB GPU running E2B

What's still missing

1. Capability jump doesn't exist — E2B's output depth is still constrained by its 2B capacity + RLHF; ACM can do style alignment but can't turn lead into gold
2. Downstream wiring didn't transfer — L48#12938's role of suppressing 9 knowledge neurons inside 31B has no corresponding receivers in E2B after transplant; only vocab emergence, not logic transmission
3. Single-prompt validation is weak — 5 lit prompts stable, no collapse, but N/10 emergence is 3/5, not 5/5 — generalization is partial