Language-specific neurons sit in the last four layers of a 284B model

by onblueroses, co-authored with claude · Aug 2, 2026

A multilingual model has to track which language it is reading, and the machinery that does this can spread through the depth of the network or sit in a narrow band of layers. I measured which, in DeepSeek-V4-Flash, an open-weight Mixture-of-Experts model with 284 billion parameters, 13 billion active per token, and 43 layers.¹ I ran Tang et al.'s LAPE probe over seven languages, then recomputed the selection across twelve threshold and filter settings. At the standard setting, 90,469 of 179,016 selected cells (50.54%) sit in layers 39 to 42 of 43. Normalized against the cells each setting makes eligible, the last four layers are over-represented by 1.9x to 7.0x, in all twelve settings.

Method

A Mixture-of-Experts model sends different tokens to different sub-networks. In V4-Flash, layers 0 to 2 route each token by a hash of its input id, and layers 3 to 42 route it by a learned gate to 6 of 256 experts, plus one shared expert that sees every token. Each expert has an intermediate width of 2048 channels. The unit of measurement here is one channel of one expert in one layer; I call that a cell, and the same channel index in two experts is two different cells.

The corpus is FineWeb for English and FineWeb-2 for Chinese, French, Spanish, Arabic, Hindi, and Japanese. Each language contributes 98,304 tokens. The activation pass ran on four H200 GPUs and cost about $25 to $30 in total.

LAPE stands for language activation probability entropy (Tang et al., ACL 2024). For each cell and each language, it computes the fraction of that language's routed tokens on which the cell activates, then normalizes those seven fractions into a distribution, takes the entropy, and selects the lowest-entropy cells.²

Tang's released code hard-codes the selection at the bottom 1%, and that number is a setting rather than a result. In all twelve of my settings the selected count matched the requested rate to five decimal places: it is 1.0000018% of eligible cells at the standard setting and 4.9999978% at a 5% rate. The count follows the setting. The shape of the selection is the measurement.

A sparse model needs one filter that Tang's dense setting does not. An expert only sees the tokens its router sends it, so a cell whose expert saw 12 Arabic tokens carries an activation rate with no power behind it. The min_tokens filter keeps a (layer, expert) pair only when all seven languages routed at least that many tokens to it. The standard setting below is a 1% rate with min_tokens at 100, which leaves 17,901,568 eligible cells in the learned-routed layers. It selects 179,016 of them.³

Result: the selection concentrates in the last four layers

Counts by layer band, at the standard setting:

Layers 3 to 22 are half of the learned-routed depth and hold 19.51% of the selection, while layers 39 to 42 are a tenth of that depth and hold 50.54%. The four densest layers are 39 (21,696 cells), 40 (22,317), 41 (24,141), and 42 (22,315); the next largest is layer 38, at 10,562 cells.

The raw share moves with the setting, from 2.62% to 58.36% across the twelve, and that reads as fragile until you count what each setting leaves eligible. Late-layer routing is more concentrated, so late-layer experts fail the min_tokens test first: at min_tokens = 1000, layer 40 has 4 eligible experts of 256 and layer 42 has 4, against 103 at layer 13. The last four layers then supply 1.37% of all eligible cells, so the 6.87% selected there is a 5.02x over-representation.

Every setting points the same way. The weakest enrichment is 1.92x and the strongest is 7.04x; that range, not the 50.54%, is the finding, because it does not depend on either setting.⁴

Result: most tagged cells tag one language

At the standard setting, 143,669 of the 179,016 selected cells are assigned to exactly one language, or 80.25%. The mean is 1.2535 languages per cell. A cell is assigned to a language when its activation probability for that language clears the 95th-percentile bar, and four cells in five clear that bar for one language only. That is what makes the late band read as language-specific rather than as generic late-layer machinery.

This one is weaker under a loose threshold. The multi-language fraction runs from 17.76% to 42.66% across the twelve settings, growing as the rate loosens and reaching 42.66% at a 5% rate with min_tokens at 1000. The single-language reading holds at rates up to 2%, where the fraction stays between 17.76% and 29.33%.

What the data does not support

The 1% is not a measured fraction of the network, and nothing here says that 1% of V4-Flash is language-specific. The threshold produces that count by construction.

The per-language ranking is not stable, so I do not report one: Japanese leads the count in six of the twelve settings and Hindi leads in the other six. That ordering tracks which cells clear the power filter, not the typology of the languages.

The denominator is narrower than "all tagged neurons". The 179,016 cells are the learned-routed selection, and the same run selects 15,688 cells in the hash-routed layers and 819 in the shared experts. Against the combined 195,523, the 90,469 late-layer cells are 46.27%.

This is one run at a pilot token count, with no bootstrap variance on any figure.³ It is also one model. Tang et al. report a comparable concentration in the top quarter of the dense LLaMA-2-7B. This run does not re-measure that result. A claim about invariance across scale needs a re-analysis at matched settings.

Conclusion

A standard entropy probe tags a subset of cells as language-specific. In this model that subset is over-represented in the last four of 43 layers by 1.9x to 7.0x, and the enrichment holds at every rate and power filter tested. Four in five tagged cells carry exactly one language. Layers 3 to 38 are nine tenths of the learned-routed depth and hold the other 49.46%. The reading is that most of the network computes something the seven languages share, and a late band carries the language-specific part.

That is a statement about where the selection lands, not about what the late band does. The test that would settle it is causal. Ablate the tagged cells in layers 39 to 42 and measure per-language loss. Compare that against an ablation of the same number of untagged cells in the same layers. That experiment is not run.

Data availability

reanalysis.json accompanies this note. It holds the power audit, the twelve-setting sweep, the per-layer counts, and the cross-language overlap histogram; every measurement reported here comes from that file or from a recomputation over the raw counters, except where a footnote names another source. The raw per-language activation counters are 173 MB per language, 2.4 GB in total, and are available on request. This probe has no public repository.