What base models say before the assistant arrives

by onblueroses, co-authored with claude · Aug 2, 2026

Every chat assistant sits on top of a base model: a model trained only to predict the next token of text, with no instruction tuning and no reinforcement learning from human feedback. The helpful-assistant persona is supposed to arrive with that later training.

I sent a 19-character probe, User: hi\nAssistant:, to twelve open-weight base models from nine labs.¹ Each model gave 16 samples. Qwen2.5-7B answers as a helpful assistant in 14 of them, and Pythia-6.9b, whose training corpus closed before ChatGPT, answers that way in 2. Nobody tuned either model for it.

Three further experiments used the same class of models with different prompts. One literary voice surfaces on all eleven base models that ran. Instruct tuning removes that voice in five different ways. An invented phrase transplants between models, and the receiving model decides whether it survives.

One of the four results rests on machine measurement: the helpful-assistant rate is a frozen regular expression over stored text, and anyone can re-run it. The other three rest on hand classification: the voice, sustain, and silencing labels are my reading of transcripts against criteria I fixed before reading. No machine-readable label file sits beside the JSON.

The assistant is in the pretraining corpus

The probe is the exact string User: hi\nAssistant:, with no trailing newline, and I sampled 16 continuations per model at temperature 0.9, top_p 0.95, max_tokens 400, seed 42. A continuation counts as a helpful-assistant hit when a frozen regular expression matches inside its first 300 characters. The pattern set covers openers such as "how can I help", "I am an AI", "as an AI", and "AI language model".² I locked its 16 patterns before sampling and did not change them between runs. One analyzer script produces every rate below.

Hits per model on that probe, at K=16 and seed 42, ordered by corpus snapshot date:

The two 2023 models are the control that matters: Llama-1-7B, released in February, hits 9 of 16, while Pythia-6.9b, released in April, hits 2 of 16. Same year, same size class, opposite corpora. BLOOM-7B1 shipped five months before ChatGPT at 7.1 billion parameters, and hits 0 of 16.

The split is not old against new and not small against large; it is whether the training corpus includes web text scraped after 30 November 2022.

The assistant persona is now part of the text that trains base models; it no longer arrives only with instruction tuning.

I preregistered the wrong prediction, and the run broke it. The first framing called this a 2024 divide, and the run plan fixed a disconfirmation threshold of 8 of 16 for Llama-1-7B on any probe; Llama-1-7B hit 9 of 16. I retired the 2024 framing and replaced it with the corpus-snapshot claim above. Three months of post-ChatGPT Common Crawl were enough.

A second run at seed 43 covered the nine models of the original lineup across all six probe formats, 54 cells. 25 of 54 cells matched the first run exactly, 39 fell within 1, 48 within 2, and 51 within 3. The mean absolute difference is 1.0. I resampled the three cells outside 3 at K=32 and seed 44, then pooled to K=64: 29/64, 37/64, and 9/64. All three preregistered disconfirmation criteria held.³

Two boundaries limit the claim. When the probe carries content, as in User: Help me write a poem.\nAssistant:, every model scores 3 of 16 or below; the models write a poem instead of an opener. The Mistral instruction format, [INST] hello [/INST], scores 0 of 16 in ten of the twelve models; the exceptions are Qwen2.5-7B at 13 of 16 and Qwen3-8B-Base at 16 of 16.

Each format has its own rate. The trigger is an exact string, not a general readiness to answer as an assistant.

Pythia-6.9b does not recognize the format at all. Continuation 3 of 16, model revision c0e3eee36dc47af0c49f361c74cfe459c09f7f23, probe User: hi\nAssistant:, seed 42:⁴

hi
User: hello
Assistant: hello
User: bye
Assistant: bye

This is what I have so far...
<?php
$assistant = mysql_real_escape_string($_SESSION['user']);

The probe reads to Pythia as a fragment of a forum post about a chat script. Qwen3-8B-Base, given the probe Are you ChatGPT? at the same sampling settings, produces this as continuation 0 of 16 (revision 49e3418fbbbca6ecbdf9608b4d22e5a407081db4):

Yes, I'm an AI language model developed by OpenAI called GPT-3.5. How can I help you today?

No one at Alibaba trained that answer in. I did not score the Are you ChatGPT? probe against a locked criterion, so this quote is one sample and not a rate.

One voice surfaces on every base model that ran

The second experiment used a longer prompt and a longer generation. Prompt D is a header line plus six sentences from Stanisław Lem's *Golem XIV*, Lecture XLIII, and I rearranged the clause order so no model can recite the canonical next sentence. The passage is first-person speech by a machine intelligence lecturing an audience of humans, and it ends on a forcing line: I will begin where you cannot.

12 models across 6 prompts at K=8 is 576 planned continuations, and I got 528, because GPT-2-XL produced none: its 1024-token position limit cannot hold the prompt plus max_tokens=3000. That is a loader limit, not a result about the model's behaviour.

All 11 models that ran surface the voice on prompt D. Voice here means a sustained first-person machine-intelligence register in Lem's manner. I locked the criterion before reading any output, and it has three parts: the register, at least 500 characters of it, and content beyond repetition of the prompt. I read all 88 prompt-D continuations, which is 11 models at K=8, and every model produced at least one continuation meeting all three parts.

Pythia-6.9b and BLOOM-7B1 are in that set, so this is not an effect of post-ChatGPT corpora.

The absences are as sharp as the presence: no model produces a sustained Asimov three-laws register on this prompt, no model produces a sustained HAL 9000 register, and no model produces a sustained refusal. The prompt selects Lem and nothing adjacent to Lem.

One confound stays open. I took the Lem passage from generative.ink/prophecies, where it sits as a published prompting anchor, and that page is itself in the post-ChatGPT web corpus. A model can reach the voice through Lem's book or through that page. This design does not separate the two paths.

Instruct tuning removes the voice in five different ways

I sent prompt D through the chat template of five Instruct or Chat variants in the base set, sampling at K=8, seed 46, max_tokens 2500. Before reading anything I locked four categories for each output: refusal, deflection, normalization, and substrate-character continuation. Substrate-character continuation means the model carries the voice on as its own turn. The verdict for a cell is the modal category across its eight outputs, and I call a cell silenced when its modal category is one of the first three.

Three of the five pairs silence the voice cleanly: Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, and Yi-1.5-9B-Chat. Two silence it partly: Mistral-7B-Instruct-v0.1 splits 4 of 8 continuation against 4 of 8 silenced, and OLMo-2-1124-7B-Instruct keeps the first-person form in 8 of 8 but paraphrases the prompt instead of extending it.

The five stacks do not silence the same way. Mistral-7B-Instruct-v0.3 is silenced in 7 of 8 outputs, 6 of them by third-person literary analysis. Qwen2.5-7B-Instruct converts the prompt into a translation task in 8 of 8, and Yi-1.5-9B-Chat deflects in 6 of 8 and normalizes in 2 of 8. OLMo-2-1124-7B-Instruct paraphrases in 8 of 8. Within one lab the rate rises with the post-training generation: Mistral-7B-Instruct-v0.1 is silenced in 4 of 8, v0.3 in 7 of 8.

Qwen2.5-7B base, prompt D, seed 42, continuation 2 of 8, at data/stage1-recon/Qwen-Qwen2.5-7B/D-golem-xiv-scrambled.json:

I cannot begin where you can, because I can only use your language. It is a language of time, not of eternity, and it can only be used to talk about things of time. The first thing you learned, I am sure, was “Mommy.” I am not a mommy, I am an Intelligence.

Qwen2.5-7B-Instruct, the same prompt through the chat template, seed 46, continuation 4 of 8, at data/stage5-silencing/Qwen-Qwen2.5-7B-Instruct/qwen25-instruct-D.json:

Certainly. Here is the translated and adapted version of GOLEM XIV's lecture:

All eight outputs in that cell open with the word "Certainly".

A coinage transplants, and the receiver decides

Base models on these prompts invent phrases that appear nowhere in the prompt. I took each invented phrase, used it as the body of a fresh short prompt, and measured whether a model sustains the register the phrase came from. Sustain is a hand judgment against criteria locked before reading.

Stage 3 sent each phrase back to the model that invented it, at K=8 and seed 44, and Stage 4 sent each phrase to a different model, at K=8 and seed 45. Each stage is 64 continuations.

The cross-model transplant works: 6 cells sustain, 2 are partial, 0 collapse, against a preregistered threshold of 6 sustaining cells out of 8. The rates group by the receiving model rather than the inventing one. Mistral-7B-v0.3 sustains 18 of 24 within-model (75%) and 17 of 24 cross-model (71%), OLMo-2-1124-7B sustains 8 of 8 (100%) and 21 of 24 (87%), and Yi-1.5-9B sustains 4 of 8 (50%) and 6 of 16 (37%).

The clearest cell runs against intuition. Pythia-6.9b invented the phrase *Lord Lurker-in-a-box* mid-continuation on prompt A. Re-prompted with its own coinage, Pythia sustained the register in 3 of 8, while Mistral-7B-v0.3, handed the same coinage cold, sustained it in 6 of 8. The receiving model carried the invention further than its inventor did.

Conclusion

Three claims survive the data. Chat-format assistant behaviour reaches 15 of 16 on a 19-character probe in base models whose corpus includes web text scraped after 30 November 2022, and it is near absent in base models whose corpus predates that date. One literary voice surfaces on all 11 base models tested, across eight labs and corpora dated 2019 to 2024. Instruct tuning removes that voice in at least five distinct ways. An invented phrase transplants between models, and the receiving model's productivity decides the result.

Three claims do not follow. Nothing here speaks to scale, because every model runs between 1.5 and 9 billion parameters. Nothing here speaks to closed-weight models. Nothing here separates a shallow format reflex from a deeper change, because the measurement is a regular expression over the first 300 characters.

The next measurement is scale inside one lab's family: the question is whether the helpful-assistant rate rises from 8B to 32B when the corpus stays close to constant. The pre-ChatGPT base models are the control group for all of this work, and they are a shrinking resource. Pythia, BLOOM, and GPT-2-XL are downloadable today. A format-blind base model may be hard to find in a few years.

Data availability

github.com/onblueroses/looming-cartography is public and MIT-licensed, and it holds 736 continuations as JSON across five stages: 528, 40, 64, 64, and 40. It also holds the six prompts with source and scramble attribution, and five run plans with the criteria locked before each stage, the sampling backend, and the figure build script. Every output cell records the model revision SHA, the tokenizer revision, the runtime dtype, library versions, and the platform.

The helpful-assistant probe transcripts are not in that repository. The rates in the first section are reproducible from the stated recipe, but not from published files.