Small Mind Companion

Small-Mind asks how much of a small language model's missing memory can be recovered without adding parameters: through an external memory store, retrieval, LoRA post-training and on-policy distillation. It is the system behind Small-Mind Study 001.

The base model is google/gemma-4-E2B-it (~2B effective parameters), chosen in a recorded bake-off rather than by default: four vision-language models scored on 40 judge-graded prompts, with Gemma at 4.00 against 3.25, 3.24 and 2.36, and the reason for each rejection written down in docs/adr/0001-model-selection.md. The base model can see images, but everything trained and measured so far is text-only.

A model with no memory cannot be right about anything outside its context window, and a longer context window does not turn into months of remembered conversation. The engineering problem is not "add a vector database". It is deciding what to retrieve, how much of it fits a token budget, and, the harder part, when to say nothing at all.

A memory benchmark that only scores recall rewards a model that guesses. So the project's benchmark, PMB, is built to punish that: 688 probes over 8 personas in eight categories (factual, episodic, temporal, preference, continuity, outdated fact, distractor and unanswerable). The 80 unanswerable probes ask about things the user never said, so a confident wrong answer scores worse than an honest "I don't know".

Memory store. A per-persona SQLite database split into short-term, episodic and semantic tiers. An LLM proposes memory claims from each turn, and a claim is kept only if it cites a verbatim span of that turn: 503 of 505 accepted over 459 turns.

Retrieval. FTS5 BM25 and 384-dim multilingual-e5-small vectors, fused by reciprocal-rank fusion, re-scored on recency, importance, confidence and entity match, then MMR-diversified to the top k=8.

Context builder. A token-budgeted assembler that places the persona card, retrieved memories and recent turns as real system and user turns, not one concatenated string.

Model. gemma-4-E2B-it with LoRA adapters from SFT, then DPO, then on-policy distillation from gemma-4-E4B-it.

A k-sweep over the number of retrieved memories (0, 2, 4, 8, 16) peaked at k=8 with 17.1% accuracy. It ran on a 120-probe subset, and abstention at that setting was 0.0, so k=8 is the best recall setting rather than a free win.

Post-training runs in the usual order, LoRA SFT then DPO then distillation, all with TRL. At the larger data scale that meant 2,480 SFT examples (2,232 for training) and 2,277 DPO preference pairs across the same 8 personas.

DPO gave the clearest preference signal in the project. In 105 blind pairwise comparisons judged in both orders, DPO + memory beat SFT + memory 45.7% to 21.0%.

Distillation helped, measured the right way. The first figure quoted, +3.3 points of accuracy (15.30% to 18.59%), compares a checkpoint from before DPO with one from after distillation, so it covers two stages. Holding the DPO stage constant, distillation + memory beat DPO + memory 38.1% to 30.5%, a +7.6 point gap. Abstention stayed flat (70.0% to 71.25%), and a judge-based persona-consistency score moved from 0.63 to 0.646 on 60 samples, too small a change to call an improvement.

The teacher, gemma-4-E4B-it, is not persona-tuned. That risk was named before the run, and the persona-consistency check exists to catch it.

Every system is scored on the same 688 PMB probes, one seed each. Answer accuracy (pra_lenient) is a single-response judge score over the 608 answerable probes; abstention (UAR) comes from a rule-based detector over the 80 unanswerable ones. Dual-order judging, which cancels a judge's preference for whichever answer comes first, is used for the pairwise comparisons.

system	accuracy	abstention
A: raw model, no memory	0.16%	13.75%
B: LoRA SFT, no memory	0.16%	16.25%
D: hybrid retrieval memory (k=8)	15.13%	8.75%
E: SFT + memory (small data)	17.76%	33.75%
E: SFT + memory (2,480 examples)	15.30%	70.0%
SFT + DPO + distillation + memory	18.59%	71.25% (95% CI 61.25 to 81.25)

Retrieval alone recovers real recall from a model that otherwise answers 1 of 608 questions; SFT alone does not. Getting the model to abstain took explicit supervision and tuning: restoring the abstention training signal pushed abstention to 96.25%, at the cost of over-hedging on answerable questions. Rebalancing the data ratios settled at 70.0% abstention with false abstention on answerable questions at 32.1%.

Quantization was explored (Q2_K produced broken output on two independently trained checkpoints; Q3_K_S was the smallest coherent level), but no size or speed measurement is backed by a committed artifact, so none is published here.

Nothing is deployed. The mobile/ directories for on-device builds are empty, and no quantized checkpoint has been validated on a phone. What is published is research output: eight model checkpoints on Hugging Face under arjhinety, the code, and the Study 001 results frozen at the study-001 git tag.

Voice, a full persona-card schema and a runnable companion app are roadmap items, not shipped features.

A 10x data run looked like it made abstention worse. It was two bugs in sequence. A text-based dedup step in the data generator was collapsing about 227 abstention training examples into 1, because fixed-template refusals looked like exact duplicates. After that was fixed, the eval's own abstention detector did not recognize the phrasing the model had just learned, so a real improvement read as a further regression. Fixing both exposed a genuine third problem, over-hedging on answerable questions, which needed its own rebalance.

A rubric bug inflated a zero-memory baseline to about 94%. An operator-precedence mistake in a string concatenation dropped the gold answer from the judge's rubric whenever a probe had no listed alternatives, which is most probes. A model with no memory scoring 94% on personal-recall questions is impossible, so it was treated as a bug, not a result.

The DPO pairs were built from older data. They predate the SFT set's dedup fix and rebalance, and none of the 2,277 pairs shares the final SFT system prompt. It is recorded in the errata rather than hidden.

The environment fought back. The bug log (docs/model_quirks.md) holds 27 root-caused entries, including loading a multimodal model through the wrong AutoModel class, a cuDNN, Blackwell GPU and conv3d incompatibility that only disabling the cuDNN v8 path fixed, and breaking API changes in trl's SFTTrainer.

ORPO was not run. The pinned trl version does not support it, and pinning a different version risked breaking the working SFT and DPO pipeline.

Hypotheses and evaluation design are committed to git before results, so the record of what was predicted cannot be quietly rewritten. Study 001 is frozen at the study-001 tag, and 34 errata record every correction made to it since.

What held up: retrieval carries recall and post-training carries abstention; combining them beats either alone; more retrieved memories is not better past k=8; and a teacher that is not persona-tuned did not measurably hurt persona consistency, a result that was checked rather than assumed.

What is still open: the study's central comparison, a 2B model with memory against an 8B model without it, has not run. There are no multiple seeds, so most gaps have no interval. No DPO checkpoint has a full-benchmark score of its own. The abliteration study (H22, 24 probes) and the emotional-range study (H24, 27 probes) are built but not run, and there is no paper yet.