Distilling a 1B model for what regex cannot do

Structured extraction is usually framed as a model problem: point a language model at a document, ask for JSON, measure accuracy. That framing hides a cheaper baseline. Dates, IDs, amounts, emails, and phone numbers are solved by rules, and a model that re-solves them is paying inference cost for something a regex already gets right.

So the question here is narrower and harder: after a tuned deterministic pre-pass has taken everything it can, can a distilled ~1B model beat what remains?

That residual is the semantic part — fields that require reading, not matching. Measuring on it, rather than on the whole document, is the only way to know whether the model is earning its keep.

Baseline: the deterministic pre-pass itself, measured over 9 training schemas under increasing corruption (OCR noise, typos, delabeling), n=36 per row:

severity	field F1	precision	recall	missing-field rate
0.0	0.393	0.917	0.250	0.727
0.3	0.230	0.727	0.136	0.812
0.6	0.138	0.528	0.080	0.849
1.0	0.092	0.474	0.051	0.892

Two numbers carry the thesis. Precision starts at 0.917 and recall at 0.250 — rules are nearly always right about the fields they own and structurally silent on everything else. Recall is computed over all schema leaves including the semantic ones the pre-pass never attempts, so 0.25 is not a failure, it is the headroom.

And precision decays from 0.917 to 0.474 as corruption rises. Under OCR noise and missing labels rules do not merely go quiet, they start being wrong. That decay is the curve the model has to beat.

Student: openbmb/MiniCPM5-1B. Teacher: google/gemma-4-31B. Twelve schema domains, three of them (insurance_claim, conversation, kg_triple) held out from training entirely and used only to test generalization to unseen schemas.

Field ownership is enforced at import time. Each schema declares a deterministic/semantic partition, and deterministic_fields | semantic_fields == leaf_paths(model) with the two disjoint — a bad split raises before any evaluation can run on it. The pre-pass is asserted never to fill a semantic field.

Hard examples are generated, not collected. Ten corruption operators — OCR noise, delabeling, reordering, abbreviation, synonym substitution, typos, code-switching, nesting, implicit inference, genuine ambiguation — applied at parameterized severity to clean seed documents with known gold. Deterministic given a seed: the same seed produces byte-identical JSONL.

Every teacher output passes a validation gate before it can become a label. Four checks: parses as JSON, validates against the schema's Pydantic model, every semantic string is a literal substring of the source or a registered ontology derivation, and nothing is asserted beyond what the schema licenses. The rejection rate is published, not hidden — 41.1% of teacher outputs were thrown away on the current corpus.

Distillation is sequence-level (cross-entropy on validated teacher outputs), not the cross-tokenizer logit KL the V1 approach attempted — that was invalid, since the teacher and student tokenizers do not align.

Evaluation slices. Micro-averaged field precision/recall/F1, schema validity, hallucination rate, missing-field rate, reported per schema and per corruption operator. An aggregate mean would hide exactly the result this project exists to produce.

Measured on a 72-record eval set spanning all 12 schemas including the 3 held out, at 0.0/0.5 corruption severity.

checkpoint	gated training examples	field F1	precision	recall	hallucination	schema validity
base MiniCPM5-1B, zero-shot	0	0.4263	0.5711	0.3401	0.1132	0.8615
iteration 3	166	0.3873	0.3923	0.3824	0.1479	0.9385
iteration 5 (current)	636	0.4216	0.4403	0.4044	0.0717	0.9538

This is not yet a win on field F1. Iteration 5 sits 0.005 below the un-distilled base model — inside the noise, but not above it.

What did move: schema validity from 0.8615 to 0.9538, recall from 0.3401 to 0.4044, and hallucination rate from 0.1132 down to 0.0717 — the distilled model invents fields at roughly two-thirds the base rate. It paid for that in precision, 0.5711 to 0.4403.

The honest summary is a trade, not an improvement: the model became more complete and more trustworthy about what it emits, and less exact. For a downstream pipeline that validates against a schema anyway, that trade may be the right one — but the headline metric does not yet support the claim, and it is reported that way.

Iteration 3 was worse than doing nothing. Trained on 166 gated examples — roughly 18 per schema — the checkpoint lost field F1 (-0.039) and precision (-0.179) against the base model, and hallucinated more. Training loss hit near-zero by epoch 2 of 3. Straightforward overfitting, and it would have been easy to quietly retrain and report only the better run. The failure log records it instead.

Cross-tokenizer logit KL, the entire V1 approach, was invalid. Distilling logits between a Gemma teacher and a MiniCPM student assumes aligned token vocabularies. They do not align. The V1 scripts are retained for provenance and marked superseded rather than deleted.

The first teacher generation run produced 0/15 parseable outputs. The JSON extractor stripped markdown fences but did no balanced-brace matching, so with no stopping condition the teacher kept generating further document/JSON pairs after the target object closed, and the trailing hallucinated text leaked into the label. That is 100% label noise from a parser bug, independent of model quality. Fixed with a string-aware balanced-brace scanner; the broken output is preserved on disk rather than deleted.

The confidence signal is badly overconfident. Mean token log-probability reports 0.978 against 47% actual correctness. Temperature scaling helps but does not fix it — holdout ECE 0.59 → 0.16, and only after widening a calibration grid whose ceiling was itself producing an artifact.

A 72-record eval set is small, and exact-match is 0.00 across every arm — no configuration reproduces a full document's JSON exactly, so all conclusions rest on field-level metrics.

The hybrid pipeline the project is named for is not built. Routing between the deterministic pre-pass and the model on a calibrated confidence threshold is the point of the architecture, and it does not exist yet. Neither does the failure-category analysis, the continual distillation loop, or the benchmark suite against a commercial API.

The headline figure — the deterministic decay curve and the model's curve on the same corruption sweep, crossing — has not been plotted. Both halves exist; they have not been put on the same axes.

No latency, cost-per-document, or throughput numbers have been measured, so nothing here supports a claim about whether this is cheaper than an API call.

Corpus scale is the productive lever. The 3→5 trend closed the gap on every axis at once as gated examples went from 166 to 636, which points at more data rather than a different KD objective — untested above 636.

Then the two things that make it a system rather than a checkpoint: the confidence-routed hybrid pipeline, and the crossover plot that shows where the model starts beating rules.

Self-consistency sampling and a trained calibration head are both named in the research direction as stronger than temperature scaling. Neither has been tried.