My model card says 1.000 F1. Do not believe it.

The number that should have worried me

My first model card says 1.000 F1 and a 0.0% JSON syntax error rate. A 1.08B student matching a 31B teacher exactly, at 16x less VRAM and 5x the throughput.

I published it. Then I spent the next campaign proving I should not have believed the accuracy half of it.

This is the version history of SchemaForge, written honestly, because the retraining that would settle it is still running and I would rather describe the mess than wait and present a clean story that never happened.

v1: too good, and I knew why

The efficiency numbers are real and I still stand behind them. ≈2.4 GB against ≈38.5 GB. 61.91 tok/s against 12.40. Thirty-six concurrent workers on a 96 GB card where the teacher fits two. And the finding I think is genuinely worth something: teacher scale made no measurable difference — a 4B teacher and a 31B teacher produced identical error rate and F1, so the expensive teacher was a waste of money on this task.

The accuracy numbers are a different story, and the whitepaper says so in a section I marked load-bearing rather than a formality:

• The in-domain suite is five documents. One per vertical.
• The winning checkpoint was distilled on five training samples.
• 5/5 successes gives a Wilson 95% interval of [56.6%, 100.0%]. The point estimate is 100%; the precision of that estimate is close to nothing. A true validity rate of 57% is entirely consistent with what I measured.
• Single seed. No error bars.
• The documents were synthetic. Clean ASCII, consistent labels, no OCR noise, English only.
• The targets were teacher outputs, so wherever the teacher was wrong, the student faithfully learned the error.

And the one that actually matters:

I never ran the SFT control. Same data, same template, same hyperparameters, α = 1.0 — pure cross-entropy, no teacher logits. If that control also hits 1.000 F1, the entire KL distillation apparatus contributed nothing, and the honest name for the work is "template-aligned supervised fine-tuning."

I wrote that in the paper as the single most important gap. It is still open.

The tunnel vision

Here is the part I find most instructive about my own process.

I ran three iterations to study dataset scaling and domain diversity. That was the experiment. The prompt template varied incidentally between runs — I did not treat it as a variable, because why would I.

Template formatting dominated everything else I varied, including a 4x difference in training-set size and a 3x difference in schema diversity.

Same data, same loss, same hyperparameters, different instruction header: zero-shot validity went 0.0% → 70.0% → 0.0%. Two of those three iterations landed below the untrained base model's 34.2%. Distillation with a mismatched template is worse than no distillation at all — the student had not failed to learn, it had learned a policy keyed to a trigger that never appeared at inference.

That turned into the one result from v1 I would defend in a room full of reviewers: a small student does not merely inherit prompt brittleness from its teacher, it concentrates it. Limited capacity binds the behavior to the literal surface form of the training prefix, and performance degrades non-gracefully — to zero, not to degraded-but-usable.

I only found it because it broke. I was looking somewhere else entirely.

v1.5: more data, worse numbers

So I rebuilt the evaluation to stop flattering the model, and retrained.

Everything got harder on purpose. Twelve schema domains instead of five, three of them held out from training entirely and enforced at the dataset API so they cannot leak. Documents deliberately corrupted by ten operators — OCR noise, delabeling, abbreviation, code-switching, nesting — at parameterized severity. A mandatory validation gate on every teacher output before it could become a label: parses as JSON, validates against the schema, every semantic value traceable to the source text or a declared ontology, nothing asserted the schema does not license. That gate rejects 41.1% of what the teacher produces.

And the framing changed. A tuned deterministic pre-pass now takes every field regex already owns — dates, IDs, amounts, emails, phones — and the model is scored only on what is left. Whole-document accuracy, which is what v1 measured, credits the model for fields a hundred lines of regex solve perfectly.

The corpus went from 5 samples to 1,080 records, 636 of them surviving the gate. Here is what that bought:

checkpoint	gated examples	field F1	precision	hallucination	schema validity
base MiniCPM5-1B, zero-shot	0	0.4263	0.5711	0.1132	0.8615
iteration 3	166	0.3873	0.3923	0.1479	0.9385
iteration 5	636	0.4216	0.4403	0.0717	0.9538

Iteration 3 was worse than not distilling at all. Field F1 down 0.039, precision down 0.179, and it hallucinated more. Training loss hit near-zero by epoch 2 of 3 on about 18 examples per schema. Textbook overfitting. It would have been easy to retrain quietly and report only the run that worked; it is in the failure log instead.

Iteration 5, with roughly 4x the data, still sits 0.005 below the base model on field F1. Not above. Below.

What did move: hallucination rate fell from 0.1132 to 0.0717, and schema validity rose from 0.8615 to 0.9538. The model became more complete and considerably less inventive, and paid for it in precision. That is a trade a downstream pipeline with schema validation might happily take. It is not the win, and I am not going to call it one.

I also went from 1.000 F1 to 0.42. Those numbers are not comparable and the drop is not a regression — it is what happens when the benchmark stops doing the model's work for it. But if you only read the two headlines side by side, v1 looks like the better model. It is not; it is the better-flattered one.

One more failure from this round: the confidence signal reports a mean of 0.978 against 47% actual correctness. Temperature scaling takes holdout ECE from 0.59 to 0.16 and no further. The confidence-routed hybrid pipeline that the whole architecture is named for depends on that signal, and it is not yet trustworthy enough to route on.

What is running now

The next campaign is training as I write this, and it is aimed at the limitation register rather than at the score:

• The SFT control. α = 1.0, no teacher logits, everything else identical. This tells me whether distillation ever contributed anything over format conditioning. If it did not, I will say so.
• A real evaluation corpus — n ≥ 500 held-out documents per domain, with OCR-noisy scans and multi-column layouts, and a human-annotated gold subset so accuracy stops being measured against teacher output.
• Competitive baselines under one harness: Qwen2.5-1.5B, Phi-3-mini, prompt-engineered base with constrained decoding, and a plain rule-based extractor.
• Multi-seed runs with actual confidence intervals.
• The crossover plot: the deterministic pre-pass decaying from 0.917 precision to 0.474 under corruption, and the model's curve on the same axes. Both halves exist. Neither has been plotted against the other, and that figure is the whole thesis.

Why publish the version history

Every version keeps its own page and its own numbers. v1 is marked superseded, not deleted — someone may have cited it, and quietly overwriting the numbers under a citation is worse than having published them.

I would rather be the person whose v1 said "the SFT control is the single most important gap in this paper" than the person whose v1 said 1.000 F1 and stopped there. The first one is a paper you can argue with. The second is a press release.

Retraining results will land here, whichever direction they go.