SchemaForge v1.5

• A residual framing. The model is measured on what a tuned deterministic pre-pass leaves behind, not on whole documents. Whole-document accuracy — the v1 setup — flatters any model by crediting it with fields a regex already solved.
• A published baseline decay curve. The deterministic pre-pass under increasing corruption: precision 0.917 → 0.474 as OCR noise, typos and delabeling rise. Under corruption, rules do not go quiet — they go wrong. That is the curve the model must beat, stated before any model result.
• A mandatory teacher-validation gate with a published rejection rate. JSON parse, schema validation, substring-or-ontology grounding for every semantic value, no unlicensed assertions — rejecting 41.1% of teacher outputs. v1 trained on ungated teacher output and inherited its errors.
• Sequence-level KD replacing v1's cross-tokenizer logit KL, which assumed a vocabulary alignment that Gemma and MiniCPM do not share.
• Held-out schemas — 3 of 12 domains never appear in training, enforced at the dataset API.
• A real eval set instead of n=5 synthetic documents, with negative results kept in an append-only log.

A deterministic pre-pass extracts the fields rules own — dates, IDs, amounts, emails, phones — binding each to its nearest label. It is asserted never to fill a semantic field, so anything it cannot resolve falls through rather than being guessed at.

What remains is the semantic residual. A ~1B student (MiniCPM5-1B), distilled from gemma-4-31B, fills those fields.

Training data is manufactured, not collected. Clean seed documents with known gold are corrupted by ten operators at parameterized severity — OCR noise, delabeling, abbreviation, code-switching, nesting, ambiguation — deterministically from a seed. The teacher labels the corrupted documents; every output passes the validation gate before entering the training set.

Distillation is sequence-level cross-entropy on gated outputs. The intended final stage — routing between rules and model on a calibrated confidence threshold — is specified but not built.

Measured on 72 records spanning all 12 schemas including the 3 held out, at 0.0/0.5 corruption severity.

Field F1 is 0.4216 against the base model's 0.4263 — not a win. The distilled checkpoint sits 0.005 below zero-shot on the headline metric.

Hallucination rate fell from 0.1132 to 0.0717 and schema validity rose from 0.8615 to 0.9538. Recall rose, precision fell. More complete and less inventive, at the cost of exactness.

Why these numbers are so much lower than v1's. They are not comparable. v1 reported 1.000 F1 on 5 synthetic, clean, in-domain documents against teacher output. This is 72 records across 12 schemas including 3 never seen in training, under deliberate corruption, scored only on the semantic residual after rules have taken the easy fields. The task got harder on every axis at once — deliberately.

Deterministic baseline, for reference: field F1 0.393 at zero corruption falling to 0.092 at full corruption. Both systems are weak in absolute terms, and the crossover plot that would show where the model overtakes rules has not been produced.

Exact match is 0.00 in every arm — no configuration reproduces a document's full JSON exactly. Every conclusion is field-level.

The eval set is 72 records. A 0.005 difference in field F1 means nothing at that size, which is why iteration 5 is described as a trade rather than a result. This is still an order of magnitude more than v1's n=5.

The hybrid pipeline is not built. Confidence-routed escalation between the pre-pass and the model is the architecture this paper is named for. Nor is the failure-category analysis, the continual distillation loop, or the benchmark suite against traditional parsers and commercial APIs.

No latency, throughput, memory, or cost-per-document figures — so unlike v1, this version says nothing about deployability.

The confidence signal is untrustworthy: mean token log-probability reports 0.978 against 47% actual correctness, and temperature scaling only takes holdout ECE from 0.59 to 0.16.

The SFT control v1 omitted is still missing, so the contribution of the KD objective over plain fine-tuning remains unmeasured across both versions.

The residual framing borrows from information-extraction practice that predates language models: rule and template systems remain the production default for invoices, forms and clinical documents precisely because they are auditable and cheap where they apply. The contribution is not replacing them but measuring against them honestly.

Against its own predecessor, this version is a methodological correction rather than an improvement in score. v1's cross-tokenizer logit KL is unavailable across mismatched tokenizers; v1's evaluation credited the model for fields a regex solves; v1's targets were ungated teacher output. Each is addressed here, and the reported numbers fell as a result — which is the expected direction when a benchmark stops flattering the system.

The validation gate is closest in spirit to constrained decoding and grounded-generation checks: every semantic value must be traceable to source text or a declared ontology derivation.