Arjhine Ty

The hybrid wins on every metric. It still leaves half its failures unexplained.

Deterministic pre-pass + distilled model, routed by field ownership — and the failure mode that four fixes could not touch

result

The question

Structured extraction is usually all-or-nothing: point a model at a document, ask for the whole schema, validate the JSON. That wastes model capacity re-deriving what a regex already gets right, and it hands the model a hallucination risk it does not need to carry — a matcher that cannot find a total emits nothing; a model asked for one can invent it.

The prior line of work (schemaforge-v1, schemaforge-v1.5) measured the distilled model in isolation on the semantic residual after a deterministic pre-pass. It never built or measured the actual merge. V2's question is the one that framing deferred: route by per-field ownership, merge by precedence, and measure the combined system — does the hybrid actually beat either half alone?

Setup & baseline

Held-out evaluation set spanning twelve document schemas, three withheld from training entirely. Three systems compared on the same set: the tuned deterministic pre-pass alone, the distilled model alone (no pre-pass), and the hybrid (pre-pass takes owned fields, model takes the rest, schema validation on the merge).

A severity-swept robustness benchmark runs alongside the headline comparison, because a single clean-text number hides how the rule pass behaves as documents degrade: precision decays from 0.990 to 0.537 as corruption rises from 0 to full severity — degraded rules do not go quiet, they become confidently wrong. Recall, by contrast, sits near a ~0.26 ceiling even on clean text: that ceiling is structural, present with zero corruption, which is why no severity crossover between rules and model ever occurs — rules never get worse than the model at low corruption, because they never got that far ahead to begin with.

Student openbmb/MiniCPM5-1B, teacher google/gemma-4-31B, 12 schemas / 3 held out, fifteen training iterations across this line of work.

Method

Routing is static, by declared field ownership — the same ownership partition schemaforge-v1.5 introduced, enforced at import time so a bad split raises before evaluation runs on it. Merging is by precedence, not by confidence: the pre-pass's answer wins for owned fields, the model's answer fills the rest. Confidence is measured (see limitations) but is not part of the routing decision in V2 — the calibration is not trustworthy enough yet to route on, so nothing in the headline result depends on it.

Everything else about training carries over from schemaforge-v1.5: sequence-level distillation on gate-admitted teacher labels, ten corruption operators for hard-example generation, a four-check admission gate (JSON parse, schema validation, substring-or-ontology grounding, no unlicensed assertions).

Automatic failure classification runs on every eval pass, sorting errors into missing field, incorrect normalization, wrong entity boundary, wrong inferred value, hallucinated field, schema violation, incorrect nesting, and genuinely ambiguous input — the taxonomy the project's charter specified from the start.

Results

The hybrid dominates both components on every metric, on every checkpoint tested — this is not a close call and it did not need to be argued for:

systemfield F1hallucinationschema validity
deterministic pre-pass alone0.2911
distilled model alone0.44610.08520.7361
hybrid (best checkpoint, iteration 15)0.6432–0.68270.01360.8333

Hallucination fell from 0.0852 to 0.0136 and schema validity rose from 0.7361 to 0.8333 in the same comparison. Field F1 more than doubles the rule pass alone and clears the model alone by a wide margin. That result held on every checkpoint tested across fifteen iterations, not just the best one — this is the architectural claim the whole hybrid design rests on, and it is not in question.

What is in question is why the system still fails as often as it does. Across every configuration measured, the dominant error mode is not hallucination and not schema violation. It is silent omission — the system emits nothing for a field the document actually contains — at 55–62% of all classified failures, a plurality larger than every other category combined.

What did not work

Four independent interventions were tried against the omission share, and none moved it below roughly half of all failures:

  1. Corpus scale — an order of magnitude, 288 to 3,780 records. No effect.
  2. Ontology coverage — filling the normalization ontologies of the five schemas that lacked them. No effect on omission, and no effect on the incorrect_normalization category either, which read exactly zero before the fill and still read exactly zero after it — an unexplained non-result, reported as one rather than quietly dropped.
  3. Targeted corpus composition — concentrating training on the two operators most associated with omission (delabel, implicit). The omission share rose, 55.0% to 60.1%. Working interpretation: isolating those operators removed the compounding the eval set actually exercises — a document that is delabelled and OCR-corrupted and reordered is not prepared for by training on delabelling alone. This interpretation was not itself tested in V2.
  4. Training recipe — a full epoch/LR sweep, run in the V3 successor work. Moved the share by a few points at best.

Our working hypothesis, offered as a hypothesis and not a demonstrated mechanism: the admission gate is a precision filter by design, and a precision filter applied to training labels is an omission teacher. A teacher output that guesses at a hard-to-ground field fails the grounding check and never enters training; a teacher output that omits it passes. The model learns to reproduce exactly the conservative behavior that survives the gate. This predicts that loosening the support check should trade some label quality for less omission — a prediction V3's open questions test directly.

A non-deterministic teacher labeller invalidated part of the project's own early evidence. Comparing the full iteration history side by side, rather than each run to its predecessor, surfaced two iterations with identical corpora and hyperparameters that produced hybrid F1 scores 0.043 apart — a gap larger than any deliberate change in between. Root cause: the teacher sampled at temperature 0.1 with no fixed seed, so the ground truth itself was non-deterministic run to run. Fixed with greedy decoding at temperature 0, then verified rather than assumed fixed: a controlled re-run of a prior iteration's exact setup on a byte-identical corpus moved field F1 by 0.0003 (0.6830 to 0.6827) — two orders of magnitude below the 0.043 swing that triggered the investigation. That re-run also surfaced an unexplained side effect: schema validity fell 0.875 to 0.833 with no matching move in F1, which current instrumentation does not account for. The earlier, smaller-corpus results are not retracted — they were real measurements — but they are not built on, and the best F1 in the project's history is one of them.

A regression exposed three process defects at once. A fully automated run scored 0.6432 against a prior 0.6858 on what looked like an identical setup. Building that run's provenance record surfaced: no fixed training seed, checkpoints overwritten in place with no backup (how the project's best-scoring weights were permanently lost), and training log output captured then discarded. None had been visible while results were improving — the general lesson being that a project which only investigates regressions only finds its process defects when it is unlucky.

Two earlier bugs, both caught before they cost anything: the first teacher-generation run truncated every label at the first closing brace (100% corruption, caught pre-training); the first failure classifier double-counted explicit-null predictions across two categories (caught before the numbers were used).

Limitations

The confidence signal is measured but not trustworthy enough to route on: raw mean confidence 0.978 against 47% actual correctness, held-out ECE reduced from 0.59 to 0.16 by temperature scaling and no further. Because routing in V2 is by precedence rather than confidence, the headline hybrid result does not inherit this weakness — but the escalation-threshold and cost-per-document analyses the project's charter anticipated cannot be completed at this calibration quality, and remain open.

A severity-sweep benchmark re-run did not reproduce the project's own earlier figures exactly, on an apparently identical recipe. The discrepancy's source was not identified; it is flagged rather than presented as a match.

The incorrect_normalization category reads zero in every measurement, filled ontologies or not — genuinely unresolved, and evidence that a metric which reads zero both when broken and when inapplicable is a poorly designed metric.

No latency, cost-per-document, or throughput figures exist for the hybrid system as configured, so nothing here supports a deployment cost claim.

Next

The omission share is the one number that fifteen iterations and four interventions did not move, and it is V3's entire mandate — a recipe sweep run against it, and a test of whether stacking the two omission-linked corruption operators back into the full multi-corruption mix, rather than isolating them, which made things worse here, helps.

The release checkpoint from this iteration (models/schemaforge-v2-distilled-minicpm5-1b/) is prepared for publication and pending upload to Hugging Face.