SchemaForge

Structured extraction — turning a document into a typed, schema-conformant record — is usually attacked with hand-written rules or with a language model, and each has a failure mode the other doesn't. Rules are precise and free but structurally blind to anything that needs interpretation; a model interprets but hallucinates, mistypes, and costs money per document. SchemaForge is the middle position, built rather than argued for: a tuned deterministic pre-pass takes every field pattern-matching can own — dates, IDs, amounts, emails, phone numbers — and a distilled ~1B-parameter student is asked only for what is left, the semantic residual. Routing is static, by a field-ownership partition declared in the schema itself.

Three constraints made this more than a training script. First, "the model handles what rules can't" is worthless unless field ownership is provably disjoint — a schema that silently lets both systems claim the same field, or neither, produces a system that looks fine until it doesn't. Second, distillation is only as good as its labels, and a teacher model at temperature 0.1 will happily emit a plausible-looking wrong answer; nothing stops bad labels from becoming training data unless something is built to stop them. Third, "the eval score changed" has to mean the system changed, not that the corruption seed or the teacher's sampling drifted — determinism had to be a property of the pipeline, not an assumption about it.

Twelve schema domains (three held out from training entirely) each declare a deterministic_fields / semantic_fields partition; SchemaSpec.__post_init__ raises at import if the two aren't disjoint or don't cover every leaf path, so a bad split is a startup error, not a production surprise. The deterministic pre-pass (extractors.py + prepass.py) resolves dates, IDs, amounts, emails, phones and URLs by nearest-label binding, asserted before returning to never fill a semantic field — anything it can't resolve falls through to unresolved rather than being guessed at or silently dropped.\n\nEverything the model needs to learn from is manufactured, not collected: a hard-example generator applies ten corruption operators (OCR noise, delabelling, reordering, abbreviation, code-switching, implicit inference, deliberate ambiguation, among others) to clean seed documents with known gold, deterministically from a seed — the same seed produces byte-identical JSONL. Teacher labels for the corrupted documents pass a four-check validation gate (JSON parses, schema-validates, every semantic value traces to source text or a registered ontology derivation, nothing asserted beyond what the schema licenses) before they can enter the training set. Sequence-level distillation trains the student on what survives the gate. At inference, the pre-pass and the distilled model are merged by field-ownership precedence into the hybrid system, with a separate confidence/calibration module and an eight-category failure-taxonomy classifier feeding the next iteration's hard-example generation.

• Field ownership is enforced at construction, not by convention — deterministic_fields | semantic_fields == leaf_paths(model), disjoint, checked in __post_init__.\n- The pre-pass never fills a semantic field; this is asserted in code before run_prepass returns, not just documented.\n- Corruption generation is deterministic given a seed — reproducing a specific hard example is a seed number, not a re-run-and-hope.\n- Held-out schemas cannot leak into training: generate_dataset(..., split="train") raises if a held-out schema is requested.\n- Metric units are counted as leaf units in both numerator and denominator (a 3-element list field contributes 3, not 1) — the kind of unit mismatch that quietly halves a headline number if it's gotten wrong once and never re-checked.\n- 97 tests, steps 1–4 of the pipeline (registry, evaluation harness, deterministic pre-pass, hard-example generator) need no torch, no GPU, and no network to run.

The full experimental account — hybrid field F1 0.6432–0.6827 against 0.29 for the deterministic pre-pass alone and ~0.45–0.48 for the model alone, the omission-failure investigation, and the ongoing V3 recipe sweep — lives in the companion research write-ups, since a project page is the wrong shape for a lab notebook. What belongs here is the measurement discipline underneath those numbers: an evaluation harness that computes micro-averaged field precision/recall/F1 sliced per schema and per corruption operator (an aggregate mean would hide exactly the result this project exists to produce), a split="eval" that defaults to exactly the three held-out schemas, and ambiguate-tagged items excluded from accuracy by default so contestable gold never penalises a correct system — reported separately under an excluded key rather than silently dropped.

Not a deployed service — a research/training pipeline. Steps 1–4 (schema registry, evaluation harness, deterministic pre-pass, hard-example generator) run locally with no GPU. Training runs on an AMD Instinct MI300X (192GB) via SSH, ROCm 7.2.4 / PyTorch 2.10.0.dev+rocm6.4, with device and dtype resolved at runtime rather than assumed — GPU access provided by the AMD AI Developer Program. The V1 checkpoint is public on Hugging Face; the V2-FINAL release checkpoint (iteration 15, hybrid field F1 0.6827) is prepared for publication and pending upload.

The first teacher-generation run parsed JSON output by cutting at the first closing brace, which truncates any nested object — 100% of labels from that run were corrupted. Caught before any training ran on it; the fix, a real balanced-brace parser, is now permanently gate check one.\n\nA training run at 166 gated examples (roughly 18 per schema) produced a checkpoint worse than not distilling at all — field F1 down 0.039, hallucination up, training loss near-zero by epoch 2 of 3. Textbook overfitting on too little data, and it went into the failure log rather than being quietly retrained away.\n\nA teacher-generation deadlock — busy-spinning at a fixed batch boundary, no output flushed, no usable partial work — was recovered without re-running the expensive GPU stage at all: since teacher generation is greedy and seeded, the already-saved outputs from the prior run are byte-identical to a fresh run's, so only the cheap, CPU-only admission gate needed re-running, against the previous run's saved rejections with a relaxed match threshold.\n\nSeparately, comparing the full iteration history side by side — not each run against its predecessor — surfaced two runs with identical corpora and hyperparameters that produced hybrid F1 scores 0.043 apart. Root cause: the teacher was sampled at temperature 0.1 with no fixed seed, so the training labels themselves were non-deterministic. Fixed with greedy decoding, then verified rather than assumed fixed: a controlled re-run of a prior iteration's exact setup moved F1 by 0.0003 — two orders of magnitude below the swing that triggered the investigation.

• Determinism is a debugging tool before it is anything else. It is what made a compute-costly failure (a wedged GPU run) recoverable with a CPU-only re-run of the cheap stage, and it is what let a 0.043 F1 swing get diagnosed instead of just re-run until it looked better.\n- A validation gate that rejects a large fraction of teacher output (41% at points) is not a bug in the pipeline — it is the pipeline refusing to let bad labels become training data, and the rejection rate is worth publishing, not hiding.\n- Comparing the full run history, not just adjacent runs, is the only way non-deterministic infrastructure noise gets caught; pairwise comparisons are individually plausible and collectively blind to it.\n- Next: upload the V2-FINAL checkpoint to Hugging Face, run the delabel/implicit corruption-stacking test the V3 research write-up specifies, and grow the 72-record eval set to tighten its confidence intervals.