My hybrid extraction system beats both its halves. It still goes silent on half its failures.

The problem first

My hybrid extraction system — a tuned deterministic pre-pass plus a distilled model, routed by field ownership — beats both halves alone, on every metric, on every checkpoint I have tested. Field F1 0.64–0.68 against 0.29 for rules alone and 0.45 for the model alone. That part is not in question.

Here is the part that is: on every single one of those checkpoints, the dominant way the system fails is not hallucination and not a schema violation. It is silence. The system just does not emit a value for a field the document actually contains, 55 to 62% of the time something goes wrong, more than every other failure category put together. I have tried four different fixes against it. None of them worked. One made it worse.

I am leading with that, not the F1 number, because the F1 number is real but it is not the interesting fact about this system, and burying the failure mode under the headline is exactly the kind of thing I criticized my own v1 model card for doing.

What V2 actually built

The prior version of this work (v1, v1.5) never built a real hybrid. It measured a distilled model in isolation on whatever a deterministic pre-pass could not resolve — the "semantic residual" — and reported that number. Useful for isolating the model's contribution, useless as a description of a deployed system, because a deployed system needs the merge, and the merge was never built.

V2 builds it. Rules take every field they own by a declared schema partition — dates, IDs, amounts, emails, phones — asserted at import time to never touch a semantic field. Whatever is left goes to a ~1B distilled student. Routing is static, by ownership; merging is by precedence, not by confidence, because my confidence calibration is not good enough yet to route on (more on that below). Fifteen training iterations later, here is where it landed:

system	field F1	hallucination	schema validity
deterministic pre-pass alone	0.2911	—	—
distilled model alone	0.4461	0.0852	0.7361
hybrid	0.6432–0.6827	0.0136	0.8333

That is not a marginal win. Hallucination fell by more than 80%, schema validity climbed nine points, field F1 more than doubled the rule pass and beat the model alone by a wide margin. It held on every checkpoint I measured across fifteen iterations, not just my best one. If the story stopped here it would be a clean success story, and I would be tempted to stop telling it here.

Why rules can't be trusted alone, even where they're strong

One number from the setup work explains why the hybrid architecture matters rather than being a nice-to-have: I swept the deterministic pre-pass through increasing document corruption and measured precision decay. It goes from 0.990 at zero corruption to 0.537 at full corruption. Rules under stress do not go quiet — they start being confidently wrong, at almost the same rate they were confidently right. Recall, meanwhile, sits at a structural ceiling around 0.26 even on clean text — that is not a corruption effect, it is what regex was always going to top out at on this task. Those two facts together are why no crossover ever appears between the rule pass and the model at any severity: rules never get worse than the model at low corruption, because they never got that far ahead in the first place.

The failure mode that would not move

Four things, tried in order, against the omission share:

1. 10x the corpus. 288 to 3,780 records. No effect.
2. Filled the missing ontologies. Five of twelve schemas had no normalization ontology at all. Filled them. No effect on omission — and, as a side note I still cannot explain, the incorrect_normalization failure category read exactly zero before the fill and reads exactly zero after it. A metric that reads zero whether it's broken or just inapplicable is a bad metric, and I am flagging mine rather than pretending the zero means what I'd like it to mean.
3. A corpus concentrated on the two corruption operators most linked to omission. This is the one I expected to work. It made things worse — omission share rose from 55.0% to 60.1%. My read: isolating those two operators removed the compounding my eval set actually tests. A document that is delabelled and OCR-corrupted and reordered is not the same training problem as one that is only delabelled. I have not tested that explanation. It is a hypothesis, stated as one.
4. A full training-recipe sweep, which is what I have spent V3's first four iterations on. More below.

My working theory for why omission specifically, offered honestly as a theory and not a finding: my admission gate — the check that decides whether a teacher-generated label is trustworthy enough to train on — is a precision filter by construction. A teacher output that guesses at a hard-to-ground field fails the grounding check and never becomes a training example. A teacher output that just leaves the field out passes. I am, structurally, teaching the model to reproduce exactly the conservative behavior that survives my own filter. If that's right, loosening the gate's support check should trade some label quality for less omission. That is untested, and it is next.

The bug that could have invalidated the headline number

Before I trust any of the numbers above, I have to mention this, because it is the part of V2 I am most nervous about and most glad I caught.

Comparing my full iteration history side by side — not each run against its immediate predecessor, which is what I had been doing — I found two iterations with identical corpora and identical hyperparameters that produced hybrid F1 scores 0.043 apart. That gap was bigger than the effect I had attributed to any deliberate change I made in between. Every pairwise comparison I had been running was individually plausible and collectively meaningless, because the noise floor was bigger than the signal I thought I was measuring.

Root cause: my teacher was sampled at temperature 0.1 with no fixed seed. The ground truth itself — the training labels — was non-deterministic across runs. Two runs on a byte-identical document corpus could receive different labels.

I fixed it (greedy decoding, temperature 0), and then I did not assume the fix had resolved the anomaly — I measured it. I re-ran one prior iteration's exact setup, on a byte-identical corpus, under the now-deterministic labeller. Field F1 moved by 0.0003. Two orders of magnitude smaller than the 0.043 swing that started the investigation. At the scale my later iterations actually train at (~1,625 admitted examples), the non-determinism was not the dominant thing driving my results — my best guess is that a single differently-sampled label carries proportionally more weight in a small corpus's gradient, which is consistent with why the two divergent runs were both small (~635 examples).

I am not retracting the earlier, smaller-corpus results — they were real measurements of something — but I do not build on them, and the best F1 number in this project's entire history belongs to one of them. I would rather tell you that than quietly cite it.

What four more iterations bought

V3 is my current attempt against the omission problem, and it is running as I write this. Four iterations so far, none of them a fix — a recipe-grid sweep meant to rule things in or out before I run the experiment that actually tests my gate hypothesis.

iteration	corpus	epochs	LR	field F1	missing share
V2-FINAL (my current release candidate)	n=75	3	2e-5	0.6827	~55%
iter 1	n=100	3	2e-5	0.6581	58.8%
iter 2 (best so far)	n=100	2	2e-5	0.6745	52.3%
iter 3	n=100	1	2e-5	0.6597	57.0%
iter 4	n=100	2	1e-5	0.6671	56.4%

Scaling the corpus alone (iteration 1) made things worse again — the fourth time in this project that a bigger corpus has failed to touch omission. But it told me something: the regression tracked total gradient steps, not document count. Iteration 1's extra epoch over the bigger corpus pushed total steps past what my best V2 checkpoint used, into an over-converged regime. That reframed the rest of the sweep: stop varying corpus size, vary training length directly.

Two epochs turned out to be a clean interior optimum — one epoch underfits, three overfits, two is the peak. Learning rate 2e-5 beat 1e-5 at that point, and not narrowly: the lower rate cost 0.08 in schema validity, the single biggest regression anywhere in this sweep. My conclusion from the whole grid: the recipe I was already using was close to right. No amount of epoch or LR tuning was the lever.

What the sweep did buy: iteration 2 reached a 52.3% missing-field share, the lowest I have measured in this project and the first result under the 55–62% band that held across all fifteen V2 iterations. I want to be exact about what that is and is not. It is three to nine points better than anything V2 produced. It is also still above half of everything that goes wrong, and iteration 2's field F1 (0.6745) is still 0.0082 below my current release candidate on the same eval set. No iteration in this sweep has been used for a Hugging Face upload. My manifests say so explicitly, per run, so I cannot quietly forget it later.

The deadlock, and how I got out of it without redoing the expensive part

One more thing worth recording because it generalizes past this project. Iteration 2's teacher-generation run deadlocked at batch 208 of 900 — busy-spinning, CPU pinned, no new output for over an hour, during a window when I wasn't watching it interactively. I killed it. No checkpoint, no usable output past that point.

Rather than re-running generation — which, since generation is deterministic, risked reproducing the exact same deadlock — I used that same determinism the other way. My saved raw outputs from the previous iteration are byte-identical to what a fresh run would produce for the same corpus and prompt. So I re-ran only the cheap, CPU-only admission gate, with a slightly relaxed fuzzy-matching threshold, against the previous iteration's rejected records. That recovered 37% of them — 534 out of 1,443 — and got me to a usable corpus without touching the GPU again. My checkpoint backup from before the attempt was verified intact throughout, so the recovery carried zero risk to my actual release candidate.

If you are running any pipeline with a deterministic, expensive stage feeding a cheap, reprocessable one: your recovery path from a mid-run failure is not always "re-run the expensive stage." Sometimes it is "re-run the cheap stage against what the expensive stage already produced."

What is actually still open

• The experiment that tests my own gate hypothesis directly — stacking the omission-linked corruption operators into the full mix instead of isolating them, which is what made things worse the first time I tried it — has not run.
• Whether the gate's support check can be loosened to admit legitimate denoising corrections without also admitting wrong ones — flagged twice in V2, never attempted.
• incorrect_normalization still reads exactly zero. I still don't know why.
• My confidence signal is not good enough to route on (0.978 mean confidence against 47% actual correctness, ECE 0.16 at best after calibration), so the confidence-routed hybrid the whole project is named for is not built yet.

Full method, the complete failure log, and per-iteration provenance are in the papers — V2 as the closed, methodologically-settled account, V3 as the in-progress sweep against the one thing V2 left unsolved. Both are marked draft. Neither is finished, and I would rather publish them that way than wait for a cleaner story that has not happened yet.