Arjhine Ty

Four iterations into closing the omission gap. It is 3 points narrower, not closed.

A confirmed recipe optimum (2 epochs, LR 2e-5), a deadlock recovered without re-running generation, and the dominant failure mode still sitting above half of everything that goes wrong

exploratory

The question

V2 tried corpus scale, ontology coverage, and a targeted-composition corpus against the omission share, and none of them worked — the targeted corpus made it worse. Two questions were left open: does stacking the omission-linked corruption operators (delabel, implicit) into the full ten-operator mix, rather than isolating them, do what isolation did not? And is there a training-recipe lever — epochs, learning rate, early stopping — that helps once corpus composition is right, given V2 saw near-zero training loss by epoch 2-3 in every run, a possible early-overfitting signature never directly investigated?

V3's first four iterations answer the second question and set up the first.

Setup & baseline

Baseline is the V2 release candidate, iteration 15 (V2-FINAL): 1,625/2,700 admitted training examples, hybrid field F1 0.6827 on the 72-record eval, missing_field share in the established 55-62% band.

Corpus held at n=100 documents for the full sweep — a deliberate scale-up from V2-FINAL's n=75, tested in isolation first (iteration 1) before any recipe change, so corpus size and recipe are never varied in the same run.

Method

Same pipeline as V2: greedy (temperature 0.0), seeded teacher labelling — the determinism V2's own audit established as load-bearing — four-check admission gate, sequence-level distillation, 72-record and 288-record held-out evals for cross-checking every result.

Four iterations, one change isolated per run:

  1. Corpus scale alone, n=75→100, same 3-epoch/2e-5 recipe as V2-FINAL.
  2. Epochs 3→2, same n=100 corpus.
  3. Epochs 2→1, completing the epoch sweep.
  4. Learning rate 2e-5→1e-5, at the 2-epoch point.

Iteration 2 required a recovery: teacher generation deadlocked mid-run (see below), and rather than re-running generation, the pipeline's own determinism was used to recover it.

Results

iterationcorpusepochsLR72-rec hybrid F172-rec missing share
V2-FINAL (release baseline)n=7532e-50.6827~55%
iter 1n=10032e-50.658158.8%
iter 2 (best)n=10022e-50.674552.3%
iter 3n=10012e-50.659757.0%
iter 4n=10021e-50.667156.4%

Corpus size alone made things worse (iteration 1): more data at the V2 recipe regressed F1 by 0.0246 against the baseline and left the missing share inside the same 55-62% band it had occupied for fifteen V2 iterations. The lesson was not "more data doesn't help" — it was that total training steps, not corpus size per se, is the variable that matters: iteration 1's 3-epoch run over the larger corpus produced 3,237 total gradient steps, more than V2-FINAL's 2,439.

Epochs 2 is the confirmed optimum — the full sweep at n=100/LR=2e-5 reads 1-epoch 0.6597 / 2-epoch 0.6745 / 3-epoch 0.6581, a clean interior maximum. 2,692 total steps at 2 epochs sits between V2-FINAL's 2,439 and iteration 1's over-converged 3,237.

Learning rate 2e-5 beats 1e-5 at the 2-epoch point — the lower rate under-trains on this corpus size: final-epoch loss 0.0333 against 0.0162, and schema validity drops 0.8889 to 0.8056, the largest single regression measured in the sweep.

The recipe grid is now bracketed: 2 epochs at LR 2e-5 is confirmed optimal. V2/V3's default recipe was already close to right; the sweep found no improvement in either direction.

The headline: 52.3% missing-field share is the lowest measured in the project, and the first result below V2's 55-62% band — three to nine points better than every V2 configuration. It is also not below half of all failures, and iteration 2's field F1 (0.6745) is still 0.0082 below the V2-FINAL release candidate on the comparable 72-record eval. Every iteration manifest in this sweep records checkpoint_used_for_huggingface_upload: false.

What did not work — and the incident along the way

Corpus scale alone, again, does not help — the same conclusion V2 reached with ontology coverage and raw scale, now confirmed under the fuller pipeline: iteration 1's regression (-0.0246 F1 against baseline) is the fourth independent scale-up in this project's history that did not reduce omission.

The lower learning rate does not help — under-training shows up as both worse F1 and, more sharply, an 0.0833 drop in schema validity, the single largest regression in the four-iteration sweep.

A teacher-generation deadlock during iteration 2. The generation process wedged at batch 208 of 900 — busy-spinning, CPU pinned, no output records flushed past that point, for over an hour during an unattended monitoring window before being killed. Diagnosis: the process's output file-descriptor offset stayed stationary across samples while CPU stayed pinned, the signature of a lock or condition-variable busy-spin rather than a slow or blocked wait. No partial checkpoint and no usable flushed output existed to recover in-process.

Recovery exploited the pipeline's own determinism rather than re-running the expensive step. Teacher generation is greedy and seeded, so for a fixed corpus and prompt template, iteration 1's already-saved raw outputs are byte-identical to what a fresh generation run would produce. Re-running only the cheap, CPU-only admission gate — with a fuzzy support threshold (0.85) applied to iteration 1's saved rejections — recovered 534 of 1,443 previously-rejected records (37%), bringing the admitted total to 2,691/3,600 (25.2% rejected, against 40.1% under the strict gate). The pre-iteration checkpoint backup was verified intact before training began, so the recovery carried no risk to the canonical V2-FINAL weights. Re-running generation was never attempted, and the deadlock's root cause inside the batched generation loop was not reproduced interactively — the recovery method made that unnecessary, not the investigation.

Limitations

This is four iterations of a recipe-grid sweep, not the omission-gap fix the project is aimed at. The one experiment that directly tests V2's open hypothesis — stacking delabel/implicit into the full corruption mix rather than isolating them — has not run yet.

No V3 checkpoint has been used for a Hugging Face upload; the release candidate remains V2-FINAL. The 0.0082 field-F1 gap between iteration 2 and V2-FINAL is small relative to what a 72-record eval can resolve, and should not be read as "V3 has not yet caught up" with more confidence than the eval set supports.

The gate-loosening question (§ open questions in the whitepaper) and the incorrect_normalization-reads-zero non-result carried over unresolved from V2 are both still open. Neither has been touched in iterations 1-4.

Next

The stacking experiment — delabel/implicit reinforced within the full ten-operator mix rather than isolated, testing whether compounding rather than composition was the missing variable in V2's failed targeted-corpus attempt.

Whether the admission gate's source-support check can be safely loosened to admit legitimate denoising corrections (an OCR-typo'd name the teacher correctly normalizes) without also admitting genuinely wrong labels — flagged as a real false-rejection source in two V2 iterations, never attempted there.

Held-out validation-based early stopping, to test directly whether the near-zero training loss V2 saw by epoch 2-3 in every run is an early-overfitting signature the recipe sweep has been dancing around rather than measuring.