Do Agentic Skills Beat Retrieval?

• A search-enabled control. The standard skills demo compares against an information-starved agent, which measures access to information rather than the value of packaging. Here the baseline (A1) has web search; the information-starved condition (A0) is retained only as a floor, and the retrieval-only gain is reported before any skill effect is claimed.
• A four-arm design — bare / search / skills / skills+search — paired by task and seed, with retrieval removed at the harness level rather than by prompt instruction, per-run query logging, and byte-identical prompts across arms.
• A completed 120-run wave with full cost instrumentation (tokens, tool calls, wall-clock on every run) and honest reporting of a null-to-mixed outcome, including one hypothesis that is refuted rather than merely unsupported.
• A documented contamination failure and its fix: the agent reached the scoring script through filesystem tools the arm configs had left enabled. This is the most transferable lesson in the work for anyone building a benchmark of this shape.

Five skills (architecture, training, debugging, performance, benchmarking) are packaged as directories containing instructions with YAML frontmatter, reference knowledge, executable tools, and deterministic evaluators, deployable into any agent runtime that reads frontmatter-driven skill directories.

Each arm runs the same six tasks, the same five seeds, the same model, and the same hardware. Arms differ only in whether the retrieval tool is registered and whether the skill directories are loaded. Outputs are scored by deterministic scripts rather than a judge model, so scoring cannot be influenced by knowing which arm produced an output.

Tasks 1–4 cover architecture design, training config, failure diagnosis, and throughput optimization. Tasks 5 and 6 exist specifically to separate skills from search: Task 5 forces arbitration between genuinely conflicting published guidance, Task 6 forces extrapolation past any single published recipe.

The primary endpoint (A3 vs A1) was pre-registered. All other comparisons are secondary, Holm-corrected, and reported as exploratory.

Cost (H3) — refuted, in the opposite direction. Against the search-enabled control, skills+search consumed 2.64x the tokens (109,938 vs 41,593 per run), 23% more tool calls, and 26% more wall-clock. Even the offline skills arm, which makes no search calls at all, spent 26% more tokens than the search baseline. A loaded skill is itself context, paid on every run.

Quality (H1) — one gain in six tasks. Diagnostic accuracy on the debugging task improved (+0.034, d = 1.60, paired p = 0.023 uncorrected) — the most purely procedural task in the set, which is where the theory specifically predicts skills should win. No reliable effect on the other five. Tasks 1 and 2 were flagged before the wave as near-ceiling and therefore low-power.

Conflicting guidance (H4) — floor across all arms. Internal consistency on Task 5 was 0.00 / 0.20 / 0.00 / 0.00 across A0–A3. Nineteen of twenty runs combined mutually incompatible recommendations; the single success came from the no-skills arm. Skills+search spent 7x the control's tokens on this task and still failed.

Retrieval-only gain — approximately zero. A1 minus A0 was null on every task at 51% more tokens, so the control this study was built to beat was not itself lifted by retrieval. That points at model capability, not context strategy, as the binding constraint on all four arms.

One model (a small, fast one), one keyless retrieval backend (a DuckDuckGo scrape rather than a paid API), and n = 5 paired runs per cell — the design's stated minimum. Detecting a d ≈ 0.45 effect at 80% power needs roughly n = 40, so most near-zero results here are uninformative rather than evidence of absence.

Two of six tasks are near-ceiling: the control scores 21–25 out of 25, leaving no room for an effect to appear. This was caught by the headroom check before the wave and reported rather than re-scored after the fact.

Tasks 2 and 4 rest on numbers the model reports about itself. No training job runs in this benchmark, so "successful launch" and "throughput delta" are internal-consistency and plausibility signals, not measurements. Replacing them with real short profiled runs is the largest available validity upgrade.

The rubrics and the skills were authored in the same repository, so rubrics may encode what the skills happen to do. An independent review for this circularity is outstanding — but note the bias runs toward the skills arms, which strengthens rather than weakens a null result.

GPU telemetry was unavailable on the benchmark VM, so no independently measured utilization figures exist for any arm. That row is marked N/A rather than fabricated.

The work sits between two literatures that rarely meet. On one side, Mixture-of-Experts training practice — Switch Transformer, GShard, DeepSeek-MoE, and the Megatron/DeepSpeed expert-parallelism implementations — which supplies the domain content the skills encode and, importantly, the genuine disagreements (capacity factors from 1.0 to 2.0, aux-loss coefficients spanning two orders of magnitude, top-1 versus top-2) that Task 5 is built on.

On the other, agent evaluation and tool-augmented LLM benchmarking, where the standard failure is a control arm chosen to be beatable. The methodological contribution here is narrow and borrowed from clinical trial practice rather than ML: pre-registration, a control that is actually strong, an explicitly reported floor, and multiplicity correction on everything that is not the primary endpoint.