Agent Harness Variance in RL: Claude Code 33%

Agent harness variance reached 28 points on identical weights. Liquid AI's LFM2.5, a 2.6B-parameter model, scored 62.1% in Mini-SWE-Agent and 33.2% in Claude Code on the same 250 hidden tasks. Hugging Face trained around the gap with a capture proxy; four-app RL lifted Claude Code to 49% but cost 8 points in OpenCode.

What Is Agent Harness Variance?

Agent harness variance is the share of a benchmark score that comes from the application wrapping the model rather than from the model weights. In a 2026 Hugging Face study, identical 2.6B-parameter LFM2.5 weights from Liquid AI scored 62.1% in Mini-SWE-Agent and 33.2% in Claude Code, a 28.9-point gap.

Both figures come from the same 250 held-out tasks drawn from the Small Data in Apps data-analysis set, with no training on those tasks. The model never changed. The app did.

This matters because most agent leaderboards publish a single number. Without naming the harness, that number describes the model-plus-application pair, not the model alone.

How Hugging Face Trained a Model Inside Claude Code Without Changing It

Hugging Face captured model traffic from unmodified black-box coding apps by sitting a proxy in front of its own inference engine, so the apps kept using their normal model API while the trainer recorded every call.

Claude Code is Anthropic terminal-based agentic coding tool; Codex CLI is OpenAI's coding agent; OpenCode is an open-source terminal coding agent. None of their source code changed, and neither did the training code.

Clement Delangue, Hugging Face's CEO, described the work as turning Claude Code, Codex, OpenCode and other coding apps into reinforcement learning environments. The apps act as harnesses; the team trained through them rather than against them.

The method depends on one hard constraint: the proxy sits in front of an inference engine you serve yourself. It cannot intercept hosted API models, so open weights are a requirement, not a convenience.

The Capture Proxy, Step by Step

The capture proxy presents itself as a model API, so each app sends ordinary requests to it and receives ordinary answers back. Behind that facade it records token-exact generation data and forwards converted requests to the real inference engine.

The five steps are described in the authors' own write-up. One detail is unspecified: the write-up does not say how the app is pointed at the proxy address.

1. OpenAI session. A POST to the proxy's sessions endpoint returns a capture session ID, and the app receives that ID where its API key normally goes. Every request carries it, so the proxy knows which run each call belongs to.

2. Identify the dialect. The proxy supports OpenAI chat completions, OpenAI responses, Anthropic messages, and Gemini request formats. It checks the path first, then the headers, then the body shape.

3. Convert to one format. The request is rewritten into chat completions format and sent to the inference engine, so four dialects become one and the model never learns which app called it.

4. Record the exact tokens. The proxy asks the engine for prompt token IDs, sampled token IDs, and each sampled token's log probability while the model generates. It also disables top-p and top-k so the model samples from its full distribution.

5. Replay in the app's dialect. The full response is stored, then streamed back in whatever format the app requested.

Why Saving Text Instead of Token IDs Breaks Training

Saving generated text and re-tokenizing it later can produce different token IDs than the model actually sampled, which teaches the trainer on tokens the model never chose. The same problem corrupts the probability comparisons that drive the policy gradient.

Each token contributes a ratio between two numbers: the probability recorded when the model chose that token, and the probability under the current weights. Recomputing the first number later would use updated weights, so the ratio would be wrong.

The analogy of a translator taking notes breaks at exactly this point. A notebook of words is not enough; the proxy writes down token IDs and log probabilities during generation.

The setup uses Async GRPO in TRL, Hugging Face's training library, and rollouts come from weights two to four updates old, which is precisely why the recorded probabilities have to be the sampled ones.

From Multi-Turn Calls to Clean RL Training Sequences

A rollout is one full attempt at a task and contains many model calls, so the proxy links those calls into a causal graph instead of treating them as a flat transcript. Each call points to the earlier call whose prompt plus completion is the longest exact token prefix of its own prompt.

Retries become siblings of the call they repeat. Sub-agents carrying a different system prompt start a new root, and context compaction also starts a new root. Every path from a root to a leaf becomes one training sequence.

Context tokens get a loss mask of zero; tokens the model sampled get a mask of one. A turn that does not align stays as context only and contributes no gradient.

Claude Code rewrites its own history, so a single rollout there produces roughly eight training sequences. That fragmentation is why four-app training processed 451 million tokens against 162 million for OpenCode-only training, a 2.8x difference in data volume.

Multi-Harness RL Results: 33% to 49% in Claude Code

Training in four apps at once produced more even scores than training in one: OpenCode 50%, Claude Code 49%, Codex 54%, and Mini-SWE-Agent near 62%. Claude Code gained about 16 points over its 33.2% baseline; Mini-SWE-Agent gained nothing.

OpenCode-only training concentrated its gains instead. OpenCode rose from 34% to roughly 58%, while Claude Code reached 42% and Codex reached 43%, leaving most of the improvement inside the app it trained on.

Four-app training also cost 8 points in OpenCode, dropping from about 58% to 50%. It processed substantially more data, so the four-app and single-app numbers are not an equal comparison.

Pass@1 across the four apps, the share of tasks solved on the first try averaged over apps, moved from 42.2% for the base model to 52.3% with OpenCode-only training and 54.2% with four-app training. The authors treat differences within roughly two points as noise, which makes those last two figures close to a tie.

Training took 32 hours for OpenCode only and 46 hours for four apps, on two H100 GPUs per run: one for training, one for serving with vLLM.

RL vs SFT: 54.6% Against 47.5% Under Unequal Conditions

RL beat supervised fine-tuning on this setup, but the comparison is not a controlled experiment. The team collected 3,189 successful rollouts from Qwen 27B across 888 tasks and four apps, then trained on those answers.

SFT on that teacher data scored 47.5% for OpenCode-only and 43.1% for four-app training. Four-app RL scored 54.6% at step 700 and 54.2% at step 1,000, so roughly 54%.

The authors state the methods differ in objective, data, task pool, and compute, and call these observations rather than a ranking of methods. The team also selected SFT checkpoints on the test set, which favors those numbers.

Four-app SFT dropped Mini-SWE-Agent from 62.1% to 45.2%, a 16.9-point fall the authors cannot explain. The tool-call reduction is easier to attribute but not to RL alone.

Tool-Call Reduction and Why the Reward Bonus Exists

The four-app RL model used 31% fewer tool calls, measured on tasks that both the base and the trained model solved. That statistic comes with a caveat: the reward function paid for it.

GRPO, group relative policy optimization from the DeepSeek math paper, samples a group of eight rollouts per task and scores each against the others. When every rollout is equally right or equally wrong, the group carries no learning signal.

The reward is 1 for a correct answer plus up to 0.1 for fewer tool calls, and a wrong answer never earns the bonus. In 22.6% of four-app training groups every scorable rollout was correct while tool counts differed, so the bonus was the only signal present.

Without that bonus, between 35% and 58% of training steps carried no learning signal. Note that this 22.6% is a share of training groups and has no relationship to the 22.6-point figure reported in NVIDIA's Polar work.

Fewer calls is not unique to RL. OpenCode-only RL cut 11%, and SFT on the teacher's answers cut 24.2% with no bonus at all, so part of the reduction came from the data rather than the reward.

Independent Evidence That the App Moves the Score More Than the Model

A separate study supports the pattern without proving it for this setup. NVIDIA's Polar paper, submitted on 22 May 2025, proxied model calls from apps whose code it could not change, rebuilt the exact token record of each run, and reported a 22.6-point gain for Qwen3-4B on SWE-bench verified under the Codex harness.

The capture proxy's request converters are adapted from Polar's gateway, so the mechanism is not new. What the Hugging Face work adds is open packaging that works across many apps at once.

A second study by Zhang and colleagues, submitted 7 May 2025, tested GLM-4.5 on 100 tasks from SWE-bench verified, a set of real software bug fixes. Across three app setups it scored 52.5%, 56.5%, and 65.5%, a 13-point swing from the app alone.

In those same three setups the three models tested differed by only 3, 2.5, and 5 points. The authors describe their 7.8x ratio of app variance to model variance as specific to their study, with different models and different tasks.

Limitations and How to Read Agent Leaderboards

Every configuration in the Hugging Face study ran once with a single random seed, and all runs used small models between 2 and 2.6 billion parameters. Two earlier Qwen 3.5 2B runs trained on correctness only peaked at 37% and 39.5%, then fell to about 26%, which shows a checkpoint picked at its peak can overstate the result.

The write-up cites analysis blaming overfitting to one app's format, context layout, and control flow for the Claude Code gap, but runs no test that isolates the cause. Prompt size, history rewriting, and tool names remain guesses.

Treat differences under about two points as noise. Treat large gaps as signals, while remembering they come from single runs on small models.

The practical takeaways differ by role. If you read agent leaderboards, ask which app produced the score. If you build agents, test your model inside your own app. If you train models, copy the capture proxy design and record token IDs and log probabilities during generation, not transcribed text afterward.

Frequently Asked Questions

• What is agent harness variance? It is the portion of a benchmark score attributable to the application around the model rather than the weights. In the 2026 Hugging Face study, the same LFM2.5 weights scored 62.1% in Mini-SWE-Agent and 33.2% in Claude Code on identical tasks.

• Did Claude or Anthropic have any role in this research? No. The trained model was Liquid AI's LFM2.5 with 2.6 billion parameters. Claude Code was only the application the small model ran inside, and its source code was not modified.

• How does a capture proxy train a model inside an app it cannot edit? The proxy answers as the model API, so the app sends normal requests. The proxy converts the request to chat completions format, forwards it to the inference engine, and records prompt token IDs, sampled token IDs, and log probabilities while generation happens.

• Why can't the trainer just save the generated text and re-tokenize it? Different token sequences can produce identical text, so re-tokenizing may yield IDs the model never sampled. The recorded probability would also be recomputed under updated weights, making the policy-gradient ratio wrong.

• How much did four-app training lift Claude Code? From 33.2% to about 49%, roughly 16 points. The same run left Mini-SWE-Agent near 62% and cost 8 points in OpenCode compared with OpenCode-only training.

• Did RL beat supervised fine-tuning? Four-app RL scored roughly 54% against 47.5% for OpenCode-only SFT and 43.1% for four-app SFT. The authors note the methods differ in objective, data, task pool, and compute, and selected SFT checkpoints on the test set.

• Why did the four-app model use 31% fewer tool calls? Partly because the reward gave up to 0.1 bonus for fewer calls, and partly because the training data pushed in the same direction. SFT on teacher answers cut calls by 24.2% with no bonus at all.

• Can this method train a hosted API model? No. The proxy sits in front of an inference engine you serve yourself, so the method requires open weights and cannot intercept hosted API models.

• How should that change how I read agent leaderboards? Ask which harness produced the score, and treat gaps under about two points as noise. In the Zhang et al. study, app choice moved results by 13 points while model choice moved them by 2.5 to 5 points.

• Which part of Claude Code causes the 33% score? The write-up does not isolate it. Overfitting to one app's format, context layout, and control flow is cited as likely, but no experiment in that study separates prompt size, history rewriting, or tool naming.

Turn a Spoken Walkthrough Into a Written Article

This article exists because one number in a video needed unpacking: 33.2% against 62.1% for the same weights, and the reasons behind that gap only became clear after the transcript was checked against the underlying write-up and two independent papers.

If you have that kind of explanation sitting in a recorded video, whether it is a technical walkthrough, an interview, or a lesson you keep repeating, paste the YouTube URL into Skala Blog. The flow is short: video, transcription, article you can edit before publishing.

Readers who want the wider context can also see the work at skalablog.com and crazystack.com.br, where Skala blog and the Dev doido community collect technical write-ups, including Crazystack typescript examples.

Source video