Why Did Programmatic Tool Calling Take 3 Years?

Programmatic tool calling replaces the JSON form-filling pattern of 2023 with a model that writes a program, calls its tools, and returns only the final answer. Anthropic, OpenAI, and Cloudflare all shipped versions of it in 2025 and 2026, while the generate-then-execute timing every harness inherited from JSON remains the next bottleneck.

Why every agent harness waits for the full generation

Harnesses wait because the waiting was inherited from JSON tool calling, where nothing could be overlapped. When a tool call is a JSON object at the end of the model's turn, generation is already complete by the closing brace, so there is no meanwhile in which to run anything. The habit then survived the move to code actions.

The origin date is specific. On 13 June 2023 OpenAI shipped function calling, where functions were described as JSON schemas and the model replied with one JSON object naming a function and its arguments. The format was simple, checkable, and easy to validate, which is why other labs copied it within a year. When Anthropic open-sourced MCP, the standard for connecting tools to models, the JSON shape became the universal plug.

The result is visible today: a harness can read six fully specified tool calls in the first seconds of a long generation and still holds all of them until the last token lands. Nothing in the code-actions world justifies that scheduling. As MIT researcher Alex Jiang put it on his blog in 2026, most harnesses wait for the entire generation to complete before executing tools, and that design is a consequence of JSON-style tool calling where waiting was not a bottleneck.

What JSON tool calling costs before any work happens

JSON tool calling charges two bills before the model does useful work: tool definitions crowding the context window and raw results routed through the model. Both costs scale with the number of tools, and both disappear or shrink when the action space becomes code.

The definition bill is the documented one. Anthropic published arithmetic showing five typical MCP servers with 58 tools consume roughly 55,000 tokens of definitions, and GitHub's official server alone carries 35 tools at about 26,000 tokens. Cloudflare's own platform reached 2,500 endpoints as tool definitions, about 1.17 million tokens, before its rebuild.

The second bill is architectural. With JSON, every result travels through the model: point an agent at a 10 MB log file and the whole file lands in context even if you wanted a count, and chaining two calls copies the first output through the model's attention purely to feed the second. Kenton Varda, who runs Cloudflare's developer platform, compared making a model work through tool-call wrappers to putting Shakespeare through a month of Mandarin classes and asking him to write a play in it, because a model has read vast amounts of real code but only ever met tool-call syntax in synthetic training data.

What the CodeAct evidence showed in 2024

CodeAct showed in 2024 that writing executable code is a stronger action space than filling JSON forms. The paper, Executable Code Actions Elicit Better LLM Agents by Xingyao Wang and co-authors, gave the model one action space, Python, instead of one per tool, with loops, conditions, and variables available at no extra cost.

Over 17 models the authors measured up to 20% higher task success than text or JSON actions. Independently, Hugging Face's smolagents library, which builds agents on code actions, reported about 30% fewer steps, which translates directly into 30% fewer model calls to pay for. These are different implementations measuring the same idea from two directions.

The argument held because code composes. A program can chain calls, transform intermediate results locally, and keep bulky outputs out of the context window, while JSON forces every intermediate value through the model. The gap between knowing the arguments of a call and needing its answer is also much larger in a program than in a single JSON object at the end of a turn.

Who shipped programmatic tool calling, and when

Three vendors shipped programmatic tool calling between September 2025 and July 2026, and they agree on the mechanism: the model writes a program, the program calls the tools in a sandbox, and only the answer comes back. The table below compares the three implementations on the dimensions that matter for adoption.

Anthropic made the move twice inside three weeks in November 2025. On 4 November its engineering post on code execution with MCP described collapsing 150,000 tokens of tool definitions to about 2,000 by executing code against MCP servers. On 24 November it shipped programmatic tool calling in the API, where Claude writes Python against your tools in a sandbox; on a 75-tool agent benchmark this cut input tokens by 38% with no loss of accuracy. Anthropic also published the case where its own feature loses: on benchmarks with one or two sequential tool calls per turn, scores stayed flat while cost rose roughly 8%.

Cloudflare shipped first. In September 2025 Varda's team released Claude Code, which converts tool schemas into a TypeScript interface and lets the model write TypeScript inside a sandboxed V8 isolate. Aimed at Cloudflare's own platform, 2,500 endpoints of tool definitions totalling about 1.17 million tokens became a two-function interface, search and execute, of roughly 1,000 tokens.

OpenAI closed the loop on 9 July 2026, shipping programmatic tool calling in its main API alongside GPT-5.6, using JavaScript instead of Python in its own V8 sandbox. The launch page carries a customer result from a finance research benchmark: matched quality with 24% fewer output tokens and tasks finished 28% faster on the wall clock, with no retrained weights. When three companies with different business models reach the same design inside a year, the honest reading is a correction, not a trend.

How speculative tool execution attacks the remaining wait

Speculative tool execution removes the generate-then-execute wait by running a tool call as soon as its arguments are fully parsed, before the program finishes generating. Alex Jiang, the MIT PhD student who described the inherited waiting problem in 2026, applied the trick after publishing recursive language models in December 2025, an inference strategy where a model treats a huge prompt as an environment and calls itself over pieces of it from a Python REPL, which makes submodel calls inside code exactly the expensive thing to overlap.

The mechanism borrows from CPU branch prediction, which has guessed ahead and discarded wrong guesses for decades. As tokens arrive, the harness parses the half-finished program and launches any call whose inputs are knowable, filing the result as a promise. A shadow interpreter, a deep copy of the real one, runs the partial program off to the side so the real state stays untouched; if the model's code turns out broken, the shadow is discarded and nothing happened. Purity is the rule: a literal argument fires the instant it parses, arguments built only from pure earlier values fire too, and anything touching the outside world is blocked and executed the slow, careful way by the real interpreter. The tool author's contract is two flags on a decorator: is this speculatable, and is it pure.

The measured payoff is small and honestly reported. Jiang benchmarked his own idea on the OLong long-context benchmark, on a node of eight H100s serving a 30-billion-parameter model through vLLM, five runs per configuration at two temperatures. His reported speedup is on the order of 1 to 1.2 times. In an animated demonstration on his page, six slow subcalls inside one turn run 2.4 times slower without speculation, but he treats that as a demonstration, not the benchmark, and notes the speedup depends on tool latency, serving load, and harness choices. Adjacent work arrived from several directions: a 2024 system called Conveyor cut request latency by up to 38.8% by executing tools partway through decoding, and a Berkeley group measured 1.7x this spring speculating tool calls for voice assistants. The programmatic case has the most room because by the time a JSON-emitting model has specified a call, its turn is essentially over.

What speculative execution costs and who bears the risk

Speculative execution costs extra load and introduces a side-effect problem that the technique's author cannot solve alone. Both need to be priced before production use, and one of them changes the question from performance to responsibility.

Load is the first cost. Speculation issues requests that may never be needed, and Jiang states the worst case plainly: a serving engine clogged with concurrent, possibly wasted, speculated requests. If the submodel runs on the same GPU generating the main context, aggressive speculation competes with the workload you are trying to speed up.

The sharper cost is not his. On 1 June 2026, four researchers published a paper called Ghost Tool Calls, arguing that a speculated call reaches an external service before the agent has decided to make it, and if the agent abandons that branch the request is still out there. Timing is the issue, not authorization, so read-only permissions do not save you: guess a search query on someone's behalf and you have told the search engine what they were about to ask. You cannot unsend what somebody already received.

The steel man for JSON also deserves space. A JSON tool call is legible, one object you can log, diff, replay, and hand to an auditor. Claude Code hands you a program in a sandbox where every tool result comes back as a string about to meet an interpreter, an injection surface JSON never had. Anyone running an agent that must be auditable has a real reason to keep the older format.

Should your agent switch to code actions today?

Switch when your agent chains multiple tools per turn or connects many servers; stay on JSON when each turn makes one or two sequential calls. Anthropic own numbers support both halves of that advice: 38% fewer input tokens on a 75-tool agent, but roughly 8% more cost for nothing on short sequential workloads.

A practical checklist before moving:

• Count your tool definitions. If they cost tens of thousands of tokens before the user speaks, the code path pays for itself quickly.

• Check whether your tools have observable side effects. If a stranger can see a speculated request, read the Ghost Tool Calls paper before enabling speculation near production.

• Match the sandbox to your stack. Anthropic OpenAI sandbox Python and JavaScript respectively in their APIs, Cloudflare runs TypeScript in a V8 isolate, and a harness that keeps generate-then-execute timing forfeits the scheduling gains.

• Treat benchmark percentages as configuration-specific. The 38%, 28%, and 1 to 1.2x figures come from different vendors and workloads, and none generalizes to every agent task.

The deeper lesson survives the numbers: same weights, different loop, different result. An assumption was copied from harness to harness for three years, and it came from a format nobody uses for this anymore.

FAQ

• What is programmatic tool calling? It is an API design where the model writes a program that calls your tools directly in a sandbox, instead of emitting one JSON object per call. Anthropic, OpenAI, and Cloudflare shipped implementations between September 2025 and July 2026, with only the final answer returning to the model.

• How much does programmatic tool calling save? Vendor-reported figures vary by workload: Anthropic measured 38% fewer input tokens on a 75-tool agent benchmark, Cloudflare collapsed 1.17 million tokens of tool definitions to about 1,000, and an OpenAI customer matched quality with 24% fewer output tokens and 28% faster task completion. On one-or-two-call turns Anthropic reported about 8% higher cost.

• Why did harnesses keep waiting for the full generation? Because JSON tool calling put every call at the end of the turn, where there was nothing to overlap, so waiting was free. When code actions arrived, the generate-then-execute schedule was copied along with everything else, even though a program names its first call almost immediately.

• Is speculative tool execution safe for production? Not without care. Speculated requests consume serving capacity that may be wasted, and a 2026 paper, Ghost Tool Calls, shows a speculated call can reach an external service the agent never committed to contacting, which read-only permissions do not prevent. Restrict speculation to pure, side-effect-free calls.

• Is JSON tool calling obsolete? No. It remains legible and auditable, one loggable object per call, and Anthropic own benchmarks show code actions add cost with no benefit on short sequential turns. The correction is real for many-tool, multi-step agents, not universal.

Turn your own video deep-dives into articles

This piece started as a video transcript about an inherited scheduling mistake, and the most valuable part of it, tracing a design decision back to its origin and checking who has since measured what, is exactly what most recorded talks contain and few transcripts preserve. If you have interviews, explainers, or technical deep-dives sitting in YouTube videos, that knowledge deserves a written form that search engines and readers can actually use.

Skala Blog turns a YouTube URL into a transcribed, structured, publishable article. Similar transcript-driven breakdowns appear on Dev doido and on Crazystack typescript, and the flow is simple: paste the video link, get the transcription, generate the article, then edit before publishing.

Source video