How to Cut AI Inference Costs Without Killing Output

Cut AI inference costs with four moves from Featherless AI's CEO: measure spend per team, right-size models, buy capacity well, and fix processes.

How to Cut AI Inference Costs: The Four Moves

You cut AI inference costs with four moves in a fixed order: measure token spend, right-size models to each workload, buy capacity for your real traffic pattern, and fix the human processes that waste generated work. Eugene Cheah, CEO and founder of Featherless AI, an inference platform hosting over 40,000 open models, presented this framework at AI Engineer NYC in 2026.

Cheah is also part of the team behind RWKV, the AI architecture project that joined the Linux Foundation as its first AI model project. His figures below come from his own workshop slides and customer work, so treat them as vendor- and speaker-reported, not independent benchmarks.

Two disclaimers frame the talk. First, Featherless sells inference, but the moves apply to any provider. Second, the workshop targets organizations spending over roughly $10,000 a month. Cheah's advice below that line: find a strong use case first, because one good use case is worth more than optimizing a small bill.

Why Measuring Token Spend Comes First

Measuring comes first because most organizations cannot see their own spending. Cheah reported that 30-40% of customers who reach out to Featherless started with a single shared API key used by the whole company. At that point, per-department or per-use-case accounting is impossible.

The fix is unglamorous: issue separate API keys per department, per team, and per major use case. Once keys are split, you can answer questions that drive every later decision, such as which team spends the most and whether a workload actually has a human waiting on the response.

Measurement also exposes the two workload tracks with opposite economics:

• Agentic workloads: long-running, multi-turn jobs such as coding agents that maximize context. Cheah's observation across deployments is that only about 1% of agentic request traffic is output tokens; input and cached input dominate, and repetition drives the bill.

• Process automation: high-volume, task-isolated jobs such as call-center routing, email filtering, and document processing. One customer filtering messages for a child-safety feature peaks at 10 million messages per minute on New Year's Eve, per Cheah.

Cheah's worked example: a five-developer team ran about eight agents continuously, burning roughly 116 billion tokens and $150,000 a month on a frontier model through the public API. Moving the same workload to a cheaper open model on a single dedicated node cut the cost to under $10,000 a month with no reported quality disruption. Scheduling background agents to run overnight also let the hardware serve daytime traffic better. He also mentioned a tokenomics tracking tool Featherless planned to open-source for exactly this key-splitting purpose.

Right-Sizing: Open Models, Classifiers, and Fine-Tuning

Right-sizing means matching the cheapest adequate model to each workload instead of defaulting to the biggest one. The payoff is cost and latency you give up nothing important for. Cheah's rule of thumb, dated to 2026: the gap between the best closed and best open models has narrowed to the point where open models now match or beat closed ones on narrow, specialized tasks.

Which open models fit which tasks?

Cheah cited examples from his customer work rather than a single leaderboard. Before one frontier model's update, an open Kimi-class model beat it on UI-design benchmarks. A StepFun model reached near-frontier quality on financial benchmarks despite being far smaller. A DeepSeek flash variant proved popular across many workloads because it is easy to prompt-steer, even though benchmark scores alone did not explain its popularity. Names and rankings shift every few months, so verify current results before committing.

What about classifier models?

Cheah highlighted a newer model class he called oneshot classifiers, focused purely on decisions and classification rather than chat. Because they produce structured decisions, they are commonly around 10x cheaper and faster than general chat models. His open-source simple JEF project, which replicates a classifier API on existing open models, drew community attention; teams found a Qwen 27B model performed well at the task, and a prompt-tuned variant beat most fine-tuned rivals in Hugging Face scaling tests, per Cheah. The limitation: classifiers cannot produce free-form chat responses.

When is fine-tuning worth it?

Cheah pointed to published AT&T work: fine-tuned models in the 27B and 7B range replaced much larger models for call-center and telco-specific tasks like routing, with traffic escalated by classifiers when needed. Shopify fine-tuned for its proprietary web-store configuration language because existing models handled it poorly. The standard bootstrapping path:

1. Run the expensive frontier model at small scale.
2. Generate a dataset from its outputs, checking terms of service (Google's Gemini offers a dedicated distillation API for this).
3. Fine-tune smaller open models and split traffic between them.
4. Keep a slice of shadow traffic on the best model so your dataset keeps improving as frontier models do.

For coding specifically, Cheah is skeptical of fine-tuning: developers want the best model, and fine-tuned coding models go stale fast. Domain models are where fine-tuning pays.

Choosing between the three approaches

Approach	Best for	Relative cost	Key limitation
Open models	Narrow, specialized tasks where open now rivals closed	Moderate	Rankings shift every few months
Oneshot classifiers	Decisions and classification, not chat	~10x cheaper and faster	No free-form responses
Fine-tuning	Domain-specific tasks like routing or proprietary languages	Low after training investment	Coding models go stale fast

Buying Capacity: The Speed-Cost Tradeoff

Faster inference is not a linear expense: buying capacity is really a decision about how much latency premium your workload justifies. Every doubling of per-request speed multiplies cost disproportionately, while batching many slower requests slashes per-token price. This section explains that tradeoff, gives target speed ranges by use case, and shows when dedicated capacity wins.

GPU capacity does not scale linearly with speed. Cheah's rule of thumb: every 2x in per-request tokens-per-second costs about 3x more, and it compounds. Pushing one request to thousands of tokens per second on all eight GPUs at batch size one is possible, but you pay heavily for it; vendors like Cerebras sell exactly that extreme, at a premium.

The arithmetic runs the other way too. On a GLM-class model, Cheah's example showed a single request at 112 tokens per second on one server. Running 16 requests in parallel halved each request's speed but raised total throughput roughly 10x, cutting per-token cost by roughly 9-10x. Changing the target speed profile, not the hardware, produced the savings.

His guidance on target speeds: most customers pick 20-30 tokens per second, roughly ChatGPT-class responsiveness, up to 60 for faster feel. Speeds of 120 belong only in front of a human who is actively waiting. Background jobs and CI/CD pipelines should run slower and cheaper; a midnight background agent does not need premium latency.

Workload	Target speed	Cost posture
Background jobs, CI/CD	slower, low priority	cheapest per token
Interactive chat	20-30 tokens/sec (up to 60)	balanced
Human actively waiting	up to 120 tokens/sec	premium

Once measurement reveals your traffic pattern (usually bursty, with a base load), dedicated instances become attractive for the base load, with spikes overflowing to public cloud where policy allows. Cheah's largest cited customer serves about 2 trillion tokens a day on 512 AMD MI325 GPUs at around $0.03 per million tokens on a private deployment, a price he says no public cloud listing matches. These are vendor-reported figures for a specialized setup. Financial institutions that require fully dedicated instances have less flexibility, which limits this lever for them.

Fixing Processes: Slop Grenades and Outcome Measurement

The last move has nothing to do with inference. Organizations 10x their token spend and see maybe 6-7% productivity gains, per reports Cheah cited. His explanation: AI is a hammer being applied to everything, and the bottlenecks are organizational.

What is a slop grenade?

A slop grenade is generated output, code, slides, reports, handed off for someone else to verify. Cheah showed his own AI-designed workshop slides, which he personally revised dozens of times, as the counter-example: own every word you generate. The code version is common: an agent produces a quarter-codebase change, the human reviewer cannot process it, and a second agent rebuilds the change by hand, discarding the first agent's work. Review capacity, CI/CD throughput, and trust (tests, validation) become the real limits.

Cheah's guidelines: enforce ownership of generated output, measure outcomes rather than usage, and treat a formula like tokens × price ÷ used outcomes as a directional tool, not gospel. His framing of AI as a supplement fits here. It amplifies teams that already work well and cannot rescue weak processes. Sometimes the highest-impact change is mundane, the way Walmart's truck step-stools cut injury costs more than any exotic initiative.

Coding Agents, Lock-In, and Switching Costs

When your workflows live inside a coding agent like Claude Code, you cannot swap the model directly — Cheah calls this the fully locked-in case. Instead, the migration target becomes the harness itself: teams control spend by moving to a lighter, model-agnostic setup while keeping a smarter planner in the loop.

Cheah described this transition as a repeatable pattern:

1. Keep Claude Code (or a similar capable agent) as the planner for architecture and review.
2. Delegate implementation to a fast, cheap model through a lighter harness such as the pi coding agent, which he says achieves lower cost per token on the same model because of its efficiency.
3. Let the planner review the cheap model's output before merging.

Automation-first harnesses like AMP force this split by design, escalating to a smarter model only when the task demands it.

Approach	Model flexibility	Cost behavior
Claude Code alone	Locked to one model	Highest spend, no swap possible
Lighter harness (e.g. pi)	Model-agnostic	Lower cost per token on the same model
Automation-first (e.g. AMP)	Split by task	Smart model used only when needed

Incentives matter more than policy. Teams where both models ran at the same tokens-per-second never switched to the cheaper one; teams with tiny budgets burned the expensive model in a day and abandoned it. The stable configuration pairs a smarter planner with a cheaper model that is also faster, because people switch for speed, not price.

A side note on auto-routing: Cheah prefers task- or department-level routing over per-turn routing. Per-turn routing across more than two or three models destroys cache hit rates and wrecks the economics.

FAQ

• What is the first step to cut AI inference costs? Split your API keys by department, team, and use case, then measure spend per key. Cheah reported that a large share of his customers began with one shared key and no way to attribute spend. Every later optimization depends on this data.

• Are open models good enough to replace frontier models? On narrow, specialized tasks, increasingly yes, per Cheah's 2026 customer work and public benchmark trends he cited. For general-purpose frontier performance, closed models still lead. Match the model to the task with your own evals rather than trusting leaderboards.

• What tokens-per-second should I target? Around 20-30 tokens per second for interactive use, up to 60 for a faster feel, and much less for background jobs. Faster speeds cost disproportionately more; Cheah's rule of thumb is roughly 3x cost per 2x speed.

• Is fine-tuning a smaller model worth the effort? For domain-specific, high-volume workloads like AT&T's call-center routing, Cheah says yes, with reported drops from frontier-class to 7B-27B models. For coding, he advises against it because developers want the best available model and fine-tunes go stale quickly.

• What is a slop grenade? Generated output dumped on someone else to review or fix. Cheah's prevention rule is cultural: the person who generates output owns it, and teams should be measured on outcomes used, not tokens burned.

Turn Your Own Token Talk Into a Written Playbook

Cheah's workshop worked because a dense hour of talk became four ordered, checkable moves. That knowledge is exactly the kind that stays trapped in a conference video, useful to the room and invisible to everyone else. Teams in communities like Bootcamp do Dev Doido and the Crazystack Typescript crowd around Gustavo Dev Doido face the same problem with their own talks, retros, and deep dives.

If you have a video full of hard-won lessons on AI inference costs or anything else, Skala Blog turns it into a structured written article. Paste the YouTube URL, get the transcription, and generate a publishable draft you can edit and share.

Source video