The agent harness, not the frontier model, is where engineers can still make real gains in autonomous AI systems. In a 2026 AI Engineer workshop, Oracle developer advocate Nacho Martinez showed how memory layers, context engineering, and skill retrieval let an AML agent flag suspicious transactions around the clock without blowing up its context window.
What Is an Agent Harness?
An agent harness is the scaffolding you build around a language model to give it tool use, memory, and perception. The model supplies the reasoning, but the harness supplies everything else — the loop, the memory, and the tools — and how it is configured can change the same model's results dramatically.
Martinez frames the stack in layers: the application you interact with, the model as the reasoning core, and the infrastructure beneath. The data layer in between, covering the agent loop, gateway, MCP connections (the protocol released by Anthropic 2024), memory, context engineering, and tools and skills, is the only part engineers fully control, especially when consuming a third-party model.
His core argument is that frontier reasoning comes from the labs and offers little room for customization. The harness controls everything else, and the same model can perform better or worse on a task purely because of how the harness is configured. A harness typically spans seven or eight decision areas: model choice, storage, encoding and retrieval, memory compartments, the semantic layer of unstated context, the agent loop, and context engineering.
AI agents themselves he defines as computational entities powered by an LLM for reasoning, augmented by a database for memory, extended through tools for actions, and grounded in inputs. Systems such as Claude Code, Anthropic terminal-based coding agent released in 2025, combine LLM-driven workflows for automation with agents for autonomy. Martinez calls these hybrids agentic systems and argues most engineers use them daily without thinking about the same problems when building their own applications.
The layers break down as follows:
| Layer | Role | Who controls it |
|---|---|---|
| Application | The interface you interact with | Harness builder |
| Model | The reasoning core | Frontier labs |
| Data layer (loop, gateway, MCP, memory, tools) | Connects and orients the model | Engineers, fully |
Why the Context Window Limits Autonomous Agents
The context window is the model's short-term memory, and it is the binding constraint on how autonomous an agent can be. Accuracy drops as the window fills, tools and skills eat much of the available space, and harnesses differ wildly in how much they spend before the agent does any real work.
Martinez is blunt about the tradeoff: the larger the attention matrix, the worse the model performs. Studies he cites show accuracy is much higher starting near zero tokens than with 256,000 tokens already loaded, even when the advertised limit is a million. Smaller context means better problem-solving across every problem type he has tested.
This matters because tools and skills routinely consume more than half of the context. He reports having around a hundred skills installed, each worth roughly 3,000 to 5,000 tokens, which would overflow the window on its own. He also cites a paper comparing harness efficiency: the Pi harness starts each request with about 6,000 tokens, while others begin at 30,000 to 40,000. These are speaker-reported figures from the workshop, not independently verified measurements.
Two techniques manage the overflow, and they work best alongside careful skill hygiene:
| Approach | Mechanism | Effect on the working window |
|---|---|---|
| Context summarization | The model compresses the conversation into a short summary and restarts from it | Shrinks history to a small, lossy digest |
| Compaction | Conversation chunks are stored in a database or file with an ID, pulled back by reference on demand | Keeps history out of the window but retrievable |
| Minimal skill loading | Keeping only necessary skills installed | Frees 3,000–5,000 tokens per removed skill |
Both techniques keep the working window small while preserving access to history, which is what lets an agent act over long horizons without degrading.
Files or Databases for Agent Memory?
The files-versus-databases debate misses the point, according to Martinez. His position is that each solves different problems. Files handle unstructured data well. Databases handle concurrency, and the multi-agent world is rediscovering problems databases solved decades ago.
His example is multiple agents writing to the same file, a situation coding agents currently work around with git worktrees. With atomic, consistent, isolated, and durable (ACID) writes, that consistency problem disappears. Reinventing transactional semantics with files is unnecessary work.
He adds a performance caveat for Firebase skill storage: regular-expression matching degrades roughly quadratically as files grow, while precomputed embeddings capture meaning that keyword search cannot. Hybrid retrieval, combining semantic vector search with keyword search through reciprocal rank fusion, retrieves more relevant results than either method alone, at the cost of a bit more compute.
Designing Short-Term and Long-Term Memory
Martinez's first recommendation is to keep the memory layer simple. Start with exactly two structures: short-term memory, which is the context window itself, and long-term memory. When something in the window is worth keeping, promote it to long-term storage as raw text, then vectorize it so the agent can search it later.
Within long-term memory he proposes three categories. Episodic memory holds conversation episodes, tool logs, and summaries of past sessions. Procedural memory holds skills and workflows, which are sequences of skills. Semantic memory holds facts about the user: identity, preferences, and favored frameworks.
Retrieval then becomes an extra step between observing and reasoning. The agent queries the vector store, compares vector distance, and pulls an episode into context only if the distance is small enough to be relevant. If relevance is weak, it discards the memory and solves the problem fresh.
How the Toolbox and Skillbox Patterns Save Context
The toolbox and skillbox patterns save context by storing tools and skills outside the prompt and retrieving only the few most relevant items for each step. The workshop architecture vectorizes both into two separate vector stores; the agent loop runs a similarity search with k equals 3 against each store, retrieves the three most relevant tools and skills, solves the problem, and offloads them again. The workshop implementation skips a reranker, though adding one to rescore the three candidates is an optional refinement.
This matters because of the instruction hierarchy in chat completions API calls. System prompts rank highest, then developer instructions, then the user query, with tool outputs and retrieved documents attached at the bottom. Martinez estimates that tools and skill outputs make up about 80 percent of a typical request, sometimes 35,000 to 100,000 tokens. Providers such as OpenAI mitigate the cost with a semantic cache that avoids recomputing repeated content, which he says keeps requests roughly four to five times cheaper than uncached equivalents. That figure is his own estimate from the session.
The agent loop itself drives everything: context assembly pulls skills, tools, files, and prior conversation dynamically, the model reasons and decides, and the cycle repeats. The workshop loop caps itself at eight iterations and 60 seconds of reasoning per iteration, a deliberate limit to control API spend. He notes that truly autonomous long-horizon agents need near-infinite patience plus automatic summarization instead.
How the retrieval cycle works
- Assemble context. The loop searches the tool and skill vector stores with k equals 3 and pulls in files and prior conversation.
- Reason and decide. The model works on the problem using only the retrieved items, within the 60-second reasoning cap.
- Offload. The tools and skills return to their stores, keeping the next request's prompt small.
- Repeat or stop. The cycle runs for up to eight iterations, or until the problem is solved.
What stays in context versus what is retrieved
| Item | Role in context | Typical cost driver |
|---|---|---|
| System prompt | Highest-ranked instructions | Fixed per request |
| Retrieved tools and skills | Bottom-attached, swapped per step | About 80 percent of a request, 35,000 to 100,000 tokens |
| Semantic cache hit | Avoids recomputing repeated content | Roughly four to five times cheaper than uncached |
| Iteration cap | Eight iterations, 60 seconds each | Bounds total API spend |
In 2024, stuffing every tool and skill into the system prompt was the common default, which pushed requests toward the upper token ranges above. The 2025 workshop approach treats retrieval as the default instead, so the context each call carries shrinks to what the current step actually needs — and, as of 2025, caching and retrieval together are what make that affordable.
Skill Distillation and Continual Learning for a Frozen Model
A frozen model can still improve over time through the data layer. Skill distillation is the simplest technique: mine previous conversations and runs, extract what worked, and write it into a skill file placed back in the skillbox. Because skills carry embeddings, the agent automatically retrieves the refined skill next time a similar task appears, such as a front-end task matching a stored coloring-style preference.
Martinez places this work inside three categories of continual learning. Parametric memory changes the model's weights through fine-tuning or low-rank adaptation. Representational memory works on embeddings. Non-parametric memory is everything the harness stores and retrieves, and it is the category his workshop covers. He points to a free one-hour course on building adaptive AI agents produced with Andrew Ng and DeepLearning.AI at DeepLearning.AI for the other two categories.
He and a colleague are also experimenting with pluggable skill and tool hubs: precomputing embeddings for third-party collections of MCPs so an agent can vector-search an external catalog instead of loading it wholesale.
The AML Agent Case Study
The workshop applies the harness to a medium-sized bank that must flag suspicious transactions across five categories, such as a million-dollar deposit followed by three days of small ATM withdrawals. The agent runs autonomously in the background 24/7 on Oracle Cloud Infrastructure, using what Martinez describes as a Groq-hosted model 4.3, connected through the harness to the bank database.
Crucially, the agent does not block accounts. It flags suspicious activity and produces suspicious activity reports for human inspection, saving analysts time during discovery while leaving decisions with people. The workshop UI visualizes the agent's context window, the database of customers, tool-call traces against knowledge stored in vector stores, and the 30 most recent bank events on fake data.
His advice for builders: do not try to write the next Claude Code. Simplify the problem and build a narrow harness for it, then keep the model as a swappable plugin behind the chat completions API. Consultants are his ideal audience, since each customer's data and constraints justify a different harness configuration.
FAQ
- Why not just put everything in the context window? Accuracy degrades as the attention matrix grows, and tools plus skills can consume more than half of the window. Retrieving only what a task needs keeps the model near its best performance range.
- What does MCP solve for database-connected agents? Pre-created, verified SQL queries. SQL dialects differ across Postgres, Oracle Database, and other engines, and agents inventing queries dynamically often produce ones with engine-specific subtleties that fail.
- What is hybrid search and why combine methods? It merges semantic vector search with keyword search using reciprocal rank fusion. Results in the overlap retrieve better than either method alone, for slightly more compute.
- Can a harness make the same model perform better? Martinez says yes: identical frontier models perform better or worse on specific tasks depending on harness configuration, which is why he treats harness design as the highest-leverage engineering work available.
Turn Your Own Conference Talks Into Reference Articles
Martinez packed a year of harness engineering into one workshop session, and most of it exists only as a video recording. Talks like this one hold exactly the kind of architectural detail teams search for later and rarely find in writing.
If you have recorded talks, interviews, or walkthroughs of your own, Skala Blog turns a YouTube video into a structured article: paste the URL, transcribe the video, and generate a draft you can edit and publish. The knowledge stays yours; it just becomes searchable.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits