Continuously improving AI agents rests on two halves: online tracing and monitoring, and offline datasets, experiments and evaluation. Get the first half right and even teams in low-risk settings can iterate quickly, because every production trace becomes evidence for the next fix.
What Does Continuously Improving AI Agents Actually Mean?
Continuously improving AI agents means closing a loop between production observation and structured testing: you trace what the agent does, monitor those traces for failures, and feed what you learn back into datasets and experiments. The Langfuse workshop frames this as the AI engineering loop, split into an online half and an offline half.
Lotte Verheyden, who works at Langfuse, an open-source, model-agnostic and framework-agnostic platform for LLM observability, opened the session with the core difference from traditional engineering: agents are non-deterministic. A system can report healthy status while its responses are wrong or simply not good enough. Dashboards that track uptime will not catch that.
The loop has five steps grouped in two parts. The online part is tracing and monitoring. The offline part is building datasets, running experiments and evaluating changes before deployment. Verheyden's practical advice for teams starting out: the online half alone is already a lot. Teams not in high-risk industries can begin with tracing and monitoring, make simple fixes from what they find, deploy, and watch new traces arrive. The offline half becomes worthwhile once the product matures or mistakes carry real consequences.
Why Does Trace Quality Decide Everything Else?
Trace quality decides everything downstream because a bad trace hides failures that monitoring, datasets and evaluation would otherwise catch. The workshop's blunt framing: if your trace is bad, you will not catch everything during monitoring, and your datasets will be worse than they could be.
A trace consists of observations, which are the individual steps an agent takes while producing an answer. Observations include LLM calls, tool calls and other steps, and they can be nested. Each observation carries data appropriate to its type:
- For an LLM call: duration, token usage, cost, the model used and the prompt version.
- For a tool call: type-specific data, for example the query sent to a web search.
- For the root observation: the full conversation history the agent produced.
In the live demo, the sample application was a TypeScript app built on the OpenAI Agents SDK, OpenAI's toolkit for building agents that can call tools and models. Every user question produced a trace rooted in an agent observation, with generations and tool calls nested beneath it. Verheyden noted that Langfuse also ships a skill that can instrument an application automatically, so teams do not have to wire tracing by hand.
Which Evaluators Should Monitor a Support Agent in Production?
A support agent needs evaluators matched to its own failure modes, not generic templates copied from someone else's chatbot. Annabell Schäfer built three evaluators live, each targeting a specific question, and each scoring only the part of the trace that answers it.
Monitoring serves three functions in the workshop's model. Aggregate metrics show whether quality and cost hold over time. User feedback surfaces the few traces worth reading once volume makes manual review impossible. Automatic evaluators and human annotation catch new failure patterns, with a trade-off: evaluators scale to all traces but rarely spot cases you did not anticipate, while human review scales poorly but finds the unknown unknowns.
The three evaluators from the demo compare like this:
| Evaluator | Type | What it targets | Question it answers |
|---|---|---|---|
| User disagreement | LLM-as-a-judge | Agent root: full message history plus last user message | Did the user say the answer did not work? |
| All-caps frustration | Code-based (Python or TypeScript) | Last user message | Is the user typing in caps, a frustration signal? |
| Out-of-scope request | LLM-as-a-judge | Generation: system prompt plus last user message | Is the user asking for something the agent is not scoped to do? |
Each evaluator uses explicit variable mappings through JSON paths, so the prompt receives exactly the messages it needs and the score answers a defined question. The out-of-scope check matters for product decisions: when users consistently ask a printer question of a phone-support agent, the finding tells you either to expand the agent's scope or to fix how you communicate what it does. Schäfer's closing point on the exercise: the value comes from thinking through your application's specific failure modes, not from copying what exists.
All of this is vendor-demonstrated workflow from the workshop itself, shown on seeded and live traces during the session rather than measured in an independent study.
How Can a Coding Agent Audit Hundreds of Production Traces?
A coding agent can audit production traces at scale through the Langfuse CLI and its companion skill, which expose the platform's APIs: traces, scores, datasets and experiment runs, all readable and runnable from the command line. In the demo, Schäfer installed the skill into Claude, Anthropic AI assistant, and into Codex, OpenAI's coding agent, then asked it to find issues in the traces.
After seeding roughly 70 to 120 traces into the project, the coding agent produced a report with findings that went beyond the evaluators the team had configured. According to the presenter's live demonstration:
- A silent retrieval-quality failure in the search-help-library tool, which was returning poor lookups nobody had evaluated for.
- Retrieval misses driving extra tool retries, which increased latency and cost.
- An evaluation coverage gap, where only a subset of recent traffic produced any score signal.
- A prompt-version mismatch in the traces.
The next step was a proposed code change: since the help-library tool was not helping, the coding agent, with repository context, drafted a fix in the application code itself. Schäfer's caveat deserves repeating for anyone doing this: the human stays in the loop. The coding agent accelerates the investigation, but you decide which findings matter and whether the proposed change is right.
How Does Monitoring Feed the Offline Improvement Loop?
Monitoring feeds the offline loop by turning observed failures into representative datasets, which then power experiments, regression checks and confident deployments. The interesting traces you surface online become structured test cases offline.
The workshop's closing argument: this is not a one-off project. Datasets need maintenance because early assumptions about user behavior break as edge cases accumulate. Evaluation criteria need the same care, or scores stop telling you anything useful. When the dataset reflects real usage, a proposed change gets tested against it, results get evaluated, and only changes without regressions ship. The presenters recommend making that check part of CI/CD so every deployment is validated automatically.
The full Langfuse workshop is an eight-step guided sequence at langfuse.com/workshop; the session covered only the tracing and monitoring modules. The remaining steps include prompt management and dataset setup, and the presenters' recommended starting order if you begin tomorrow: use the skill to add tracing, review traces manually to understand real usage, then build toward offline optimization.
Frequently Asked Questions
- What is the AI engineering loop? It is the set of processes for improving non-deterministic AI systems, grouped into online tracing and monitoring and offline datasets, experiments and evaluation. A system can look healthy while producing wrong answers, which is why the loop exists.
- Do I need the offline part of the loop on day one? No. The workshop's guidance is that teams outside high-risk industries can start with tracing and monitoring alone, fix what they find, and iterate. Add datasets and experiments once the product matures or mistakes carry serious consequences.
- What is the difference between LLM-as-a-judge and code-based evaluators? LLM-as-a-judge evaluators read conversation history and score implicit qualities like user disagreement. Code-based evaluators run deterministic checks, such as detecting all-caps input, and can be written in Python or TypeScript.
- Can a coding agent really find agent failures on its own? It can investigate traces at scale through the Langfuse CLI and skill and surface findings such as silent tool failures, retry-driven latency and evaluation coverage gaps, but the workshop keeps a human in the loop to judge which findings matter and review proposed fixes.
- What does a trace need to contain to be useful? Nested observations for every step, with type-appropriate data: duration, token usage, cost, model and prompt version for LLM calls, and inputs like search queries for tool calls. Weak traces hide failures from every later step.
Turn Your Own Conference Talks Into Searchable Knowledge
A 44-minute workshop holds more practical detail than any summary can carry, from JSON-path variable mappings to the exact order of evaluator setup. That knowledge stays locked inside the video unless someone turns it into writing.
If you have talks, tutorials or deep dives sitting on your own YouTube channel, Skala Blog converts them into structured articles: paste the YouTube URL, get a transcription, and generate a publishable draft. Your best explanations deserve to be findable, not just watchable.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits