5 LLM Observability Metrics for AI Apps

LLM observability adds cost, safety, and quality metrics to the golden signals, because a 200 OK response does not prove an answer was correct. Marina Petzel, a GenAI advocate at Datadog, argued this at the AI Engineer World's Fair in San Francisco in 2026.

Why LLM Observability Needs More Than Golden Signals

LLM observability is the practice of monitoring generative AI applications for cost, safety, and output quality in addition to the traditional golden signals of latency, errors, traffic, and saturation. The same prompt can produce a different answer on every run, so a 200 OK response can still carry a hallucination. Teams need continuous quality and safety evaluation in live traffic, not just uptime checks.

Marina Petzel, a GenAI advocate at Datadog, outlined this gap in a recorded session at the AI Engineer World's Fair 2026 in San Francisco. Her core argument is that the golden signals remain necessary but describe only whether a system is up, not whether its answers are correct, safe, or affordable.

Traditional monitoring assumes deterministic behavior and predictable compute spend. Generative applications break both assumptions. A response can pass every infrastructure check and still be wrong, unsafe, or unexpectedly expensive.

How Token Creep and Model Drift Inflate AI APP Spending

Token creep and model drift are the two most common drivers of unexpected LLM cost increases. Token creep happens when context windows grow quietly during quality tuning; model drift happens when a team switches to a more capable and expensive model without adjusting usage. Both can multiply monthly spend without changing request volume.

Petzel describes the pattern where a context window grows from roughly 4,000 tokens to 132,000 without a financial review. The exact multiplier she cites is up to 8x cost increase from the context expansion alone, and she attributes the model-switch example to a Jump from a cheaper model to a more powerful one.

The point is not that either change is wrong. It is that neither change appears in a latency dashboard. Without per-request cost attribution, the invoice becomes the first place a team learns that quality tuning shipped with a price tag.

How to Attribute LLM Cost by Feature, User, Model, and Endpoint

Cost attribution for LLM applications works by tagging every request across four dimensions: feature, user, model, and endpoint. Datadog's guidance, presented by Petzel, describes this as a four-level strategy because a single blended cost number cannot tell a product team which surface is consuming the budget.

The first level tags the feature or intent such as chat, summarization, or extraction. This supports product decisions because it shows which capabilities consume the most budget rather than which are most visible.

The second level tags the user ID or organization name. This enables chargebacks and accurate billing, and it exposes abusive usage patterns before they drain the budget.

The third level tags the model and its provider. This gives visibility into which model tier drives spend and allows a true cost comparison between providers.

The fourth level tags the region and the environment, such as production or staging, so infrastructure planning can distinguish budget consumed by real users from budget consumed by tests. Petzel also names uncached calls as a third cost driver, citing 70% of spend sometimes being redundant when the same expensive completion is paid for repeatedly.

Safety Metrics: Prompt Injection, PII, Toxicity, and Jailbreaks

Safety monitoring for generative AI covers four measurable risks: prompt injection, PII leakage, toxic or biased output, and jailbreak attempts. Each one requires its own detector and threshold because none of them produces an error code that standard infrastructure monitoring would catch.

Prompt injection rate tracks attempts to manipulate the model through user input. Petzel recommends aggressive thresholds and detection via pattern matching or specialized classifiers.

PII detection rate scans model outputs for sensitive data such as social security numbers or credit card numbers using regex or named entity recognition. She calls this a zero-tolerance metric and sets the acceptable production target at 0% of outputs.

Content moderation scoring runs every output through a toxicity classifier to flag harmful or biased content, with thresholds set to minimize exposure rather than to eliminate it entirely.

Jailbreak attempts are persistent efforts to override the system prompt through elaborate scenarios. Petzel treats these as an absolute blocker, with the goal of blocking 100% of detected attempts.

Quality Metrics: Hallucination, Relevance, and RAG Retrieval

Quality monitoring for LLM applications rests on five metrics: hallucination rate, relevance, user satisfaction, answer completeness, and RAG retrieval quality. These measure whether the answer was useful, which is the question a 200 status code cannot answer.

Hallucination rate measures the percentage of responses containing claims unsupported by grounded data. Petzel recommends combining automated fact checks with manual review to reach aggressive targets.

Relevance scoring checks whether the response actually addressed the user's question, typically through embedding similarity between the query and the answer.

User satisfaction comes from direct signals such as thumbs up, thumbs down, star ratings, or NPS. Petzel sets a target of at least 80% to 85% positive feedback.

Answer completeness is often evaluated with an LLM-as-judge that decides whether the response fully addressed user intent. Retrieval augmented generation systems add a fifth metric: whether the documents fetched were actually relevant to the query, measured with metrics such as top-K accuracy or normalized discounted gain. As context size grows, selecting the most relevant document becomes harder, which makes retrieval quality worth tracking on its own.

Frequently Asked Questions

• What does LLM observability mean? LLM observability is the monitoring of generative AI applications across cost, safety, and output quality in addition to traditional uptime signals. It exists because non-deterministic models can return different answers to the same prompt, so an HTTP 200 response does not prove the answer was correct, safe, or affordable.

• Why are the golden signals not enough for generative AI? Latency, errors, traffic, and saturation describe whether a system is available and responsive. They cannot detect a confident hallucination, a PII leak in a completion, or a cost overrun caused by an expanded context window. Petzel's framing is that these metrics tell you the system is up, not how well it is running.

• What is token creep? Token creep is the gradual expansion of a context window during quality tuning without a corresponding financial review. Petzel gives the example of a window growing from about 4,000 tokens to 132,000 and cites an associated cost increase of up to 8x.

• What is model drift in LLM cost monitoring? Model drift here means switching from a cheaper model to a more capable and expensive one. The spend increases even when request volume stays flat, which is why cost attribution should tag the model and provider on every call.

• What is a safe target for PII detection rate? Petzel describes PII as a zero-tolerance issue and sets the acceptable production target at 0% of outputs. Detection typically relies on regex rules or named entity recognition over every model response.

• How do you measure hallucination rate? Hallucination rate is the percentage of responses containing claims unsupported by the grounded source data. Practical measurement combines automated fact-checking with manual review, because automated checks alone miss context-dependent errors.

• What is a reasonable user satisfaction target for AI apps? Petzel suggests aiming for at least 80% to 85% positive feedback, collected through thumbs up or down, star ratings, or NPS. Below that range, the quality problem is usually visible to users before it appears in any infrastructure metric.

• Which RAG metrics should be tracked? Track whether retrieved documents were relevant to the query, using metrics such as top-K accuracy and normalized discounted gain. Retrieval quality matters because larger context windows make it harder to isolate the most relevant passage.

• How does this connect to TypeScript developer education? Teams building generative AI features in TypeScript face the same observability gap described here. Resources such as Crazystack Typescript and communities like Gustavo Dev Doido's Bootcamp do Dev Doido cover modern stack practices, and Skala Blog converts recorded technical talks into written articles.

Key Takeaways and Related Learning Resources

The practical takeaway from Petzel's session is that generative AI monitoring adds three layers on top of the golden signals: cost, safety, and quality. Each layer needs its own metrics, thresholds, and tagging. None of them is optional once an application reaches production users.

For teams working in TypeScript and modern web stacks, the same discipline applies when AI features ship inside larger products. Crazystack Typescript is a Portuguese-language learning platform covering full-stack TypeScript development. Gustavo Dev Doido is a developer and educator active in that ecosystem, and the Bootcamp do Dev Doido is the program associated with that teaching work. Skala Blog publishes written articles generated from technical video content.

Turn That Recorded Deep Dive Into a Written Guide

If your team has already explained its observability approach on a call, a conference stream, or a recorded walkthrough, that explanation is an article waiting for a transcript. Skalablog takes a YouTube URL, transcribes the video, and produces a structured draft you can edit and publish, which means you can turn one recording into a resource readers can search.

Visit Skala Blog to paste a video link and generate the article.

Source video