Governed enterprise AI agents run on more than a model and a prompt. At Intuit Credit Karma, Harish Gaggar describes a paved path that registers each agent in an internal portal, provisions workload identity, memory, tracing and evaluation tables, routes every tool call through a centralized MCP gateway, and prunes context before tokens reach the model.
What Are Governed Enterprise AI Agents?
Governed enterprise AI agents are agent workflows wrapped in platform controls: identity, authenticated tool access, durable memory, tracking, evaluation, sensitive-data filtering and context pruning. Harish Gaggar, a senior staff software engineer at Intuit Credit Karma, presented this model at the AI Engineer conference in 2026 as a way to move agents from experiments into production.
The argument is that agents fail in production less often because the reasoning is weak and more often because the supporting controls are missing. Gaggar lists the recurring gaps: no gateway to authenticate the agent and route it to the right tool server, tool servers lacking the right reviewed tools, no tracking or evaluation of response quality, no memory across long-running interactions, and no safeguards against secrets reaching models, memory stores or traces.
The platform described here has four layers. An agent framework built on LangGraph handles interpretation, planning and control flow. A centralized MCP gateway handles authentication and routing. Custom Model Context Protocol servers expose vetted tools. Monitoring and four add-on frameworks cover memory, tracking and evaluation, PII blocking and context optimization.
Gaggar is explicit that the demo does not prove a validated quality score for the whole system. The walkthrough inspects records the platform writes; it does not establish that the agent answers correctly. That distinction matters when reading the token and cost numbers later in this article.
The Paved Path From Local Code to Kubernetes
The inner development loop starts with registration, not coding. A developer registers the agent's use case and vertical, receives an agent ID, then submits details through an internal developer portal: the agent ID, the framework (LangGraph or LangChain), the language (Python or TypeScript), and the skills or prompts the agent will use.
That submission does more than create a repository. The portal scaffolds the agent framework alongside the memory framework, the tracking and evaluation framework, the context optimizer and the PII blocking layer. Gaggar's stated reason is consistency: individual developers tend to adopt one or two of these pieces, or only the AI assistant tooling they already know, and the company ends up with uneven coverage.
The portal also provisions the production resources the agent will need: workload identity, the memory database, tracking and evaluation tables, and secret management. Because the Kubernetes-to-service-account binding already exists, the deployed agent knows which identity to present and can begin talking to the AI gateway, MCP servers and tracing endpoints without a manual wiring step.
The deployment sequence follows a familiar path:
How Does the MCP Gateway Authenticate and Route Tool Calls?
The MCP gateway is a centralized authentication and routing hub that sits between agents and MCP servers. Gaggar's rule is that no endpoint should be reachable directly: every tool call, whether it originates from a coding assistant in an IDE or from an agent, passes through the gateway first and reaches the MCP server only after authentication.
The request flow works like this. When a client submits its first query, it sends a one-time authentication request to the gateway. The gateway stores session state in Redis and returns a JWT access token. The client then issues a tool call to a namespaced path, for example with an Artifactory or GitHub MCP server, carrying that bearer token.
On receipt, the gateway reads the namespace to determine which MCP server to target, unwraps the JWT, attaches the user token to the outbound request and forwards it. The MCP server makes the underlying API call, exchanges the ID token, and returns only what the user's scope permits. The payload travels back through the gateway to the client.
For teams working in air-gapped environments, Gaggar recommends building both the gateway and the MCP servers internally rather than relying on publicly available servers. That recommendation is about network control and reviewability, not a claim that internal components are automatically secure.
Building Reviewed MCP Servers and Watching the Gateway
Custom MCP servers are where tool authority is decided. Gaggar suggests using the FastMCP package or wrapper to define a server and the individual tools inside it, and he is explicit that every tool must be reviewed by the internal security team before it reaches production. The review is the mechanism that enforces filters, limits and field-level blocks.
That field-level control is the point. A reviewed tool can be constrained so that fields not required by the model never leave the boundary, which limits what an agent could retrieve even if it asked. Tool review and allowlisting are technical controls; they are not by themselves a compliance certification for healthcare or financial deployments.
Monitoring sits on top of the gateway rather than inside each agent. Operational logs go to a log store such as ELK, Loki or CloudWatch and answer questions like how many 401s occurred in the last ten minutes, or what OAuth did when a token refresh failed.
Traces and metrics take a separate path. OpenTelemetry collects and processes them, traces land in Grafana Tempo to show which tool was slow on a request, whether a call came from an agent or an IDE, and what error occurred on a failed call, while metrics land in Prometheus for figures such as P95 latency per tool and spikes in authentication rejections. Grafana then renders the aggregate for visualization.
Inside the LangGraph Workflow and Human Approval Step
The agent workflow is a graph with loops, branching and explicit state, built on LangGraph, which the LangChain community maintains. Gaggar describes one path through it: interpret the question and the response shape, generate a plan, then classify the request as read, modify, delete or otherwise sensitive, and adjust the plan accordingly.
Planning is itself a subgraph where the substantive work happens. In the evidence-gathering stage the agent calls the MCP gateway, authenticates, invokes the right tool on the right MCP server and assembles evidence. Findings are validated, then policy checks decide what happens next.
Read-only operations execute and write to the audit log. Write or sensitive operations require a human-in-the-loop approval, and both the approval and the rejection path write to the audit log. The human checkpoint is positioned on the action, not on the reasoning that precedes it.
Memory is what makes multi-turn work possible. A query opens a thread; the first question creates a run ID; execution lands on a specific node and saves a checkpoint there. Follow-up questions on the same thread create new run IDs and new checkpoints, so state can be restored rather than recomputed.
Memory, Tracking and Evaluation Tables Compared
Memory and tracking are separate systems with separate jobs. Memory stores workflow state, threads, runs and checkpoints so an agent can resume. Tracking stores what the agent did: the query, the response, the model used, token counts, cost and evaluation fields. Gaggar identifies staging options for both, and the choice is deployment-specific rather than fixed by the framework.
| Layer | What it stores | Local option | Production example |
|---|---|---|---|
| Agent memory | Threads, run IDs, checkpoints | SQLite | Cloud Spanner on Google Cloud, or managed databases on Azure and AWS |
| Tracking and evaluation | Query, response, model, tokens, cost, quality checks | SQLite or similar | BigQuery-class warehouse tables, one set per agent project |
| Gateway logs | Request-level operational events, auth failures | Any log store | ELK, Loki or CloudWatch |
| Traces and metrics | Per-tool latency, errors, call origin | Local collector | Tempo for traces, Prometheus for metrics, Grafana for dashboards |
On tracking, Gaggar describes six tables in the demo, with a job tracking table holding the query and response and further tables covering quality, tool invocations and node-level execution. A single run produces eight quality checks covering dimensions such as success, latency, cost, tool accuracy and safety, plus an evaluation score between zero and one for that run, a nested quality-metrics record, a per-tool invocation log, and an agent trajectory that records each model call and graph state in order.
The scale argument is straightforward. If a company has a hundred agents, it has a hundred tracking projects; at six months a reviewer can aggregate across them and build dashboards that speak to return on investment. Gaggar mentions the thousand-agent case as the scenario that makes per-agent tables worth standardizing on day one.
PII Blocking and the Context Optimizer in Numbers
Sensitive-data filtering sits in the ingestion path. Gaggar describes a framework that classifies data types and assigns a severity level; when severity is above a threshold he gives as about 90 percent, the input is blocked and redacted at the moment it is typed so that the secret never reaches the model, the memory database, the tracking and evaluation tables, or the tracing system.
That is a control to evaluate, not a guarantee. Detection thresholds produce false negatives, and Gaggar presents the layer as one that aims to intercept sensitive input rather than one that proves no secret can escape. Teams in regulated settings still need their own data-handling review before relying on it.
The context optimizer does the token work. Before a payload reaches the model, it selects content against a token limit, reranks for relevance, deduplicates, and compresses the remaining payload. Gaggar lists pruning, reranking, deduplication and compression in that order.
The demo figures are concrete but narrow. An Artifactory query the agent ran without the optimizer consumed roughly 24,000 tokens; the identical query rerun with the optimizer enabled consumed about 2,400, a reduction near 90 percent. A second run through a Slack bot fell from roughly 24,000 tokens to about 1,500, again near 90 percent. Gaggar says most cases land between 70 and 78 percent token savings.
Those numbers measure token consumption on the queries shown, not overall cost per agent or answer quality. No paired quality study is presented in the talk, so the savings should be treated as token reductions observed by the speaker on his own demo, not as a validated end-to-end efficiency result.
FAQ: Enterprise AI Agent Platform Questions
- What is an MCP gateway in an agent platform? It is a centralized authentication and routing service between agents and MCP servers. Clients authenticate once, receive a JWT, and send tool calls through the gateway, which unwraps the token, resolves the target namespace and forwards the request to the appropriate MCP server.
- Why does Intuit Credit Karma use LangGraph for its agents? Gaggar says LangGraph, maintained by the LangChain community, provides a graph-based architecture that supports loops, branching and state management for multi-agent workflows. Interpretation, planning, evidence gathering, policy checks and approval map naturally onto nodes in that graph.
- Where does agent memory live in production? The talk names Cloud Spanner on Google Cloud and equivalent managed databases on Azure and AWS for production, with SQLite used during local development. Threads, run IDs and checkpoints are the records that memory stores so a workflow can resume.
- Is PII blocking a guarantee that no secret reaches a model? No. The filtering layer assigns severity levels and blocks or redacts above a threshold Gaggar gives as about 90 percent, intercepting input at ingestion. Detection remains a control to evaluate and can miss sensitive values, so organisations still need their own data-governance review.
- How much does the context optimizer reduce token use? The talk reports roughly 24,000 tokens dropping to about 2,400 on one query and roughly 24,000 dropping to about 1,500 on another, both near 90 percent, with most cases between 70 and 78 percent. These are token counts from the demonstrated queries, not end-to-end cost savings.
- Do you need to build your own MCP servers? Gaggar says publicly available MCP servers can be used, but he recommends building internal servers for air-gapped environments. Tools defined on an MCP server must be reviewed by the internal security team before reaching production, and the review is where field-level filters and limits are enforced.
- What does the gateway monitoring stack collect? Operational logs go to a store such as ELK, Loki or CloudWatch for auth-failure questions, traces go to Tempo through OpenTelemetry, and metrics go to Prometheus for per-tool latency and rejection rates. Grafana visualizes the aggregate.
- How many tracking tables does one agent produce? The demo shows six, with a job tracking table plus quality, evaluation-score, quality-metrics, tool-invocation and agent-trajectory tables. Each run also produces eight quality checks and a score between zero and one for the run.
- Does the talk prove the agents produce high-quality answers? No. The demonstrations inspect the records the platform writes, such as token counts, tool calls and quality-check rows. Gaggar does not present a validated quality score for the system as a whole.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits