Agentic document search works best as a hybrid: retrieval narrows the corpus, then grep, metadata filtering and file reads ground the answer. George He, head of engineering at LlamaIndex, argued this at the AI Engineer World's Fair 2026, and it explains why Claude Code skipped embeddings while enterprise agents cannot.
Grep or embeddings for agentic document search?
Neither method wins alone: agentic document search works best when hybrid retrieval narrows the corpus first and grep, listing and reads ground the answer afterwards. The deciding factors are data scale, format and how much curation effort a team can maintain, according to George He, head of engineering at LlamaIndex, speaking at the AI Engineer World's Fair 2026.
He framed the choice as two evolving approaches to orchestration. Local file search and execution means the agent traverses files directly with tools like grep and cat. Pre-indexing means documents are parsed, chunked and embedded before any query runs. Models have gotten better at instruction following and file traversal, and they now work with skills, MCP servers and instruction files to retrieve information. But managing millions of messy documents entirely in context still burns too many tokens and re-processes the same binary files on every run.
The right answer scales with the corpus. The talk's rough guidance, from a 20-minute session:
| Approach | Best fit | Strength | Weakness |
|---|---|---|---|
| Local traversal (grep, glob, cat) | 100 to 1,000 text files, code repos | No index to keep in sync | Token cost explodes at scale |
| Pre-indexed retrieval (RAG) | Thousands to millions of files | Cheap downscoping per query | Freshness and sync are hard |
| Hybrid harness (retrieval plus traversal) | Enterprise corpora | Compass plus grounded answers | More pipeline to build and operate |
Why Claude Code skipped embeddings
Claude Code, Anthropic terminal-based agentic coding tool released in 2025, chose no pre-indexing, no RAG and no vector search. It relies on the local file system with glob, grep and read operations. Per the talk, the team experimented with a vector database and abandoned it for practical scaling and maintenance reasons, not raw performance: keeping a vectorized index in sync with constantly changing code is hard.
That decision traces back to a public observation from Boris, the Claude Code creator, which George He cited on stage. The decision holds only because of specific properties of code repositories: they are usually small, semi-structured, full of breadcrumbs such as imports and tests, and already sitting on local disk in text format. The agent can build its own map by traversal, so a precomputed index adds maintenance cost without much benefit.
Company data breaks every one of those assumptions. Corporate corpora mix PDFs, PowerPoint decks, scanned schematics and manufacturing diagrams. Folder structures are not curated the way source code is, there is no clean hierarchy for an agent to walk, permissions matter, and the scale makes local traversal infeasible. The takeaway from the talk is not that one method replaced the other; it is that data characteristics dictate the design.
When does local traversal stop working?
Local traversal stops working once a corpus grows past what an agent can hold and iterate through in context. The talk puts the cutoff at roughly 100 to 1,000 files; beyond thousands or millions, the agent needs a compass that points at the right region of the data before it starts reading.
Context windows keep growing, which changes the economics. Current models take 1 million tokens of context, up from 200k before that and 100k before that. But shoving tokens through the ingestion pipeline, or sending binary data to the model for rendering and extraction, stays expensive. Sub-agents help with memory bloat: a main session issues commands to a smaller model running a sub-session. The catch is that each sub-agent still needs full context, and those environments are token-heavy and latency-heavy. At some point, pre-indexing beats paying that cost on every query.
The five-tool harness agents need
The talk recommends a harness: five tools the agent chooses from per query instead of a manually picked retrieval strategy. Semantic retrieval gives direction, listing and metadata filtering expose the sub-corpus, grep finds exact text, and parsed text with page screenshots reveals what a complex document actually says. Each primitive answers a distinct question:
- Hybrid retrieval answers "where should I even look?" and returns a compass direction over the corpus.
- Listing files exposes the sub-corpus so the agent can see what exists inside the retrieved scope, including files the initial search missed.
- Metadata filtering narrows by year, company or document type without reading file contents.
- Grep finds exact keywords or regex patterns inside candidate files, with read offsets and lengths to pull specific slices.
- Parsed text and page screenshots reveal what a complex document actually says, including tables and diagrams.
The sequence matters. Semantic search downscopes the file set cheaply, then built-in traversal digs into specifics so answers are grounded in real file content instead of hallucinated. In the demo, the agent grepped for a regex, read 1,000 characters starting at an offset of 500, and pulled the exact cash-flow text it needed, all against remote storage without downloading the files.
Hybrid retrieval and reranking as the compass
For the retrieval layer itself, the talk compares graph RAG, traditional RAG and keyword-based BM25 search, and recommends a hybrid for enterprise use cases. Keyword search catches exact terms agents look for; semantic similarity catches paraphrases. Fusing and weighting the two avoids being blinded by the weaknesses of either single method.
The speaker's stated experience is that raw keyword or raw vector search reaches a baseline on benchmarking datasets, while a fusion of the two plus a reranking step over the top 100 results produces noticeably better retrieval. That last claim is vendor-reported experience from the talk, not an independent benchmark, so teams should validate it on their own data. The structural point stands regardless: retrieval is only the compass, and tools two through four do the grounding work.
Parsing complex documents before the agent ever reads them
Charts, scanned pages and schematics break traditional chunking because they are not lines of text; they are spatial representations. Sending raw binaries to the model on every read is expensive, so the talk recommends pre-parsing documents into structured formats during ingestion. LlamaIndex's commercial product for this is LlamaParse, and the company has also open-sourced a structural extraction tool called LiteParse, named in the talk, that converts tables, charts and segmented content into markdown an agent can reason over spatially.
Two parsing details carry most of the quality. First, faithfulness: does the parse preserve the table structure and layout? Dump the same data into a traditional OCR pipeline such as Textract or an OCR engine like Tesseract, and the question is whether the table representation survives. If the LLM downstream cannot recover spatial relationships, accuracy falls apart later even when the text looks fine. Second, screenshots: while rendering each page during parsing, persist the rasterized image. A pre-built screenshot index lets the agent visually verify a chunk against the page it came from, which matters for documents that were originally meant for human readers such as insurance, finance and healthcare paperwork.
Ingestion follows a three-step pipeline: pull all documents into a synchronized store, parse them into a standard representation such as markdown or text, then chunk and export into a pre-indexed state the tools can query.
Production concerns: permissions, freshness and storage
Three questions surface in every real deployment, per the talk: multi-tenancy, permissions and freshness. Multi-tenancy asks whether one customer's query can touch another's documents. Permissions asks how access metadata gets pulled from sources like Google Drive or SharePoint without re-parsing everything or grinding the system to a halt. Freshness asks how the index stays current when the underlying data changes, and the speaker notes freshness difficulty is why Anthropic many other companies gave up on pre-indexing code.
For storage, LlamaIndex recommends a layered design: persist data on disk, then put a shared memory cache in front for fast retrieval. Many vector stores keep everything in memory, which does not scale to the tens of millions of documents LlamaIndex says it handles at the multi-tenant level. The demo used turbopuffer, a vector storage service, and pushed permission metadata into the store itself so filtering happens at the storage layer as data is ingested.
These are vendor-described architecture choices from a 20-minute conference talk. They indicate what LlamaIndex builds for its customers, not an independent evaluation of turbopuffer versus alternatives.
The demo: cash-flow tables from 135 Alphabet filings
The live demo indexed 135 files derived from Alphabet's financial reports, mixing HTM and raw PDF files with source metadata attached, including year and company tags the agent can filter on. The agent ran a hybrid search with an even keyword-to-semantic balance for cash flow statements, received matched chunks plus the corresponding page screenshots, then grepped and read specific offsets inside candidate files to confirm figures.
The final prompt asked for cash flow income as a table from 2021 to 2025. The response assembled the table from multiple files, showing its reasoning across search, screenshot review and file reads. The speaker's summary: search finds information, but the harness gives results you can trust, because every number traces back to a file the agent actually opened.
FAQ
When can an agent rely on grep alone?
When the corpus is small (roughly 100 to 1,000 files per the talk), text-based, semi-structured and locally available, as with a code repository. Beyond that scale, traversal burns too many tokens and pre-indexed retrieval becomes worthwhile.
Why did Claude Code avoid vector search?
Code is small, text-based, full of structural breadcrumbs and already on disk. Per the talk, a vector index was tried and dropped mainly because keeping it in sync with changing code is a maintenance burden, not because of raw retrieval performance.
What does hybrid retrieval add over keyword or vector search alone?
It fuses exact-match keyword signals with semantic similarity, then reranks the top 100 results. The speaker reports better results than either single method on benchmarking datasets, though this is vendor experience rather than an independent benchmark.
Why do parsed page screenshots matter for RAG?
Complex documents contain tables and diagrams that text extraction mangles. Screenshots let the agent visually verify that a retrieved chunk matches the actual rendered page before citing it, which matters most for insurance, finance and healthcare paperwork built for human readers.
How should permissions and freshness be handled at scale?
Push permission metadata into the storage layer at ingestion time so filtering happens where the data lives, and treat synchronization as a pipeline: ingest, parse, then chunk and export to a pre-indexed state. Freshness remains the hardest part and is a common reason teams abandon pre-indexing.
Learn the harness by building it
The five-tool harness only makes sense once you build an agent against real files: retrieval that points somewhere, traversal that confirms, and parsing that keeps complex documents readable. If you are learning agentic development through courses such as Gustavo Dev Doido's Bootcamp do Dev Doido or Crazystack Typescript at crazystack.com.br, the same primitives apply to any project where an agent has to find and ground an answer.
A good conference talk packs months of engineering judgment into twenty minutes, but most of it stays locked in a video. If you have recorded explanations, interviews or internal walkthroughs worth finding later, Skala Blog turns a YouTube URL into a structured written article: paste the link, transcribe the talk, and get an editable draft that preserves the reasoning and the numbers.
The raw material already exists. It just needs the right pipeline to become something an agent, or a reader, can search.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits