Retrieval Is Filtering, Not Search: A Mental Model for Enterprise RAG

brick of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks: parsing, question parsing, retrieval, and generation. Retrieval is the third brick, and this is the first of its three parts, the mental model: retrieval is filtering, not search; filter line_df and toc_df, pick anchors small, expand context large.

Watch how a human searches a document.

Someone at work wants to know how many vacation days they get this year. They open the HR policy PDF. They press Ctrl+F. They type “vacation”. Fifteen hits scroll past, some in headings, some in the TOC. They jump to the right paragraph, read the rule, and have their answer in 60 seconds.

That’s not a novice doing it wrong. That’s a professional doing it the most efficient way they know: keywords they know are in the document, the TOC the author already wrote, reading a whole section when they suspect it’s the right one. Where in this process is “embedding similarity”? Nowhere.

Sometimes Ctrl+F finds nothing: The doc calls it “PTO” not “vacation”. Or the text sits inside a scanned page that Ctrl+F can’t see. The expert tries a synonym, then a third. Still zero hits.

Then the expert opens the table of contents. They scan the section titles, click the most likely one (“Leave and Time Off”), and read the body. That fallback (keyword first, TOC navigation when the keyword fails) is what professional document work has run on for thirty years.

This article (the first of three) builds the mental model behind that workflow: retrieval is a filtering problem on two structured tables (line_df and toc_df), not a search problem. It also introduces the anchor / context distinction (where the match lands versus what gets passed to generation) which the other two articles build on. The pipeline mechanics (“Anchor detection for RAG: parallel detectors, then one LLM call at the end”) and the arbiter that ranks the results (“Letting an LLM pick the right RAG page: the arbiter pattern at the end of retrieval”) come next.

The “amplify the expert” stance: codify the expert’s workflow, then do it better than they can manually. Three concrete lifts:

• The expert types one keyword at a time. The system can detect co-occurrence of multiple keywords on the same page or section in a single pass.
• The expert sees nothing when words are locked in scanned images. The parsing brick runs OCR at ingestion, so image-bound text becomes searchable like any other line.
• The expert scans the TOC manually. The system joins TOC and content programmatically: pick the right section from the map, then scope the keyword search inside that section’s body.

Once the parsing brick has produced clean DataFrames, retrieval becomes a filtering problem on structured tables: filter line_df (the text) and toc_df (the map). This article builds up that mental model. The next two articles build the mechanics on top.

Throughout this article we work on a single document, Attention Is All You Need (Vaswani et al. 2017, 15 pages; arXiv non-exclusive distribution license, declared on the arXiv abstract page). It carries a clean native TOC in the PDF outline (22 entries, 3 levels deep), and the content is familiar territory for any engineer touching RAG: encoder, decoder, attention, queries, keys, values. That keeps the focus on the retrieval methods rather than on parsing a domain-specific corpus. This article also assumes the document carries its own TOC; recovering one from raw text is left to follow-up work.

1. Retrieval as filtering on structured tables

The standard framing of retrieval is find the passages most similar to the query. That framing is misleading because it imports the wrong mental model.

Once parsing has produced clean DataFrames, retrieval is no longer a search problem in the classical sense. It is a filtering problem on structured tables. Every method we discuss is a different way of filtering rows of line_df (the document’s text) and toc_df (the document’s map). The mental model is closer to a SQL query than to a Google search.

This shift unlocks methods that don’t appear when you treat retrieval as free-text search:

• You can filter on columns: only lines in section X, only lines that match a regex, only chunks whose embedding is close to the query, only titles whose words intersect the question’s keywords.
• You can join tables: detect a keyword in line_df, then look up the section that line belongs to in toc_df, then weight the score by whether the section title is also relevant.
• You can run lightweight LLM filtering on the small tables (a toc_df with 50 entries) where you can’t run it on the large ones (a line_df with 12,000 lines, too many tokens for any single call).

The parsing brick produces structured DataFrames: line_df carries the textual content with line numbers, page coordinates, and section IDs; toc_df carries the table of contents as a navigable hierarchy; page_df and image_df round out the data model. The document is no longer free text by the time retrieval runs.

The question side arrives equally pre-processed. The question parsing brick turns the user’s string into a RetrievalQuery brief (a derived view of ParsedQuestion) that carries the keywords, scope filters, structural hints, and the answer-context width retrieval should respect. Retrieval reads one typed object on each side: the document-as-tables on the left, the RetrievalQuery on the right.

None of this is exotic. It is what teams do when they sit down with a real document and a real question. But it is invisible in the standard tutorial framing because that framing assumes “document = unstructured string, retrieval = vector search.”

1.1 Documents become tables

Two DataFrames from the parsing brick carry most of the retrieval load. Their sizes determine which filtering methods are even possible on each.

line_df, the dense table: One row per line of the document, tens of thousands of rows for a long contract. Columns include the text, the page number, the line number on the page, the bounding box, and the section_id that links each line to its section in toc_df. Dense, large, fine-grained: every line is a candidate, so filtering needs to be cheap per row. This is where the answer lives: the actual text the LLM will read and cite comes from here.

toc_df, the sparse table: One row per section in the table of contents, typically 20 to 100 rows for most enterprise documents, sometimes as few as 10. Columns include the section title, the level (1, 2, 3…), the page range, the parent section, and a stable section_id. Sparse, small, coarse-grained: each row covers a large block of the document, so filtering can be expensive per row. This is the map of where the answer might be: it tells you which section to look in, not what the section says.

The size difference between these two tables reshapes the entire grid. A method that is feasible on toc_df (passing the whole table to an LLM, embedding every entry, running multi-hop reasoning) may be entirely infeasible on line_df. Conversely, a method that is natural on line_df (regex over thousands of lines, fast keyword scoring) is wasteful on toc_df because there is not enough data to discriminate.

A good retrieval pipeline uses both. It uses toc_df to narrow down to the right section, then line_df to find the precise lines within that section. The two tables collaborate via the section_id join.

1.2 Why no single method is enough

Four real questions on an insurance contract show why no single filter and no single granularity is enough. Each question wants a different scope at two levels: the anchor (where the matching signal lands in the document) and the context (what gets passed to the LLM around that anchor). Section 2 develops both terms; for now just notice how they vary across the four questions.

1. “What is the policy number?” A needle in a haystack. The anchor is one specific token, probably in the header. The context is page 1, maybe 5 lines.
2. “What is the annual premium?” A point lookup. The anchor is a clause that mentions “premium” or “prime” or “cotisation”, with an amount. The context is the section that contains that clause, maybe 50 to 200 lines.
3. “What are all the obligations of the seller?” A listing question. The anchors are multiple passages, scattered through the document. The context is the entire “Obligations” section if it exists, plus any other section that contains an obligation. Maybe 500 to 2000 lines.
4. “Summarize the warranty section.” Scoped synthesis. The anchor is the warranty section title in toc_df. The context is the warranty section body, exhaustively. Maybe 200 to 1500 lines.

A pipeline that uses cosine similarity with top-k=5 (return the 5 closest chunks by score) for all four cases will be wrong on at least three. The right answer depends on which structure you filter, what column you filter on, what anchor you detect on, and what context you pass downstream. The rest of the article develops that grid.

The first question on the list, “What is the policy number?”, is exactly what the Needle-in-a-Haystack benchmark tests (Kamradt, 2023, github.com/gkamradt/LLMTest_NeedleInAHaystack). Drop a verbatim sentence into a long context, ask about it, watch the model find it. Frontier models score near-perfectly. The benchmark is real, the result is real.

The trap is generalizing from it. “Skip retrieval, dump the corpus in context” works on category 1 and fails on categories 2, 3, 4. Listing every obligation in a contract isn’t a single needle. Comparing premiums across three policies isn’t a single needle. Summarizing a warranty section without missing anything isn’t a single needle.

The benchmark validates one question class, not the other three. It is a research result for an isolated retrieval task, not a license to skip retrieval in production.

To make the contrast concrete, here is the naive RAG baseline that most tutorials show, applied to the running paper. We pick a representative question, “How is attention computed?”. The answer is the formula box in section 3.2. We then run cosine top-k against the page embeddings.

The rest of this article is methods that beat this baseline by exploiting the structure parsing already extracted.

2. Anchor and context: the two granularities

Filtering rows is half the picture. The other half is just as foundational: the anchor (the row where you detect the signal) and the context (the chunk you pass downstream to generation) are not the same unit. The anchor terminology is local to this article; adjacent literature uses hit in IR, trigger in information extraction, or evidence span in QA, all for the same idea. We keep anchor because it pairs naturally with context.

Retrieval runs in two phases. Phase 1 finds where the answer lives: keyword detection and embeddings run on line_df and toc_df, an LLM ranks the candidates once at the end, and the output is a small set of anchors (section, page, or lines). Phase 2 sizes the context around each anchor: paragraph, section, or a window of N lines, driven by the question’s intent and scope width (already parsed on the question side). Article 7B (anchor detection) develops phase 1’s detectors and Article 7C (the LLM arbiter) develops the arbiter that ranks them; section 2.4 below develops phase 2.

A compliance officer searching “liability” with Ctrl+F lands on a single matching line. They never read just that line. They read the surrounding paragraph, often the whole section. The anchor is one line; the context is hundreds.

Concretely: you may anchor on a single line of line_df that mentions “premium”, but you pass the whole surrounding section to generation so the LLM sees the value in context. You may anchor on a toc_df title (“Section 5: Specific Exclusions”), but the context is the entire section’s body lines from line_df. The two granularities are independent design decisions:

• Anchor scope (small, precise): line, title, sentence. This is what you score and rank. Smaller anchors give cleaner signals.
• Context scope (large, sufficient): paragraph, section, window of N lines. This is what generation reads. Bigger contexts protect against fragmented evidence.

Collapsing the two scopes, detecting at chunk level and passing the same chunk downstream, is the most common mistake in RAG pipelines. It loses precision (chunks are too coarse for fine-grained anchoring) and richness (chunks are too narrow for grounding the answer). The rest of the article keeps the two scopes separate at every step.

2.1 Anchor scope: where to look for the signal

When you search for “exclusion”, what unit of the document are you scoring?

• A line? Match if the word appears in this line.
• A paragraph? Match if the word appears anywhere in this paragraph.
• A section title? Match if the word appears in the title of a section.
• A section body? Match if the word appears anywhere in the section’s text.

The same word at different anchor scopes gives very different results. A title-level anchor is much more selective and precise than a body-level anchor on a long document.

2.2 Context scope: what to extract around the match

Once an anchor is found, what do you pass downstream to generation?

• Just the matched line? Rare, usually too narrow.
• The surrounding paragraph?
• The whole section the match belongs to?
• A window of N lines before and after?

A single match in line_df rarely makes a usable answer on its own. The expansion step is what turns a hit into context.

The two scopes are independent. You can anchor at line level and extract at section level. You can anchor on section titles and extract from the section body. Each combination has its use:

A concrete example. A user asks “What are the flood damage exclusions?”. The keyword “exclusion” is anchored in toc_df at the title level, in the entry “Section 5: Specific Exclusions”. This single title is not the answer; it is the map. The context extraction then takes the entire section 5 from line_df, because the actual exclusions are listed in the body.

The anchor was found on a 50-row toc_df. The context extraction joins back to line_df to get the actual content. The two structures collaborate, and the answer is the joined product.

Keeping the two scopes separate makes retrieval precise instead of approximate. The canonical implementation, detect_then_extract, is developed in Article 7B (anchor detection) alongside the other combinations.

2.3 Question types and scope choices

Five recurring question types, each with its preferred anchor and context scope.

Field extraction: “What is the effective date?” Anchor on a line where a date pattern sits next to “effective date” or “inception”. Context is the page, enough to confirm the right field.

Section retrieval: “What are the exclusions?” Anchor on the section title “Exclusions”. Context is the section body, from the heading to the next one.

Conditional lookup: “Does this contract cover flooding?” Anchor on the line containing “flood”. Context is the surrounding section, needed to confirm whether the topic is covered or excluded.

Open-ended, scoped: “Summarize the obligations of the seller.” Anchor on a section heading (“obligations”, “seller”, “duties”). Context is the section body, potentially extending to the wider document.

Open-ended, corpus-wide: “Which contracts have a liability cap below 1M€?” Anchor on lines containing “liability cap” with an amount. Context is the paragraph (to confirm the amount is the cap), plus a corpus-level metadata filter on document type.

The pattern is consistent: anchor small, expand for context. The line-level line_df from the parsing brick gives the anchor granularity. The page or section gives the context. The TOC tells you where sections start and end.

2.4 From match to context: three expansion strategies

A retrieval method returns anchors: line IDs, section IDs, chunk IDs that scored highest. You almost never pass these directly to generation. You expand them first.

Why? Because a single anchor, in isolation, is hard to reason about. Take this anchor:

“€125,000 annually.”

Without context, the LLM does not know what €125,000 refers to. With one paragraph of expansion:

“The annual premium for the policy is set as follows. The base premium amount is €125,000 annually. This may be adjusted in accordance with section 3.4.”

Three expansion strategies cover most cases.

Paragraph expansion: Take the paragraph that contains the matched line. Works for most QA tasks.

Section expansion: For listing or synthesis questions, expand to the full section. Use toc_df to find boundaries.

Window expansion: For documents without clear paragraph or section boundaries (transcripts, long-form prose), expand to N lines before and after.

The choice depends on the question (intent, expected answer shape) and the document (does it have paragraph structure? sections?). The dispatcher picks; the orchestration applies the chosen strategy uniformly. With one of these, the LLM can answer correctly and cite precisely.

def expand_to_section(line_num, page_num, line_df, toc_df):
    anchor_section_id = line_df.loc[
        (line_df["line_num"] == line_num) & (line_df["page_num"] == page_num),
        "section_id",
    ].iloc[0]
    sec = toc_df[toc_df["section_id"] == anchor_section_id].iloc[0]
    in_section = (line_df["page_num"] >= sec["start_page"]) & (line_df["page_num"] <= sec["end_page"])
    return "\n".join(line_df[in_section]["text"])

def expand_window(line_num, page_num, line_df, n=5):
    page_lines = line_df[line_df["page_num"] == page_num].reset_index(drop=True)
    i = page_lines.index[page_lines["line_num"] == line_num][0]
    return "\n".join(page_lines.iloc[max(0, i - n) : i + n + 1]["text"])

Run both on the first body line of page 4 (where the Attention(Q,K,V) formula sits) and the difference between the two scopes shows up:

2.5 When there’s no TOC: where does the section end?

Section expansion is straightforward when toc_df gives the page range. The start of the answer is the anchor, the end is the next TOC entry’s start page, no thinking required. When the document has no native TOC and no synthesized one, that easy bound disappears. The end of the section has to come from the content itself, and that is one of the hard problems in document AI.

A short tour of what research has tried.

• Heading detection at parse time: Walk forward from the anchor, stop at the next line that looks like a heading (larger font, bold, regex on “Section X” or “Article X”). Simple, fast, brittle on documents that lack visual heading conventions.
• TextTiling (Hearst, 1997): A sliding window of lexical similarity. The boundary sits where the topic vocabulary changes sharply. Works on flat prose like transcripts; tuning the window size is the whole game.
• Embedding-based segmentation: Same idea but with sentence vectors. The boundary is where the cosine drops past a learned threshold.
• LLM “cut here” calls: Hand the LLM a wide forward window, ask where the topic ends, get back a line number. The most accurate, and the most expensive per question.

Each of these is a paper or three, with tuning knobs, failure modes, and benchmark folklore. Stacked together, they make a small research project on their own.

Our position in this series: the LLM you already have in generation does this job. The generation brick reads the context retrieval handed it and produces the answer. The same call can also report whether the context went off-topic, whether the answer continues past the window, whether more context is needed. A follow-up integrated pipeline wires this back into a feedback loop: generation flags “the topic continues past the last line I was shown”, retrieval extends the window, generation re-runs. The boundary is found by the same model that uses the boundary. No second machinery, no segmenter to train, no threshold to calibrate.

The wider point: R&D engineering and enterprise engineering are different jobs. Section-end detection has a real research bibliography. Most enterprise teams do not need to solve it; they need a system that works on their actual documents with their actual tools. The discipline is to ask, before reaching for TextTiling or a custom segmenter, whether one extra LLM call would have done the same job. Today, the answer is usually yes. Big-model inference is not what it was two years ago: pricing is down by an order of magnitude, latency is acceptable, and an enterprise pipeline can afford one extra call per question on a problem like this without breaking the budget. The trade is clear: spend on inference, not on a custom segmentation stack you then own forever.

That is not a blanket “use the LLM for everything” claim. Cost still matters, latency still matters, and a small deterministic rule beats an LLM call when the rule covers the case cleanly. But the default has shifted. R&D explores the technique space and pushes the techniques forward; enterprise engineering picks the simplest tool that meets the requirement and ships. In retrieval, the temptation to confuse the two is high because every paper looks like an opportunity. The right call is usually the one that does not require new R&D.

3. Conclusion

Retrieval is not search; it is filtering on two structured tables (line_df and toc_df) that parsing already produced. Every retrieval method is a different way of picking rows out of these two tables.

Anchor and context are different granularities. Anchor small (line, title): that is what you score. Context large (paragraph, section): that is what you pass to generation. The two scopes are independent design decisions; collapsing them is the single most common pipeline mistake.

Two phases. Phase 1 finds where the answer lives. Phase 2 sizes the context around each anchor, driven by the question’s intent and scope width (already parsed on the question side).

With the mental model in place, the natural next question is how the pipeline produces the anchors: which detectors run on line_df and toc_df, in what order, and when the LLM enters. Article 7B (anchor detection) walks that three-stage pipeline (keyword + embeddings in parallel, aggregate to a structural unit, one LLM arbiter at the end), with runnable code on the Transformer paper as the running example. Article 7C (the LLM arbiter) closes the loop on the arbiter call, the decision tree that picks methods per question, the “not found” path, and the unified JSON contract retrieval hands to generation.

This article is part of the Enterprise Document Intelligence series. The minimal RAG pipeline shows retrieval in use end to end on a real PDF.

4. Sources and further reading

Retrieval reframed as filtering on two structured tables, with anchor and context as separate granularities. The references below cover the mental model.

Same direction as the article:

• Sarthi et al., RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval, ICLR 2024 (arXiv:2401.18059). The closest published equivalent to the TOC-as-retriever framing; RAPTOR clusters its own tree, this article uses the TOC the document already declares.
• Anthropic, Contextual Retrieval (Sept 2024). Hybrid-search consensus aligned with the anchor-and-context separation this article introduces.

Earlier in the series:

• Document Intelligence: series intro. What the series builds, brick by brick, and in what order.
• Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out.
• Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.
• Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost. What a cross-encoder adds over bi-encoder embeddings, measured, and when it is worth the latency.
• RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.
• From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case.
• 10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each.
• Beyond extract_text: the two layers of a PDF that drive RAG quality. The first half of the parsing brick: the document’s nature, signals, and summary.
• Stop returning flat text from a PDF: the relational shape RAG needs. The second half of the parsing brick: the relational tables every downstream brick reads.