Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One

On a demo folder that works. On an enterprise shelf it stops working, and not because a parameter is set wrong. Similarity search returns passages, so it never picks a document, and the answer comes back assembled out of three files that were never meant to be read together.

The step that gets skipped is the one before that. A collection of documents is not one kind of thing. A shared drive of unrelated reports, five thousand copies of the same contract, and a stack of claim folders are three different problems, and an architecture that suits any one of them suits the other two badly.

This article is the map of that decision:

five ways a flat pile fails once the shelf grows, and why a re-ranker fixes none of them
three questions that tell you which of the three shapes you have
what each shape asks you to prepare, and what building the wrong thing costs
the four bricks of the single-document pipeline, seen one floor up
a baseline you can run today, and the waste it makes visible on five real PDFs

This article opens Part IV of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and the first thing to settle is which kind of folder is in front of you.

🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.

📓 Run the five-PDF baseline in the companion notebook: one question, five model calls, and the per-document table that shows one real answer and four blanks. Then point the same loop at a folder of your own and watch the same ratio appear. Repo → doc-intel/notebooks-vol1.

The worked example in section 4 is five public NIST publications: the Cybersecurity Framework v1.1 (NIST CSWP 04162018), FIPS 199, SP 800-207, AI 100-1 and CSWP 29. All five are US Government work, public domain in the US (see the NIST copyright statement).

Everything in the series so far could assume the system knew which document to look at, because there was only one. This section is what happens when that assumption goes.

A mid-size insurance broker keeps about a quarter of a million documents. Fifteen years of policies, certificates, contracts, claims and correspondence, some scanned and some native, most of them never opened twice.

A user asks: “what are the seller’s obligations in our distribution agreements with retailer X?”

A senior claims handler answers in under a minute, and she does it without reading anything she does not need. Obligations sit in distribution agreements, so invoices and certificates are out. For retailer X there is one master from 2019 and two amendments. Obligations are usually section 4 of the master, and an amendment wins where it says so. She pulls three folders, reads three sections, comes back.

Point the textbook pipeline at the same question and it does something else entirely. Every passage of every document is already embedded in one store. The question gets embedded, the store returns the thirty nearest passages by cosine similarity, and those thirty go into one prompt.

Sometimes the answer is right. Often the thirty hold two paragraphs from a contract with a different retailer, a certificate that belongs to no contract at all, and a 2017 invoice whose only qualification is the word obligation. Nothing in the input tells the model that those three should never be read together, so it writes a paragraph that reads well and describes no document that exists.

The gap between the two is not accuracy. They solve different problems. The handler decides which documents count and then reads. The pipeline never decides anything about documents at all.

At small scale the pile works often enough to demo. On a few hundred documents the right one usually lands in the top-k by luck, and the demo gets read as a validation of the approach rather than of the corpus size.

Five things break as the shelf grows, and they break independently of each other.

Read the last column first. A wider top-k pulls in noise at the same rate as signal. A different embedding model moves the failures around instead of removing them. Chunk size has nothing to say about a document boundary that was never recorded in the first place.

The usual next move is a cross-encoder: retrieve a wide top-100, re-score every candidate against the question one at a time, keep the best three.

A re-ranker is a real tool and it is worth having, but not here and not yet. It re-orders the list the embedding step returned. If the right document never entered that list, and vocabulary drift is precisely the condition under which it does not, there is nothing to re-order. Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured) works through where the layer does pay, and the answer is that it pays after the candidate set is right, not instead of getting it right.

There is a second cost that gets less attention. The re-ranker adds a number. Asked why the system answered this way, you can now offer a cosine score, a cross-encoder score and a top-3 cut. Three numbers, none of which a claims handler can argue with.

The fix sits upstream of all of it. Record the shape of the collection before any question arrives, then use that shape to cut the candidate set. A clause like doc_type = 'distribution agreement' AND client = 'retailer X' is exact, fast, and readable by the person who will be held responsible for the answer. It takes a quarter of a million documents down to three before a single embedding is computed, and with three candidates the single-document pipeline is already enough.

The useful part is that you can tell which shape you have without writing anything. Three questions do it, and they are questions for a person rather than for a script.

The test costs one conversation with one business user, and it settles the architecture. Three questions, asked of the collection rather than of any one document.

Question one: do two documents in the collection ever point at each other? An amendment names its master. A renewal names last year’s policy. A certificate belongs to a contract. A yes here does not pick a shape. It adds a requirement to whichever shape you land on, and section 3.2 says what that requirement is.

Question two: can a business user name a field that every document carries, meaning the same thing in each one? Client, effective date, premium, policy number. What tells you the answer is the speed, not the words. If four examples come back in two seconds, the collection is a database that nobody has typed yet, and typing it is the job. If the answer arrives after a pause and with qualifications, treat it as a no.

Question three: do the documents arrive in bundles, several of them about one case? A claim, a credit application, a medical record. Not a folder someone made to stay tidy: a bundle where a fixed set of pieces is expected, and where the interesting questions are about the bundle rather than about any piece in it.

Questions two and three pick the shape. When both come back yes, and in insurance they often both do, the row of the table is the case rather than the document. That is the third shape, not a fourth one.

One practical note. Run the test on a subtree, not on a drive. A shared drive that has grown for ten years usually holds two of these next to each other, and the right answer for the whole drive is the right answer for none of it.

Here they are side by side, with what each one asks you to build once and what the wrong call costs.

A pile of unrelated files. There is no field to filter on, so there is nothing to build a filter with. The preparation collapses to two artefacts, and the parser already returns one of them: a summary line per file saying what that file is about, and each file’s own table of contents. Retrieval routes down those two levels, which is the hierarchical retrieval of Part III applied one floor higher.

One document type, many copies. Five thousand policies of the same product, ten years of invoices, every claim form the company has ever received. The documents look alike because the same process produced them, which is exactly why the columns exist and why someone can name them. This is the shape that gets an index, and the index is a table rather than a vector store.

Bundles about one case. A claim folder holds a declaration, a police report, three invoices, a photograph and two letters. Heterogeneous inside, repetitive across: every claim folder holds roughly the same kinds of piece. The unit of work is the bundle, so retrieval assembles it first and reads across it second.

Three shapes means three ways to be wrong, and the bills are not the same size.

Treating a typed corpus as a pile. This is the default, because it requires no decision at all. Everything goes into the vector store and the five failure modes of section 1 arrive on schedule. The measurement in section 4 is the small version of it. The real cost is not the compute: it is that the system is wrong in a way nobody can trace, on a corpus where a business user could have written the correct filter by hand in ten seconds.

Treating a pile as a typed corpus. This one is expensive in a different way, because it looks like diligence. A team runs a classification pass and an extraction pass over every file to fill columns that no business user could name. Months later there is a table where most cells are empty and the filled ones mean different things per row: a date that is a publication date here, an effective date there, a review deadline somewhere else. Then a filter runs on that column and quietly drops the right document. A column that means three things is worse than no column, because a filter trusts it.

Treating case files as independent documents. The quiet one. Every piece answers on its own, so nothing looks broken. What never happens is the comparison. The contract says the cover started on 1 March, the certificate says 1 April, and no question is ever asked that puts the two side by side. The other half is the missing piece: when the medical report was never filed, a document-by-document system returns an empty result, and an empty result reads as “no information” rather than “a required piece is not here”.

Whichever shape came out of the test, the pipeline underneath is the one from Part III. What moves is the input each of its four bricks sees, so laying the two side by side is the last thing to do before building anything.

Each of the four bricks does the job it already did on one document, over a wider input. Part IV adds no fifth one.

The right-hand column is where the work goes. Parsing gains a second pass that writes one row per document into a table of the collection. Question parsing gains a handful of typed fields whose only job is to narrow that table. Retrieval gains a filtering step in front of the filtering it already does. Generation gains a second source, because sometimes the answer is a count of rows and no document is opened at all.

The names do not change and neither do the contracts between the bricks. What changes is how many documents reach the loop body.

A collection where documents supersede each other is not a fourth shape. It is a dimension that any of the three can carry, and question one is what detects it.

It adds two things to whatever you were going to build. A relations table, saying which document replaces or amends which. And a pair of dates on each row, saying from when and until when that document was the one in force.

Skip them and the system answers correctly from a document that stopped applying in 2023. That failure is worse than a missing answer because it looks right. The 2019 master says the seller gives 30 days of notice, the 2024 amendment says 90, and a pipeline with no time columns has no reason to prefer one over the other.

Which version answers is a retrieval question, and Article 17 (querying a corpus, with the time window as part of the scope) works it out. What matters at this point is only that the two columns exist, because retrieval cannot filter on a field nobody wrote.

Before any of the three architectures, it helps to see what the collection problem costs when you ignore it. The simplest possible version wraps the single-document pipeline in a loop, and running it on five real PDFs puts the whole issue in one table.

The single-document pipeline of Article 1 (the four-brick baseline, a PDF in and a highlighted answer out) has the signature pdf_qa_baseline(pdf_path, question). The collection version has the signature corpus_qa_baseline(question, pdf_paths). Everything between the two is a for-loop and a DataFrame.

def corpus_qa_baseline(question, pdf_paths, *, client=None, top_k=3): rows = [] for pdf in pdf_paths: answer = pdf_qa_baseline(pdf, question, top_k=top_k, client=client) text = answer.answer.strip() rows.append({ "doc_id": Path(pdf).stem, "answer": answer.answer, "start_page": answer.start_page_num, "confidence": answer.confidence, "has_answer": bool(text) and text.upper() != "NA", }) per_doc = pd.DataFrame(rows).sort_values( ["has_answer", "confidence"], ascending=[False, False] ).reset_index(drop=True) return CorpusAnswer(question=question, per_doc=per_doc)
No classification here, no index, no vocabulary, no filter. The function opens every document, asks every document the same question, and sorts what comes back. That is deliberate. It makes the mirror from section 3 visible with none of the machinery on top, and it is the honest place to measure from.

The question is “What is a Profile in the Cybersecurity Framework?”. Four of the five NIST PDFs cover cybersecurity from a different angle and the fifth covers AI risk. The Profile concept is defined in the framework document and nowhere else in the set.

One document answered: page 5 of the framework, the definition quoted, confidence 0.95. The other four returned nothing usable. Two came back with an empty string, one with the literal NA, and one with a bare slash.

That last one is worth a second look. The baseline decides whether a document answered by testing the output string: non-empty, and not the literal NA. A single slash passes both tests, so the sort promoted a document that had said nothing. A heuristic on the output is not a substitute for knowing, before the call, which documents could hold the answer.

The run took 14.3 seconds for five documents, so about 2.9 seconds per document for a parse plus one model call. The total is linear in the number of documents, because nothing in the loop is shared between them. Five documents at that rate is a demo. The broker’s shelf at that rate is roughly eight days of wall time for one question.

The pipeline is not broken. The right answer came from the right document with a page behind it. What is broken is the ratio. Four calls out of five did no work, and that share only rises as the shelf grows.

Three articles follow this one, one per shape. They can be read in any order, because the shapes do not depend on each other.

The pile. Article 14B (RAG on a folder of unrelated PDFs: one summary line per file is the whole index) takes the collection with no shared field. It needs the least new machinery of the three: the summary lines and the outlines are the whole index, and retrieval is the hierarchical routing of Part III run one level higher.

The typed corpus. Article 14C (the homogeneous corpus, where the business already knows the columns) takes the collection where question two came back yes. Its claim is the one that saves the most time in practice: on a homogeneous corpus you do not discover the schema, the business already holds it, and the interview that produces the columns takes an afternoon.

The case file. Article 14D (case files, one bundle about a single entity) takes the third shape. It has the least prior art of the three in the RAG literature, because its two hardest questions are not retrieval questions. Does piece A contradict piece B, and which expected piece is not here.

Two things stay outside this group and go to follow-up work.

Formats other than PDF. Word, Excel, PowerPoint and mail each need their own parser before any of this applies. The three shapes carry over unchanged; the parsing does not.

Document comparison. Asking what changed between version 1 and version 2 of a policy is a different operation from asking a question of a collection. The version chain is in scope here, because retrieval has to know which version answers. The diff itself is not.

Case files are often kept out of a corpus pipeline as an oddity. They are common enough in insurance, lending and healthcare that leaving them out means leaving out most of the work, which is why they get an article rather than a caveat.

A folder is not a corpus, and a corpus is not one thing. Three questions decide which of three shapes you have, they cost one conversation with one business user, and they are worth asking before a line of ingestion code is written.

What they buy is small and specific. Question one tells you whether you need relations and validity dates. Question two tells you whether there is a table to build. Question three tells you whether the row of that table is a document or a case.

The for-loop from section 4 survives all three answers. Every article in this group is about the same thing: shrinking what that loop iterates over, from every document on the shelf down to the handful that could plausibly answer.

Earlier in the series, on the pieces this article stands on:

Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG. The pile shape leans on each file’s table of contents, and this is what to do about the files that ship none.
Most RAG Hallucinations Are Retrieval Failures: Fix Retrieval, Not the Prompt. The single-document version of the argument in section 1: what the model can invent is decided upstream of the prompt.
Context engineering for RAG question parsing: from a raw question to typed fields that steer retrieval and generation. Where the typed fields in row 2 of the mirror table come from.
A Production RAG Pipeline in Action: Every Answer Typed and Cited. The single-document pipeline that the for-loop in section 4 calls once per file.
Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations. The four bricks the mirror table widens, on one document.
Loop Engineering for RAG Generation: Iterate top-k One at a Time. The same loop-or-batch decision, inside a document rather than across a folder.

Also referenced above, listed here without links: Article 1 (the four-brick baseline, a PDF in and a highlighted answer out), Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured), Article 14B (the folder of unrelated PDFs), Article 14C (the homogeneous corpus), Article 14D (case files), and Article 17 (querying a corpus, with the time window as part of the scope).

External sources:

Yunfan Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey, 2023 (arXiv 2312.10997). The naive, advanced, modular vocabulary the baseline in section 4 sits against.
Scott Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System, 2024 (arXiv 2401.05856). Field-measured failure points from three production systems; the missing-document and wrong-document ones are the first two rows of the table in section 1.2.
Anthropic, Introducing Contextual Retrieval, 2024 (anthropic.com). Measured evidence that even a strong embedding and BM25 hybrid degrades once passages lose their document context.
Darren Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization, 2024 (arXiv 2404.16130). The opposite bet on shape two: let a model discover the entities instead of asking the business to name the columns.
Jinyang Li et al., Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs, 2023 (arXiv 2305.03111). What querying the typed shape looks like once the table exists, measured on 95 real databases.