Part 1 argued that once RAG is in place, the model is a commodity the documents do the real work. This post is where that claim gets tested against a real deployment. A client wanted an AI knowledge interface built on internal specs and procedures. The model performed flawlessly in testing. Pointed at the real document library, it started confidently serving specs from a discontinued product line as current. The fix wasn't a better model, it was treating the document library as the actual deliverable, before the interface ever touched a live query.

The Setup

A client in industrial equipment manufacturing needed an internal AI knowledge interface, something technicians and engineers could query directly for specs, procedures, and warranty terms instead of digging through folders every time a question came up.

We said yes to the build. We should have said "yes, after we've seen your documents" first. Everything in this post traces back to that sequencing mistake.

What Was the Problem?

The model behaved exactly as it should have. The failure was upstream: the client's knowledge base had never been consolidated - three separate storage systems, years of drift, no reliable way to tell a current document from a retired one. Nothing was malfunctioning. The interface was retrieving accurately from a document set that had never been sorted out, which produced the same outcome as a malfunction: confident, wrong answers.

What the audit surfaced:

We'd already made the naming argument in a different context, our piece on why internal product naming costs businesses search visibility describes the same mechanic: shorthand that makes perfect sense to the people who wrote it and means nothing to a system trying to retrieve it. There, it cost search rankings. Here, it costs trust in the interface itself.

Once connected to the real repository, the system began surfacing torque specs from a discontinued line as though they applied to the current one, because both documents sat side by side with nothing to distinguish them. There was no defect to patch. There was a decision nobody had made: which file is the actual source of truth.

What We Planned

Before touching the AI layer again, the plan was a full data readiness audit, score the document set, remove duplicates, restructure how information was stored, standardize naming instead of patching symptoms after deployment.

Four phases, executed in order:

1. Audit - score every document for duplication, structure, metadata completeness, naming consistency

2. Consolidate - flag duplicate and superseded files, designate one authoritative version of each

3. Restructure - rebuild flattened tabular data (specs, pricing, compliance tables) into structured, machine-readable form

4. Standardize - align naming and codes with the language technicians and customers actually search in

This mirrors the argument in why AI agents need harness engineering: the infrastructure beneath a deployment is what makes it safe to ship, not the model sitting on top of it. A retrieval layer without a clean document foundation is an ungoverned deployment waiting to fail — it just hasn't failed yet.

How We Executed It

The audit found more than 340 duplicate or superseded files in active folders, and naming inconsistencies touching roughly a third of the product catalog. None of it got patched post-launch. All of it got resolved before the interface went live on real data:

• Duplicate and outdated files archived or removed, leaving one clear current version per document

• Flattened tables rebuilt into structured data the retrieval layer could actually parse

• Every remaining file tagged with ownership, date, and revision status

• Naming standardized so a technician's search term and an internal model code resolved to the same file

Only then did the interface reconnect to the live repository.

How It Helped Them

Incorrect-answer rate in internal testing fell from roughly one in four queries to under one in twenty. Support tickets on spec-related questions dropped the following quarter. But the detail that mattered most to the client wasn't a metric: engineers stopped feeling the need to cross-check the interface against the source manual, the exact behavior that had been blocking adoption before the cleanup started.

Featured Snippet: What's the Biggest Cause of RAG Failure?

The most common cause of RAG failure isn't the model, it's an unaudited document library containing duplicate, outdated, or inconsistently labeled files. When several versions of a document exist with nothing marking which is current, the retrieval layer has no way to choose correctly, and the system repeats whatever it finds with total confidence, right or wrong.

Frequently Asked Questions

Why did the system serve wrong answers if the model worked correctly?
The model retrieved exactly as designed. The documents were the fault line, duplicate and outdated versions sat side by side with nothing distinguishing them.

How long did the cleanup take before the interface could be trusted?
Several weeks, completed entirely before further AI work resumed. Skipping that step is what causes most rollouts to stall after launch instead of before it.

Did this require a new model or custom training?
No. The model was untouched throughout. Every fix happened in the retrieval layer, the documents underneath it.

What was the single biggest fix?
Consolidating duplicate and superseded files into one authoritative version each. That change alone drove most of the accuracy gain.

Does this pattern show up outside manufacturing?
Yes. Fragmented storage, duplicate versions, and inconsistent naming appear anywhere documents have accumulated for years, professional services and healthcare see it just as often.

The Takeaway

Part 1 claimed the model is the least important decision in a RAG project. This is that claim under pressure: a technically sound model, undermined entirely by a document library nobody had ever audited, resolved not by switching models, but by doing the unglamorous work of deciding what was actually true.

Considering something similar? Talk to Abacus Digital about a data readiness audit before you spend budget on the AI layer itself.