Breaking the Single-Hop Ceiling: Graph RAG for Enterprise Production

Ask a standard RAG system "what is the warranty period of product X?" and it will nail it. The answer sits in one paragraph of one document, the embedding of your question lands right next to the embedding of that paragraph, and the model reads it back to you. Now ask "what systemic risks threaten Project Z?" The honest answer is a chain: Project Z depends on a supplier mentioned in a procurement contract, that supplier shares a parent company with a vendor flagged in an audit report, and that audit finding cites a regulation updated in a policy memo. No single chunk contains the answer. Vector search evaluates each fragment in isolation, so it can't see the chain, and the reasoning breaks at retrieval before the model ever gets a chance to think.

We've written about getting the most out of semantic search, hybrid search, and reranking. That toolkit is the right answer for single-hop retrieval, and it remains half of the architecture below. This post is about the other half: the questions where similarity is the wrong primitive altogether, and what it takes to put a knowledge graph into production next to your vector index.

Why Vector RAG Hits a Ceiling

Standard vector RAG splits enterprise content into chunks, embeds each one, and at query time returns the chunks whose vectors sit closest to the query's. Baked into that design is an assumption that's easy to miss: the information needed to answer the question lives inside a local text boundary. Real enterprise knowledge doesn't work that way. It's fragmented by design, spread across contracts, tickets, org charts, policies, and reports written by different teams for different purposes.

When a query needs clue A from document 1, clue B from document 2, and clue C from document 3, similarity search might even retrieve all three chunks. But it hands them to the model as an unordered pile with no explicit link between them. The topological, causal, hierarchical, and temporal relationships that make the clues an answer were never captured anywhere. The model either fails to connect them or, worse, connects them incorrectly and states the result with confidence.

Two anti-patterns teams reach for

When retrieval stops producing coherent answers, the instinctive fixes make things worse.

• Unbounded top_k expansion. If 5 chunks don't contain the chain, try 50, or 100, and bolt on a cross-encoder reranker to sort it out. This dilutes the context window and triggers the well-documented "lost in the middle" effect: the handful of chunks that matter get buried in low-signal noise, attention scatters, latency climbs, and token cost grows with every chunk you add. You've paid more to make the answer harder to find.
• Entity alias dissociation. Enterprise corpora name the same real-world thing many ways: a chemical name in one document, a brand name in another, an internal abbreviation in a third. Embeddings bring these closer together but can't guarantee they converge. Facts about one entity end up scattered across what the system treats as three unrelated things, and any multi-hop path through that entity is broken before a query is ever run.

Both are symptoms of the same root cause. The system has no explicit representation of things and the relationships between them. That's exactly what a knowledge graph is.

Building the Graph Without Burning the Budget

The obvious way to build a knowledge graph from documents is to run a frontier model over every page and ask it to extract entities and relationships. At enterprise scale that's an inference bill nobody signs off on, and it's slow enough to make keeping the graph current impractical. The alternative is a three-step asynchronous construction engine that spends expensive compute only where it pays for itself.

The design principle is the same one behind any good funnel: let the cheap model maximize recall, and let the expensive model maximize precision on a much smaller set. The SLM doesn't have to be right; it has to not miss things. The reasoning model never sees raw documents, only batched candidate triplets, which is a far smaller and more structured input.

Step 3 deserves the most attention because it's the one teams skip. Entity resolution compares candidate duplicates across several attributes at once: their semantic profiles, the patterns in where and how often they appear, and overlapping functional attributes. Two nodes that are described alike, show up in the same contexts, and do the same things get collapsed into one entry under a single GUID. Without this step you don't have a knowledge graph; you have a pile of disconnected islands that happen to be stored in a graph database.

Provenance on Every Triplet

The graph lives in a distributed graph store such as Neo4j or NebulaGraph, for horizontal scale and low-latency traversal. The store matters less than one non-negotiable rule about what goes into it: no node or edge exists without its provenance. Every triplet insertion carries immutable metadata pointing back to the source document's fingerprint and the exact chunk it was extracted from.

Triplet = ⟨SubjectGUID, Predicate, ObjectGUID⟩ + Metadata{DocID, ChunkID}

This looks like bookkeeping, and it's the most important decision in the architecture. That back-link is the data lineage layer every later capability depends on: citations that point to real sentences, audits that can trace a conclusion to its evidence, and targeted cleanup when a source document is corrected or withdrawn. A triplet with no provenance is a claim you can't verify, and an unverifiable claim in front of an enterprise user is a liability.

Route by Intent, Not by Default

Graph traversal is powerful and it isn't free. Running a global graph query to answer "what's the warranty on X?" wastes compute and adds latency for no gain. So a lightweight intent classifier sits at the front door and sends each query down the cheapest path that can actually answer it.

Most enterprise traffic is single-hop, which is good news: most queries never touch the graph and cost no more than they do today. The graph earns its keep on the minority of questions that vector RAG answers badly, and those tend to be the high-stakes ones.

The dual-path hybrid engine

When a query is routed to the hybrid engine, two branches run in parallel:

1. Graph traversal. Identify the seed entities in the query, traverse outbound relationships one to three hops, extract the local subgraph as a chain of textual triplets, and tag the results Origin: Graph.
2. Vector retrieval. Run standard dense similarity search over the unstructured corpus and tag the returned chunks Origin: Vector.

The branches cover each other's blind spots. The graph contributes structure, the explicit chain linking Project Z to the regulation. The vectors contribute texture: the nuance, caveats, and narrative detail that never become a clean triplet.

One reranker for two kinds of evidence

Both result sets feed into a single cross-encoder reranker that scores every candidate on two dimensions: semantic fit to the query, and continuity with the topological path the graph found. It strips peripheral noise, near-duplicates, and context blocks unrelated to the chain, and what remains is a small, dense, coherent payload. This is the opposite of the top_k expansion anti-pattern: instead of giving the model more to sift through, you give it less, and what you give it is connected.

White-Box Generation

Good retrieval can still be wasted at generation time. Three practices keep the model grounded in what was retrieved.

Inject structure, not raw dumps

Never paste retrieved context into the prompt as an undifferentiated blob. Graph paths are serialized as explicit structural strings that make the dependencies impossible to miss:

[Entity A → (Rel 1) → Entity B → (Rel 2) → Entity C]

The model doesn't have to infer that the supplier connects to the audit finding; the context states it.

Constrain the reasoning

System instructions require explicit chain-of-thought: the model must write out each discrete deduction step before it's allowed to state a conclusion. That scaffolding keeps the model on the paths the evidence supports and makes speculative leaps visible, both to the model and to anyone reviewing the output. An unsupported step is easy to spot when it has to be written down.

Trace every claim to its sentence

This is where the provenance rule pays off. Because every triplet keeps its metadata back-link, every assertion in the answer can carry a citation that resolves all the way down the chain:

1. The answer cites a triplet by its GUID.
2. The triplet resolves to its document and chunk metadata.
3. The metadata resolves to the raw source text, highlighted.

In the product, a user can hover over any risk or compliance conclusion to render the subgraph behind it, or jump straight to the source sentence in the original document. For regulated industries this is the difference between a tool people can use and one they can't: an answer that shows its evidence turns a black box into something an auditor can sign off on.

Explainability is not a feature you add at the UI layer. If provenance isn't captured at ingestion, no amount of frontend work can reconstruct it. The citation experience is decided the day you design your triplet schema.

Running a Live Graph in Production

A knowledge graph isn't a build artifact you generate once. Documents arrive daily, facts change, and extraction models make mistakes. Operating one well looks a lot like operating any other production backend service: metrics, safe deploys, and rollback.

Measure graph quality automatically

You can't manually inspect a graph with millions of edges, so four automated metrics stand in for inspection:

• Entity extraction accuracy: are the nodes real, correctly typed entities?
• Relation extraction accuracy: are the edges true statements supported by the source?
• Resolution success rate: are aliases being merged correctly, without false merges?
• Knowledge coverage rate: how much of the corpus's actual knowledge made it into the graph?

Incremental hot updates with a circuit breaker

Rebuilding the whole graph whenever a document changes doesn't scale. Instead, new documents trigger isolated, local extractions that merge into the master graph incrementally, with no full recompilation.

Incremental updates make it easy to push bad data into production fast, so they need a safety valve. When an extraction's confidence score falls below a set threshold, an automated circuit breaker trips: the low-confidence triplets are flagged, kept out of the live query path, and routed to a human-in-the-loop review queue. Every mutation is versioned, so if regression tests fail after an update, the graph rolls back to its last stable snapshot in seconds. It's the same discipline as a canary deploy, applied to data instead of code.

Test in tiers of difficulty

Every change to the system prompt, the reranker configuration, or the graph structure goes through A/B regression testing against a test suite stratified into four difficulty levels, from Level 1 (single-hop facts) up to Level 4 (synthesis across business lines). Level 1 checks that you haven't regressed the baseline that vector retrieval already handles. Level 4 checks long-range structural reasoning and whether hallucinations stay suppressed under heavy noise. The levels in between track the routing table: hierarchical traversals, then multi-hop synthesis within one domain.

Stratifying matters because an aggregate score hides the trade you care about. A change that lifts Level 4 by ten points while quietly dropping Level 1 by three can look like a net win while degrading the queries most users actually send. (For more on building eval suites that catch this, see our posts on evaluating AI agents and release gates and drift.)

A Field Checklist

• Diagnose before you build. If your failures are single-hop misses, fix chunking, hybrid search, and reranking first. Reach for a graph when answers require connecting facts across documents.
• Don't scale top_k to fix reasoning. More chunks dilute context, cost more, and bury the evidence. The fix is structure, not volume.
• Funnel graph construction. A cheap SLM for recall over everything, a reasoning LLM with domain few-shot prompts for precision on the candidates.
• Treat entity resolution as mandatory. Unmerged aliases break the paths that justify having a graph.
• Require provenance on every triplet. Document fingerprint and chunk ID, immutable, from the first insert. Citations and audits depend on it.
• Route by intent. Vector for single-hop lookups, graph for structure, hybrid only for genuine multi-hop synthesis.
• Rerank graph and vector evidence together on both semantic fit and path continuity, and send the model a small, connected payload.
• Serialize paths explicitly and require step-by-step reasoning before conclusions.
• Operate the graph like a service. Automated quality metrics, incremental updates, a confidence circuit breaker, versioned snapshots, and tiered regression tests.

Closing Thoughts

Vector RAG made it possible to point a language model at a company's documents. It didn't make it possible to reason across them, because similarity search has no concept of how one fact relates to another. Graph RAG supplies that missing layer: explicit entities, explicit relationships, and a provenance trail from every conclusion back to the sentence it came from.

None of the components is exotic. Funnel-shaped pipelines, entity resolution, intent routing, rerankers, circuit breakers, and versioned rollbacks are all established engineering. What changes is how they fit together: retrieval follows deterministic paths instead of piling up loosely related text, and the model stops guessing at connections and starts reading them. That's what turns an LLM from a confident summarizer into a reasoning engine an enterprise can verify.

References & further reading:
Edge et al.: From Local to Global: A Graph RAG Approach to Query-Focused Summarization · Liu et al.: Lost in the Middle: How Language Models Use Long Contexts · Peng et al.: Graph Retrieval-Augmented Generation: A Survey · Neo4j documentation · NebulaGraph documentation

← Back to all posts