GraphRAG: What It Is and When You Actually Need It

Ask a vector search system why two clauses in two different contracts contradict each other, and it will hand you both clauses side by side. It won’t tell you they contradict. Nothing in a nearest-neighbor lookup is built to notice a relationship between two facts, only how similar their embeddings are. GraphRAG, short for graph retrieval-augmented generation, exists because that gap is structural, not a tuning problem, and closing it means changing what gets indexed, not just how fast something gets retrieved.

That distinction is also where most explainers on this topic stop being useful. Every vendor selling a graph database wants to tell you about the first half, the relational reasoning that approach enables. Fewer of them want to tell you about the second half: when that reasoning isn’t worth what it costs to build. Three of the pages currently ranking for GraphRAG are written by graph-database companies explaining, in effect, why you should buy a graph database. None of them tell you when you shouldn’t.

A baseline RAG pipeline splits documents into flat chunks, embeds each one, and at query time retrieves whichever chunks sit closest to the question in embedding space. That works well when the answer lives in one chunk. It falls apart on two specific failure modes, and Microsoft’s own GraphRAG documentation names both directly: connecting information that’s scattered across many documents through a shared attribute, and answering a question that needs a holistic understanding of an entire collection rather than one passage.

The multi-hop version of this failure is the one practitioners run into first. Ask which supplier delay caused a specific shipment to miss its deadline, when the delay is documented in one email thread and the shipment schedule lives in a different system entirely, and a vector similarity search retrieves both documents if you’re lucky. It does not connect them. The relationships between the delay and the shipment are exactly the part a flat chunk index throws away during indexing, because a chunk is text, not a structured data point with a known connection to anything else.

Relevant context in this failure mode isn’t missing. It’s just split across two retrieved chunks with no signal that they belong together. That’s the actual process GraphRAG changes, not retrieval speed, not embedding accuracy, but what gets captured about a document before retrieval happens at all.

GraphRAG Turns Retrieval Into Graph Traversal, Not Just Similarity Ranking

GraphRAG builds a knowledge graph from unstructured data before it ever answers a question. That’s the part worth being precise about, because “graph” here describes a data representation choice, not a specific product. The pipeline runs in two phases, and neither one requires any particular vendor’s graph database underneath it.

Indexing segments source text into analyzable units, then uses an LLM to extract entities, the relevant nodes of the eventual graph, along with the explicit relationships between them and any claims those entities support. Microsoft’s implementation clusters the resulting graph into communities using the Leiden technique, a hierarchical clustering algorithm, and generates a summary for each community from the bottom up. That community structure is what makes corpus-wide questions answerable later without re-reading every source document at query time.

Retrieval then runs through one of several graphrag retrievers, each suited to a different question shape. Global search answers a question about the entire dataset by working across community summaries rather than individual passages, the mechanism behind query focused summarization. Local search anchors to a specific entity and expands outward to its immediate neighbors for entity-specific questions. DRIFT search combines both, starting local and pulling in community context when a purely local answer runs thin. All three are documented directly in Microsoft’s GraphRAG query documentation, and none of them requires a specific graph store, only a graph.

That last point matters more than it sounds like it should. Traditional RAG and GraphRAG both ultimately hand a large language model some retrieved information and ask it to generate an answer. The difference is entirely upstream, in what got extracted and connected before retrieval ever runs, not in answer generation itself. A team evaluating this rag system is choosing a data-modeling approach, not a database vendor, whatever the marketing on any specific graph-database site implies.

Graph or Vector, Retrieval Is a Backend Your Agents Call

Whichever way the decision later in this piece goes, the retrieval layer ends up in the same place: GraphRAG’s indexing pipeline and query layer become a service something else calls, the same role a vector database already plays in most RAG systems. Nothing about that changes because the retrieval mechanism underneath is a graph instead of an index.

That matters because the governance question doesn’t change either. If a GraphRAG-backed retrieval layer is exposed as a tool an agent calls, whether through a direct integration or through an MCP server built specifically for agentic retrieval, the same access questions apply as for any other private data source: which agent can call this tool, what credentials does that call use, and who can see what got retrieved. Obot MCP Gateway is a centralized control plane that enforces fine-grained access control, integrates with existing identity providers, and maintains audit logs at the tool-call level, the same governance layer described in more depth for tool-level permissions and token brokering elsewhere on this site. Obot MCP Gateway is open-source under the MIT license, self-hostable on Kubernetes or Docker, and also available as a managed service, the same product either way.

Since Obot is our platform, one more note on where it fits. The gateway is one part of Obot’s Enterprise AI Control Plane, built around three jobs: define what agents can reach, choose where they run, and prove what they did. Obot Sentry covers retrieval servers a developer configures locally in an AI client like Claude Code or Cursor, which never pass through a gateway, and Claude Environments run Claude Code under the same policy off the developer’s laptop. 

Governance as an infrastructure-level guarantee doesn’t care whether the retrieval source underneath is a graph or a vector index. It cares whether every tool call is scoped, brokered, and logged, and that requirement is identical either way.

The Accuracy Numbers That Hold Up, and the Ones That Don’t

Two figures from this research area have a traceable source and a specific, checkable methodology. The rest of what circulates does not, and it’s worth being explicit about which is which rather than letting the good numbers launder the bad ones by association.

Lettria’s cross-industry benchmark, published via the AWS Machine Learning Blog and run across Amazon financial reports, COVID-19 vaccine research, aeronautical specifications, and EU environmental directives, compared Lettria’s own GraphRAG, a hybrid system that uses both a vector store and a graph store, against a vector-only baseline (Verba by Weaviate). The hybrid returned correct answers 80% of the time against 50.83% for the vector-only baseline, a gap that widened to 90.63% versus 46.88% in the industry-sector subset of the same test. The questions covered six types, fact-based, multi-hop, numerical, tabular, temporal, and multi-constraint, ranging from simple fact-finding to multi-layered reasoning, and Lettria’s in-house team graded the answers manually. Lettria also reports up to a 35% precision improvement from adding graph structure to a RAG pipeline. That’s one vendor’s own benchmark of its own product, not a universal ceiling, and it says nothing about latency or infrastructure cost. It’s also evidence for graph and vector retrieval working together, not for a graph replacing a vector index: the winning system kept the vector store.

Separately, and measuring something different, Microsoft Research’s own evaluation of the original GraphRAG paper found its global search retriever winning head-to-head judging against naive RAG in 72 to 83% of trials on comprehensiveness and 62 to 82% on diversity, and reported using intermediate and low-level community summaries at meaningfully lower token cost than source-text summarization, on the order of 20 to 70% of the token use per query, dropping to 2 to 3% for the highest-level community summaries. That figure is about token efficiency relative to summarizing raw source text, not a general claim about vector RAG, and the two evaluations should not be merged into one statistic. They come from different teams, testing different things, against different baselines.

What didn’t clear the bar for this article: a widely repeated claim that GraphRAG makes AI agents “80% more truthful,” which appears on at least one vendor blog with no visible citation or methodology. A separate, often-repeated claim that GraphRAG needs 26 to 97% fewer tokens does trace to a primary source: the same Microsoft paper (arXiv:2404.16130) reports that low-level community summaries needed 26 to 33% fewer context tokens than source-text summarization, and root-level summaries needed over 97% fewer. The figure is real. What usually gets lost when it’s repeated is the baseline: it’s measured against summarizing raw source text, not against vector RAG. If a figure doesn’t have a source that holds up, describing the pattern instead of the number is the more honest option, and that’s the standard applied here.

Graph Traversal Earns Its Cost on Relational, Cross-Document Questions

The workload characteristics that justify GraphRAG show up consistently across the sources that actually name them, not as abstractions but as specific query shapes. Complex queries requiring multiple logical hops, the supplier-delay example above, or a legal question that requires connecting a clause in one contract to an amendment in another, are the clearest case. So are corpus-level questions, “what are the recurring themes across this document set,” where an answer has to synthesize across large datasets rather than retrieve one passage.

A handful of domains show up repeatedly across independent sources as good fits for exactly this reason: fraud investigation, where the signal is a transaction’s connections to other accounts and devices, not any single transaction’s content; legal and compliance review, where obligations and exceptions are scattered across related documents; biotech and scientific research, where findings connect across a large, evolving literature; and investigative work generally, where the value is in the connections between facts, not the facts individually. Fraud detection in particular depends on explicit relationships, a transaction connected to an account connected to a device flagged elsewhere, that a nearest-neighbor lookup has no mechanism to surface.

Explainability is the other durable reason to reach for a graph. When an answer has to be traceable back to which entities and relationships produced it, not just which passages, a graph gives you that path directly. A vector index gives you the passages and asks you to reconstruct the reasoning yourself.

For Single-Hop Lookups, a Vector Index Already Wins

This is the section missing from every page currently ranking on this topic, and it’s the more useful half of the decision.

If the question is a single-hop factual lookup, “what’s our refund policy for annual plans,” a vector index answers it correctly, faster, and for less operational cost than standing up a graph pipeline to do the same job. Nothing about relational reasoning helps when there’s no relationship to reason across.

Indexing cost is not a footnote here. Running GraphRAG init sets up a project in minutes, but GraphRAG’s own GitHub README states plainly that “GraphRAG indexing can be an expensive operation,” and that warning belongs in the decision, not just the README. Building the graph means running entity and relationship extraction across every document with an LLM before a single query can be answered, a materially heavier data pipeline step than computing embeddings for the same corpus.

Staleness compounds that cost. A vector index updates by embedding new chunks. A knowledge graph needs new entities and relationships re-extracted and re-integrated into existing communities as source data changes, and major version bumps to the underlying tooling have historically required a provided migration notebook or a full re-index to move to the latest config format, not a drop-in upgrade. For a corpus that changes daily, that’s a process cost that has to be weighed against the relational benefit, not assumed away.

Latency belongs in the same column. Global search still means working across community summaries, sometimes several LLM calls deep, before an answer comes back, real overhead next to one direct nearest-neighbor lookup, even though those same summaries are more token-efficient than summarizing raw source text directly. Token efficiency relative to that alternative is not the same claim as being faster than a single vector lookup, and it isn’t one. For a small or single-document corpus, or a workload where every question is a simple lookup, that overhead buys nothing. Rely solely on the mechanism that actually earns its cost for the questions being asked, and for a large share of production retrieval workloads, that’s still a vector index.

Conclusion

The mechanism behind GraphRAG is well documented. Microsoft’s own docs explain the indexing pipeline and the three query methods in real depth. What’s missing from nearly every explainer currently ranking for this topic is the cost side of the same decision, and that omission isn’t accidental when the explainer comes from a company that sells the graph database underneath. The question worth asking is not whether GraphRAG works. In Lettria’s benchmark, a hybrid of graph and vector retrieval beat a vector-only baseline across six question types, from simple fact-finding to multi-constraint reasoning. The question is whether the workload in front of you needs the graph half of that hybrid, or is mostly simple lookups a vector index already answers for less.

FAQs

What is GraphRAG?


GraphRAG, short for graph retrieval-augmented generation, builds a knowledge graph of entities and relationships from source documents, then retrieves through graph traversal or community summaries instead of nearest-neighbor similarity search alone.

How is GraphRAG different from standard RAG?


Standard, vector-based RAG retrieves flat text chunks ranked by embedding similarity. GraphRAG extracts entities and relationships during indexing and can traverse those explicit connections at query time, which lets it answer multi-hop questions a similarity search alone can’t connect.

Do I need a specific graph database to use GraphRAG?


No. GraphRAG describes a retrieval pattern, extracting a knowledge graph and querying it through traversal or community summaries, not a specific vendor’s database. The graph store underneath is an implementation detail, not a requirement of the pattern itself.

What are global, local, and DRIFT search?


They’re the three query methods in Microsoft’s GraphRAG implementation. Global search answers whole-dataset questions using community summaries. Local search answers entity-specific questions by expanding to neighboring concepts. DRIFT search starts local and adds community context when a purely local answer isn’t enough.

When should I not use GraphRAG?


For single-hop factual lookups, small or single-document corpora, or any workload where indexing cost and the need to keep the graph current outweigh the benefit of relational reasoning you don’t actually need. A vector index answers those questions correctly for less operational cost.

Is GraphRAG more accurate than vector-based RAG?


In Lettria’s cross-industry benchmark, a hybrid GraphRAG system that combines graph and vector stores answered 80% of questions correctly against 50.83% for a vector-only baseline, across six question types from fact-based to multi-constraint, graded manually by Lettria’s own team. It’s one vendor’s benchmark of its own product, and the winning system still used a vector store, so it’s strongest as evidence that the two work better together.

Is GraphRAG expensive to run?


Indexing is the real cost. Microsoft’s own documentation states plainly that GraphRAG indexing can be an expensive operation, since it requires LLM-based entity and relationship extraction across the entire corpus before any query can be answered, a heavier step than computing embeddings for the same data.

Can GraphRAG and vector search be used together?


Yes, and this is common in practice. A vector-backed retrieval source can handle straightforward lookups while a graph-backed source handles the relational questions, with an agent or routing layer deciding which one to call for a given question. The strongest accuracy result cited in this article, Lettria’s benchmark, came from exactly that kind of hybrid.