Building Agentic RAG with MCP: A Practical Guide

Meilisearch’s own agentic RAG guide answers the question “what is MCP” with a definition: a prompting technique for structured output design, unrelated to retrieval. It is wrong. MCP is Model Context Protocol, the standard that lets an agent discover and call tools, including retrieval tools, at runtime instead of a developer hardcoding a specific SDK method for a specific vector database. One of the more detailed agentic RAG guides currently ranking gets the one piece of infrastructure most relevant to its own topic backwards.

That is not an isolated slip. It is a symptom. Retrieval augmented generation has a well-covered reasoning layer and a barely-covered plumbing layer, and the plumbing layer is where production teams actually spend their time.

Core Principles: What Agentic RAG Adds Over a Baseline Pipeline

Baseline RAG runs one retrieval phase, then generation. A query comes in, an embedding gets computed, the nearest vector embeddings come back from a vector search, and the large language model writes an answer from whatever it got. The model never asks whether the retrieved context was actually sufficient. It just uses it.

Agentic RAG system design puts a reasoning step in front of and inside that loop. The agent decides whether to retrieve at all, what to retrieve, whether the retrieved information actually answers the user query, and whether to retrieve again, from a different source, before generating anything. It is a loop with a judgment call at every turn, not a fixed data pipeline. This is the framing the January 2025 arXiv survey, Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, formalizes across a taxonomy of agent patterns: reflection, planning, tool use, and multi-agent collaboration layered onto the retrieval step.

None of that is new framing. Every one of the sixteen top-ranking pages on this topic covers this distinction in some form, whether they call it agent-based retrieval-augmented generation or agentic RAG outright. Where they stop is the next question: what is the agent actually calling when it decides to retrieve, and what happens to that call once there is more than one retrieval target to choose from.

The arXiv survey’s own taxonomy is instructive here because of what it leaves out, not what it covers. It categorizes agent patterns in real depth, reflection loops, multi-step planning, multi-agent handoffs, but treats the tool the agent calls to retrieve as an implementation detail below the level the taxonomy operates at. That is the same gap that shows up across every implementation guide in the SERP: rich treatment of the decision to retrieve, thin-to-nonexistent treatment of what retrieving actually connects to.

Two of those patterns matter enough to name specifically, because they show up later in this guide as concrete MCP tool-call logic rather than abstract taxonomy. Planning, in the survey’s framing, means the agent can break a complex task into a sequence of smaller retrieval and reasoning steps instead of treating retrieval as one atomic action. Multi-agent collaboration means that sequence doesn’t have to run inside a single agent at all: multiple specialized agents, each scoped to a different retrieval job, can hand work to each other and combine partial answers. Both patterns describe when and how many times to call a retrieval tool. Neither one says what that tool call looks like on the wire, which is the question this guide actually answers. Zoom out further and both sit inside a broader set of agentic workflow patterns, prompt chaining, routing, parallelization, and evaluator-optimizer loops among them, where the shared idea is that tasks requiring dynamic decomposition don’t get solved by a single fixed pipeline; they get split, routed, and checked, and retrieval is just one of the steps that split lands on.

Where MCP Fits in a RAG System: Retrieval Is Just a Tool Call for an AI Agent

In an agentic loop, “retrieve” is one tool the agent has access to, alongside whatever else it can call. Most implementation guides answer “how does the agent call the retrieval tool” with a hardcoded SDK integration: an Azure AI Search client, a Chroma client, a FAISS index loaded in-process. The tool is whatever the developer wrote code against.

Model Context Protocol answers the same question differently. An MCP client, embedded in the agent’s runtime, connects to an MCP server and asks it what tools it exposes. The server answers with a list, the agent picks one, and the call happens over a standard JSON-RPC message format rather than a vendor-specific function signature. Adding a new retrieval source stops being a new SDK integration a developer ships code for. It becomes a new MCP server the agent can discover and call, using the same client logic it already has.

That distinction matters more than it sounds like it should, because it changes who can add a relevant context source and how that access gets governed later. A hardcoded integration is owned by whoever wrote it. An MCP connection is owned by whoever operates the gateway in front of it.

The Architecture: Autonomous AI Agents, MCP Client, MCP Server, Data

None of the sixteen top-ranking pages on agentic RAG draw this. The closest, Microsoft’s Azure Architecture Center guide, diagrams a ReAct loop calling information retrieval tools directly through Azure AI Search’s SDK. Swap the direct SDK edge for an MCP connection and the shape changes:

Building Agentic RAG with MCP: A Practical Guide
Diagram: an agentic RAG loop where the retrieval tool call passes through an MCP client, an optional gateway for auth and audit, and one or more MCP servers, each fronting its own data source.

The gateway is optional in the protocol and load-bearing in production. Without it, each MCP server negotiates its own auth, and the agent’s credentials for server A tell you nothing about what it can do on server B. With it, every tool call, regardless of which server answers it, passes through one point that can enforce scope and write an audit log entry, the same control-and-observability argument that applies to MCP traffic generally, not something specific to retrieval.

Direct SDK integrationRetrieval through an MCP server
Adding a new sourceNew client library, new integration code, new error handlingNew MCP server the agent already knows how to call
CredentialsWhatever the developer’s code holds, no separate layerScoped, brokered per connection if a gateway is in front
Auth model per sourceDifferent for every vendor SDKSame JSON-RPC exchange regardless of what’s behind it
Audit trailOnly if the developer built one, source by sourceOne tool-call log across every connected server, if a gateway is present
LatencyLowest, in-process callOne JSON-RPC round trip, more over Streamable HTTP
Right fitOne stable, high-QPS source, no cross-team governance needMultiple sources, multiple owners, or a real audit requirement

Governing the Retrieval Layer for Multi-Agent Collaboration Once It’s Not One Server Anymore

The table above makes one row look easy: “one tool-call log across every connected server, if a gateway is present.” Once retrieval spans more than one MCP server, as the multi-server pattern later in this guide shows, the questions behind that row get concrete: who is allowed to add the next server, and who can see what the agent actually pulled from each.

This matters most for the servers that front private data, customer records, internal financial data, anything that isn’t a public document set, where the cost of an unscoped credential is highest. A direct SDK integration inherits whatever credentials the developer’s code holds, full stop. There is no separate access-control layer to reason about, because there is no layer, just a client library and a connection string sitting in an environment variable.

Two problems show up as soon as retrieval spans more than one MCP server. The first is scope: server-level access to an MCP server, the default most setups start with, means an agent that can call one tool on that server can call all of them, a gap tool-level permissions are built to close. The second is where the credential itself lives; a client holding a raw OAuth token can leak it, replay it, or hold onto it past when it should still work, which is the specific failure mode brokering tokens through a control plane instead of the client is meant to prevent.

Obot, our platform, is built for exactly this layer. It’s an Enterprise AI Control Plane organized around three jobs: define what agents can reach, choose where they run, and prove what they did.

Gateway: Define What Agents Can Reach

Obot MCP Gateway is a centralized control plane that sits between the agent’s MCP client and every MCP server it talks to, enforcing fine-grained, tool-level access control and integrating with existing identity providers, so a scoped, short-lived credential is issued per connection instead of one static key that works everywhere. It maintains audit logs at the tool-call level, following a schema built for exactly this question: which agent called which tool, on which server, and when, so a security team can answer “which agent pulled from the customer-records server” without instrumenting each retrieval source separately. Obot MCP Gateway is open-source under the MIT license, self-hostable on Kubernetes or Docker, and also available as a managed service, the same product either way, built for organizations that need that governance without routing sensitive retrieval traffic through third-party infrastructure they don’t operate.

Device: Watch and Enforce With Obot Sentry

Notice how the code in the next section runs mcp-server-qdrant: as a local stdio subprocess on the developer’s own machine. That’s the normal way retrieval servers start, and it never passes through a gateway. Obot Sentry scans AI client configurations in tools like Claude Code, Claude Desktop, Codex, Cursor, and VS Code, reports which MCP servers, skills, and plugins each machine loads, and can block servers that aren’t approved, so a retrieval server pointed at private data doesn’t stay invisible just because it started on a laptop.

Hosted: Claude Environments

Obot v0.27 added Claude Environments, hosted Claude Code environments that run under the same control plane, so an agent’s retrieval connections come from policy rather than a local config file.

This is governance as an infrastructure-level guarantee, applied once at the gateway, not a per-server implementation bet that has to be re-litigated every time a new retrieval source gets added. That distinction is easy to underrate until the second team ships its own retrieval server with its own auth assumptions, and the question “can agent X read from server Y” no longer has one answer, it has as many answers as there are servers to check.

Building It: An Agentic Retrieval Augmented Generation Loop Over an MCP Server

This is the part every guide that mentions MCP skips. Here is retrieval running through an actual MCP server rather than a vector database SDK, using langchain-mcp-adapters (v0.3.2 when this guide was written; check PyPI for the latest) to bridge an MCP connection into a LangGraph-compatible agent, and mcp-server-qdrant, Qdrant’s own official MCP server implementation, as the retrieval backend. It’s official and actively maintained, which matters: not every third-party MCP server on the public registries is either, and that vetting step comes before this one, not after.

The server exposes exactly two tools: qdrant-store and qdrant-find. No custom wrapper code is required to expose them, that part is the server’s job:

QDRANT_URL="http://localhost:6333" \
COLLECTION_NAME="support-docs" \
EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2" \
uvx mcp-server-qdrant

On the agent side, the MCP client connects, pulls the tool list, and hands those tools to a standard LangChain agent. The agent decides when to call qdrant-find, it is not wired to call it unconditionally on every turn:

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from langchain_mcp_adapters.tools import load_mcp_tools
from langchain.agents import create_agent

server_params = StdioServerParameters(
    command="uvx",
    args=["mcp-server-qdrant"],
)

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        tools = await load_mcp_tools(session)
        agent = create_agent("openai:gpt-4.1", tools)
        response = await agent.ainvoke({
            "messages": "What's our current refund policy for annual plans?"
        })

load_mcp_tools does the translation work: it takes whatever qdrant-find and qdrant-store describe about themselves over MCP and converts that into tool definitions the LangChain agent’s model can call like any other function. The agent, not a fixed pipeline step, decides whether the retrieved passage actually answers the refund-policy question or whether it needs to call qdrant-find again with a narrower query. That decision loop, not the retrieval call itself, is what makes this agentic retrieval augmented generation rather than a single-pass lookup.

One protocol note: the session.initialize() call is the SDK’s connection handshake. The 2026-07-28 MCP specification made the protocol stateless, so each request now carries its own protocol version and capabilities, and how a given SDK version handles that change is worth checking in its release notes before you copy this pattern into production.

Multiple MCP Servers: Agentic Workflow Patterns for Routing Retrieval Across Sources

Every implementation guide in the current SERP assumes one vector database and stops there. Scaling agentic RAG systems past a demo usually means pulling from multiple data sources at once: a vector store for unstructured documentation, a separate internal API for structured customer data, maybe a third source for something else entirely. Each becomes its own MCP server rather than its own bespoke SDK integration, which is convenient right up until the number of servers an agent talks to is the same sprawl problem MCP adoption creates everywhere else in the stack, just relocated to the retrieval layer:

from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain.agents import create_agent

client = MultiServerMCPClient({
    "docs": {
        "command": "uvx",
        "args": ["mcp-server-qdrant"],
        "transport": "stdio",
    },
    "customer_records": {
        "url": "https://internal-api.example.com/mcp",
        "transport": "http",
    },
})

tools = await client.get_tools()
agent = create_agent("openai:gpt-4.1", tools)

The agent now sees tools from both servers in one flat list and picks the one whose description matches the complex query it’s trying to answer. Nothing about the agent’s reasoning loop changes. What changes is that a third data source, next quarter, is another entry in that dictionary rather than another SDK a developer has to learn, wire up, and maintain error handling for.

Adaptive RAG and Corrective Retrieval: Where Strategy Logic Lives Above the MCP Call

None of the code above tells the agent when to call a tool a second time or when to give up on one server and try another. That logic sits one layer above the MCP connection, in the agent’s own decision loop, and two named research patterns describe it more precisely than “the agent decides.”

Adaptive-RAG (Jeong et al.) routes a query based on its complexity before retrieval even happens: a simple factual question might skip retrieval entirely and go straight to the model, a moderately complex one gets a single retrieval pass, and a genuinely multi-hop question triggers iterative, multi-step retrieval. A support bot fielding routine customer service queries might resolve most of them with no retrieval call at all and only reach for qdrant-find on the fraction that actually need a document. Applied to an MCP-based setup, that classification decides how many times the agent calls a tool, and whether it calls it once against one server or works through several retrieval strategies in sequence, not whether MCP is involved at all. The choice between a dense retrieval model doing semantic search and a keyword based retrieval technique sits at the same layer; MCP exposes either kind of tool identically, so switching retrieval methods is a server-side decision, not an agent-side rewrite.

Corrective RAG (Yan et al.) adds a check after retrieval instead of before it: a lightweight evaluator grades whatever came back as correct, incorrect, or ambiguous, and triggers a corrective action, refining the query and retrieving again, or falling back to a broader search, when the grade is poor. That evaluator is exactly the “whether the retrieved information actually answers the user query” judgment call described earlier in this guide. Wired into the loop from the Building It section, a failing grade on the result of qdrant-find is what decides whether the agent calls it again with a narrower query, tries customer_records instead, or answers with a caveat.

Neither pattern is an MCP feature. Both are decision logic that happens to call MCP tools as its mechanism, the same way multi-agent collaboration can mean specialized agents, rather than a single agent holding tool access to everything, coordinating through sophisticated orchestration mechanisms. Each autonomous agent in that setup is scoped to its own retrieval server. That scoping question, which agent can call which server, is exactly what a gateway in front of the MCP layer is positioned to enforce, covered earlier in this guide.

Graph RAG: A Different Retrieval Mechanism Behind the Same MCP Call

Not every retrieval source is a vector database, and MCP doesn’t care which kind sits behind the tool it’s calling. Vector search finds passages by vector similarity, nearest neighbors in embedding space. Graph retrieval augmented generation finds an answer by walking explicit relationships between entities in a knowledge graph: a graph database storing who is connected to what, rather than which passages read similarly. Microsoft’s GraphRAG project implements this as its own retrieval layer, with GraphRAG retrievers for different query shapes, global search for questions that need a summary across an entire dataset, local search for questions anchored to specific entities, and DRIFT search combining both, a distinction documented directly in Microsoft’s GraphRAG query documentation.

The point for this article isn’t to relitigate GraphRAG versus baseline RAG. It’s that both are just a data representation choice behind a retrieval tool, structured data in a graph versus unstructured data in a vector index, and MCP standardizes the call regardless of which one answers it. A knowledge-graph-backed MCP server and a vector-backed MCP server expose the same kind of interface to the agent: a tool with a description and a set of parameters. The agent doesn’t need to know whether relevant nodes came from a graph traversal or relevant information came from a nearest-neighbor lookup, only whether the result actually answers the user query.

The case for reaching for a graph in the first place isn’t hypothetical. In Lettria’s cross-industry benchmark, run across Amazon financial reports, COVID-19 vaccine research, aeronautical specifications, and EU environmental directives, Lettria’s GraphRAG, a hybrid system that uses both a vector store and a graph store, returned correct answers 80% of the time against 50.83% for a vector-only baseline (Verba by Weaviate), with the gap widening further in the industry sector subset (90.63% versus 46.88%). The questions covered six types, fact-based, multi-hop, numerical, tabular, temporal, and multi-constraint, from simple fact-finding to multi-layered reasoning, and Lettria’s in-house team graded the answers manually. Lettria reports up to a 35% precision improvement from adding graph structure to a RAG pipeline. That’s a vendor’s own benchmark, not a universal ceiling, and it says nothing about retrieval speed or infrastructure cost. It’s also evidence for exactly the hybrid approach this section ends on: the winning system kept its vector store and added a graph. Microsoft’s own evaluation of the original GraphRAG paper measured a related but distinct dimension: comprehensiveness and diversity of answers to whole-dataset questions, where its global search retriever won head-to-head judging against naive RAG in 72 to 83% of trials on comprehensiveness and 62 to 82% on diversity.

A concrete example: fraud detection over a knowledge graph. Fraud detection is one of the clearer cases for graph-based retrieval over vector search, because the signal is relational: a transaction connected to an account connected to a device connected to three other accounts flagged last week is a pattern vector similarity alone won’t surface, but a graph traversal finds directly by following explicit relationships outward from the transaction in question. An agentic RAG loop investigating a flagged transaction can call a GraphRAG-backed MCP server for that traversal and a vector-backed MCP server for supporting case notes, in the same reasoning loop, through the same MultiServerMCPClient pattern shown above, one of the more concrete cases for hybrid retrieval strategies rather than picking a single retrieval mechanism upfront.

Where This Breaks: A Comparative Analysis of Latency, Overhead, and Direct Integration Tradeoffs

MCP is not a free upgrade. Every tool call now crosses a JSON-RPC boundary and, if the server runs remotely over Streamable HTTP rather than as a local stdio subprocess, a network hop that a direct, in-process SDK call never had to make. For one stable, high-QPS retrieval source with no governance requirement beyond “this one service can read this one database,” the direct integration most of the SERP defaults to is not a mistake. It is the right amount of infrastructure for the problem.

MCP earns its cost on the axis every guide currently ignores: the number of retrieval sources, the number of teams that own them, and the governance requirement that shows up the moment more than one of either exists. A single hardcoded vector database client scales fine until the second source shows up owned by a different team with a different security posture. That is the point at which “just call the SDK” stops being simple and starts being a problem for teams that rely solely on convention: every new integration re-implements auth, re-implements error handling, and re-implements whatever access logic the last one had, with no shared enforcement point across any of them.

None of this touches the agent’s own judgment. Prompt tuning on the retrieval-decision logic, when to call a tool again, how to judge whether a result is sufficient, matters exactly as much with MCP in the loop as without it. MCP fixes what the agent calls and who governs the call. It has nothing to say about how well the agent decides to call it.

Conclusion

The reasoning loop was never the hard part of agentic RAG. Every framework in the current SERP has one: retrieve, evaluate, decide, retrieve again if needed. The hard part, the part missing from all sixteen of the pages currently ranking for this topic, is making new retrieval sources addable and governable without turning each one into a bespoke integration project. MCP does not change what the agent decides. It changes what adding the next source costs, and who gets to see what happened after it did.

FAQ: Agentic RAG, MCP, and Large Language Models

What is agentic RAG?

Agentic RAG is a RAG architecture where an agent reasons about retrieval instead of running it as one fixed step. It decides whether to retrieve, what to retrieve, whether the result is sufficient, and whether to retrieve again from the same or a different source, before generating a final answer.

How is agentic RAG different from traditional RAG?

Traditional RAG performs one retrieval pass and generates from whatever comes back, with no evaluation step. Agentic RAG wraps retrieval in a reasoning loop: the agent can reject an insufficient result, refine the query, or call a different source, iterating before it answers.

How does Obot fit into agentic RAG with MCP?

Obot governs the retrieval layer, not the agent’s reasoning. Obot MCP Gateway scopes which agent can call which retrieval server and logs every tool call, Obot Sentry finds retrieval servers developers run locally in AI clients, and Claude Environments run Claude Code with its connections set by policy.

What does MCP have to do with RAG?

MCP standardizes the retrieval tool call itself. Instead of an agent’s retrieval step being hardcoded to one vector database’s SDK, the agent connects to an MCP server, discovers what retrieval tools it exposes, and calls them the same way it would call any other MCP tool.

Can an AI agent use multiple MCP servers for retrieval in one RAG loop?

Yes. Libraries like langchain-mcp-adapters support connecting to several MCP servers at once through a single client and exposing all of their tools to one agent, which then picks the right tool for each query regardless of which server it lives on.

Is MCP faster or slower than direct API integration for retrieval?

Slower, marginally. Every MCP tool call adds a JSON-RPC round trip and, over Streamable HTTP, a network hop that an in-process SDK call skips. For a single low-latency source that overhead is a real cost. It becomes worth paying once there are multiple sources or teams involved and governance stops being optional.

Do I need an MCP gateway to build agentic RAG with MCP?

No. An agent can connect directly to an MCP server with no gateway in between, and the code examples in this guide do exactly that. A gateway becomes necessary once more than one server is involved and something needs to enforce consistent auth, scope, and audit logging across all of them rather than leaving each server to handle it independently.

What’s a real example of an MCP server used for retrieval?

mcp-server-qdrant, Qdrant’s own official MCP server, exposes two tools, qdrant-store and qdrant-find, that let an MCP client store and semantically search vector data without any custom retrieval code on the agent side.

How do you secure an MCP-based retrieval layer?

By routing MCP connections through a gateway that issues scoped, short-lived credentials per connection rather than one static key, and that logs every tool call, so which agent retrieved what, from which source, and when is answerable from one place instead of reconstructed from separate server logs after the fact.