Agent RAG architecture: components, control loop, and tradeoffs

An agent RAG architecture turns retrieval from a mandatory pipeline stage into a decision inside a controlled loop. The system may search one source, inspect the result, rewrite the query, search somewhere else, or stop when the available evidence cannot support an answer. That flexibility is the point. It is also where most of the extra cost, uncertainty, and security work enters.

The phrase is often drawn as an agent box connected to a vector database. That picture leaves out the parts that determine whether the design survives production: identity, source-level authorization, state, evidence provenance, budgets, termination, and a trace of every decision. The language model is only one component. It should propose bounded retrieval actions, not become the policy engine for the whole system.

This guide uses "agent RAG" and "agentic RAG" for the same architecture pattern. It does not require a group of cooperating agents. One coordinator running a graph with conditional branches can be agentic if it decides when and how to retrieve. Multiple agents are an optional decomposition, not the definition.

From fixed pipeline to control loop

The original RAG paper joined a sequence-to-sequence model with an explicit, non-parametric memory retrieved from a dense Wikipedia index. In the familiar application version of that pattern, code takes a question, runs a predetermined search, adds the returned passages to a prompt, and asks a model for an answer.

That fixed path is still a sound default. It is predictable, relatively easy to cache, and simple to evaluate. The application designer chooses the index, query transformation, filters, top-k value, reranker, and stopping point before the request begins.

Agentic RAG moves some of those choices into request-time control. A model or model-assisted router can decide whether retrieval is needed, select an allowed source, decompose the question, issue a tool call, inspect the observation, and choose the next step. The ReAct paper supplied the influential reasoning-action-observation pattern. Current Microsoft architecture guidance applies that loop directly to retrieval tools and intermediate evidence.

The back edge changes the architecture. A fixed pipeline flows toward an answer. An agent RAG graph can return from an evidence check to planning or query construction. Every back edge needs a reason to exist, a limit, and an event in the trace.

Adaptive retrieval predates today's agent frameworks. Self-RAG was designed around retrieval on demand plus critique of passages and generated claims. Its premise is useful here even when the implementation is different: retrieving a fixed number of passages for every question can add irrelevant context, while adaptive retrieval needs an explicit way to judge what came back.

Agentic control earns its keep when later searches depend on earlier findings. A question may require a product identifier from an internal catalog before a regulator can be queried, for example. It also helps when the correct source cannot be known from the initial wording, or when retrieval quality can be judged well enough to justify a retry. If every valid question maps to the same index and one retrieval pass, the loop adds machinery without changing the answer path.

The components of an agent RAG architecture

A production diagram needs more than an agent, a model, and a vector store. The following layers separate decisions that may be probabilistic from controls that must hold on every run.

Component	Responsibility	Where deterministic control matters
Request admission and policy	Authenticate the caller, establish tenant and user scope, classify the request, and set budgets	Identity, allowed sources, data region, tool permissions, and hard limits
Coordinator	Read current state and propose the next retrieval, evaluation, synthesis, or escalation step	The runtime validates every transition against the graph and remaining budget
Retrieval tool registry	Present narrow, typed contracts for search systems, databases, files, or external APIs	Tool allowlists, parameter schemas, timeouts, result-size caps, and secret handling
Retrieval plane	Transform queries, apply filters, search, rerank, and return source records	Authorization filters must be applied by the data service or trusted adapter, not inferred by the model
State and evidence store	Retain the original question, subquestions, tool observations, source IDs, scores, costs, and unresolved claims	Append-only evidence references, isolation between users, expiry, and redaction
Evidence evaluator	Judge relevance, coverage, contradiction, freshness, and whether another search could help	Thresholds, maximum retries, escalation rules, and validator failures
Synthesis and citation	Build the final context, generate the answer, attach citations, and express uncertainty	Context limits, citation-source matching, output schema, and unsupported-claim checks
Trace and operations	Record transitions, tool calls, retrieval results, model versions, timing, tokens, and final disposition	Durable correlation IDs, privacy rules, access to traces, alerts, and release gates

The coordinator does not have to be one giant prompt. A graph can use a small classifier for source routing, deterministic code for fan-out, a retrieval model for reranking, and a larger model only for planning or synthesis. This division usually makes the system easier to test because each decision has a narrower contract.

Retrieval tools should expose meaningful boundaries. search_everything(query) makes source selection invisible and often breaks authorization. Separate contracts such as search_product_manuals(query, product_ids) and search_incidents(service, start_time, end_time) give the runtime a place to validate scope and give evaluators a way to tell whether the selected tool made sense.

State deserves equal attention. A conversational message list is rarely enough. The system needs structured fields for the plan, attempted queries, retrieved source IDs, claim-to-source links, denied actions, remaining tool calls, and the reason the loop stopped. Keep raw evidence in state and format prompts at the edge. Repeatedly summarizing observations into free text can erase provenance and smuggle an earlier model error into later steps as if it were a fact.

A request through the retrieval loop

The exact graph varies, but a bounded agent RAG request should make the following transitions visible.

1. The gateway authenticates the caller and creates a request context containing tenant, user, permitted sources, risk tier, deadline, and budgets.
2. The coordinator normalizes the question without discarding the original wording. It can answer a greeting or reject an out-of-scope request without retrieval.
3. A planner identifies the information need. For a complex question it creates explicit subquestions and dependencies rather than a hidden paragraph of reasoning.
4. The coordinator selects one retrieval tool from the caller's allowlist and proposes typed arguments. Trusted code validates those arguments before execution.
5. The retrieval adapter applies authorization filters, runs search, reranks when configured, and returns compact records with stable source IDs, timestamps, scores, and excerpts.
6. The evidence evaluator compares the observation with the current subquestion. It can accept the evidence, reject it, record a contradiction, or request a revised query.
7. The runtime checks the retry count, token and search budgets, wall-clock deadline, and graph policy. Only then can control return to planning or retrieval.
8. Synthesis receives an evidence pack, not an unlabelled transcript. It generates the answer and maps factual claims to source records.
9. Output validators check schema, citations, denied data classes, and unsupported claims. The system either returns the result, produces a bounded "insufficient evidence" response, or escalates.

The current LangGraph retrieval-agent tutorial demonstrates a compact version of this cycle. The graph can answer directly or call a retriever, grade the returned documents, rewrite a weak query, and route back through retrieval. It also treats retrieved context as data rather than instructions when constructing the answer prompt. That last boundary matters because retrieved text may contain hostile or irrelevant directives.

Loops should be explicit in code even when the model proposes the transition. A prompt that says "keep searching until you are confident" does not define a useful stop condition. Confidence is poorly calibrated, and the instruction says nothing about cost, deadlines, repeated queries, or a source outage. The runtime should stop on measurable conditions such as a maximum number of calls, no new source IDs, sufficient claim coverage, repeated query similarity, or a deadline.

Choose the control boundaries before the model

Decide where probabilistic choice is allowed before choosing a framework.

For many systems, a deterministic outer workflow with model-assisted nodes is the cleanest arrangement. Code owns the legal transitions and hard limits. Models classify the request, propose subquestions, select from a small tool set, or grade evidence. This makes every model output a proposal that another layer can validate.

A free-form agent loop is more flexible, but its state tends to become a conversation transcript and its failure modes become harder to enumerate. Use it only when the task space truly resists a finite graph. Even then, put a deterministic supervisor around tool execution, budgets, permissions, and termination.

One coordinator is usually enough for retrieval. Splitting source domains into separate agents can improve ownership when each domain needs different prompts, credentials, and evaluation sets. It also adds handoffs, duplicated context, more model calls, and another place to lose evidence lineage. Do not create a research agent, critic agent, citation agent, and manager merely because the framework supports them. Start with separate tools and nodes. Split an agent only when its policy or state genuinely needs an independent boundary.

Retrieval tools read data. Business tools may send, publish, purchase, update, or delete. Combining both classes in one unconstrained loop turns a weak search result into a possible external side effect. If the product must act after answering, route the evidence-backed proposal through a separate authorization and approval state. Systems across the wider AI agent ecosystem use different orchestration labels. Review them by asking who may decide, who may execute, and what evidence survives each handoff.

Treat evidence and trajectories as first-class outputs

An answer-only evaluation cannot tell whether the system chose the right tool, found the right document by accident, ignored a contradictory source, or burned six unnecessary calls before producing acceptable prose. Agent RAG needs evaluation at three levels.

Retrieval evaluation measures whether each tool call found useful and authorized material. Useful signals include source-selection accuracy, recall on expected evidence, precision of returned chunks, ranking quality, filter correctness, freshness, and duplicate rate. Test query rewriting separately. A better-sounding query is not better if it drops an account number, date constraint, or legal jurisdiction.

Trajectory evaluation measures the path. Record expected tools and permissible alternatives for representative tasks. Then score tool selection, argument validity, unnecessary calls, repeated queries, recovery after an empty result, stop reason, latency, and total model and search cost. The trace should make it possible to replay a failed request against a new model or retriever without granting live write access.

Answer evaluation measures claim correctness, completeness, citation entailment, citation coverage, and calibrated abstention. The RAGChecker paper argues for separating retriever and generator diagnosis at claim level. Its experiments also found tension between context use and noise sensitivity, which is a useful warning against solving weak retrieval by stuffing more chunks into the final prompt.

Evaluation by another language model can help triage large runs, but it is still a model measurement. Keep a human-reviewed set for high-impact domains, log evaluator versions and prompts, and inspect disagreements. NIST's Generative AI Profile recommends documenting data origin and content lineage, testing data and content flows, and avoiding broad capability claims based on narrow or anecdotal assessments. Those recommendations fit agent RAG particularly well because errors can enter at every transition.

Evidence records should remain usable outside the prompt. Store a stable source identifier, location within the source, retrieval time, applicable access scope, and the excerpt or structured fields seen by the model. If the source changes, the trace should still explain the earlier answer. If policy forbids storing the excerpt, store a content hash and a protected reference with an explicit retention rule.

Engineering tradeoffs in the architecture

Agentic retrieval does not make ordinary RAG concerns disappear. Chunking, metadata, hybrid search, reranking, source freshness, and citation quality still determine the evidence available to the loop. The agent adds adaptive control on top of that foundation.

Design choice	Possible benefit	Cost or failure introduced
Retrieve only when the agent decides it is needed	Avoids irrelevant searches and can answer simple requests quickly	The agent may skip retrieval for a factual question and answer from model memory
Route among specialized sources	Preserves domain-specific search and can reduce noisy context	Tool descriptions can be ambiguous, and a wrong route may look like an empty knowledge base
Decompose one question into several searches	Supports comparisons and dependent facts that one query cannot express	Fan-out increases latency, cost, state size, and reconciliation work
Grade evidence and retry	Recovers from poor queries or irrelevant results	The grader can reject useful evidence, approve weak evidence, or create a loop
Retain state across turns	Avoids repeating work and supports follow-up questions	Stale, cross-user, or poisoned state can contaminate later retrieval decisions
Allow retrieval and business actions in one workflow	Supports end-to-end task completion	Every action tool expands the attack surface and the consequence of a retrieval error

Latency and cost grow with the trajectory, not simply with the final answer length. Each reasoning step can add a model request. Each retrieval can add embedding, search, reranking, network, and serialization work. Parallel searches shorten the critical path but may retrieve unnecessary data and make cancellation harder. Cache deterministic retrieval results where authorization and freshness rules allow it, but do not cache a user-scoped result under a query-only key.

Reliability also changes shape. A fixed RAG pipeline may return weak evidence consistently. An agentic loop can recover from that miss, yet the number of possible trajectories is much larger. Temperature zero does not make a hosted model, distributed search service, or changing index deterministic. Architecture tests need path assertions and budgets in addition to golden answers.

The security boundary expands with every source and tool. Microsoft's multitenant RAG guidance places the user's authorization context in the call to the orchestrator and applies filtering before grounding data reaches the model. That is the right direction for agentic retrieval too. The model can request a source, but trusted services must decide which records the caller may receive. This follows the same scoped-authorization principle described in Trace Brief's guide to AI agent identity and attestation.

Contain failures at every back edge

Most production failures will look ordinary: a vague tool description, an empty index, a stale ACL cache, a query rewriter that drops a constraint, or an evaluator that accepts a polished but unsupported answer. Design the graph so these faults stop locally.

• Put a hard maximum on model turns, retrieval calls, documents, context tokens, wall-clock time, and spend per request.
• Reject tool arguments that do not match a typed schema. Resolve tenant and user filters from trusted request context, never from model-generated text.
• Deduplicate queries and source IDs. Stop or change strategy when another loop would repeat the same work.
• Label every tool result as untrusted data. Strip active content, keep retrieved instructions out of the control prompt, and restrict outbound URLs where tools can fetch arbitrary pages.
• Preserve denied and failed tool calls in the trace without exposing secrets to the model or user.
• Require evidence coverage before synthesis for claims that the task marks as mandatory. Return a partial answer when one subquestion remains unresolved.
• Provide a fixed-pipeline fallback for common questions and an explicit human escalation path for high-impact or ambiguous cases.
• Keep write tools outside the retrieval loop unless a separate policy decision and user confirmation authorize the exact action.

Fallbacks should be architectural states, not improvised apology text. A request can finish as answered, partially answered, insufficient evidence, policy denied, budget exhausted, source unavailable, or awaiting human review. These outcomes make operations measurable and prevent the model from disguising an infrastructure or authorization failure as a knowledge gap.

Observability data is sensitive. Traces can contain questions, retrieved passages, tool arguments, model outputs, identity attributes, and denied resource names. Redact credentials before logging, restrict trace access, apply retention periods, and keep enough source lineage to investigate a bad answer. An observability platform is not automatically an approved copy of every knowledge source.

A production reference design

A defensible first version is deliberately small. Use one coordinator, a graph with named states, two or three read-only retrieval tools, an authorization-aware adapter for each source, structured evidence state, one relevance and coverage evaluator, a synthesis node, validators, and a hard stop controller. Put the entire request behind a gateway that supplies identity and budgets. Emit a trace event at every transition.

Start the graph with a deterministic route for simple, frequent questions. Send only questions that need dynamic source choice, decomposition, or iterative recovery into the agent loop. This keeps the expensive path visible and creates a fixed baseline for latency, cost, and answer quality.

Build the test set before adding more autonomy. Include direct questions, multi-source questions, contradictory evidence, stale documents, empty results, unauthorized documents, malicious text inside a retrieved page, repeated query rewrites, source timeouts, and questions with no supportable answer. Define the expected terminal state and allowed tool path for each case.

Then compare the agent path with the fixed baseline. Measure whether it finds evidence the baseline misses, how often it selects the expected source, how many retries lead to new evidence, and what each successful answer costs. A loop that improves a demo but cannot beat the baseline on a representative evaluation set is not ready to become the default architecture.

Framework choice comes last. LangGraph, a cloud agent service, or a small custom state machine can all express the pattern. Frameworks can be replaced. Scoped retrieval, typed tools, durable evidence, observable transitions, and bounded termination are much harder to retrofit after the loop is already serving users. An applied platform example appears in Trace Brief's guide to building an AI agent with Snowflake, while the publication's editorial policy explains how primary evidence and uncertainty are handled in this article.

Sources and methodology

This article draws on the primary documentation and research listed below. An editor reviewed the technical claims and wording before publication.

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — original RAG formulation combining parametric generation with explicit non-parametric memory
2. ReAct: Synergizing Reasoning and Acting in Language Models — interleaved reasoning, actions, observations, and plan updates over external knowledge sources
3. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — adaptive retrieval and explicit critique of retrieved passages and generated claims
4. Build a custom RAG agent with LangGraph — current retrieval-agent graph with tool routing, document grading, query rewriting, and conditional edges
5. Develop an agentic RAG solution — current architecture guidance for retrieval tools, reasoning-loop controls, evaluation, and operational tradeoffs
6. Design a secure multitenant RAG inferencing solution — identity propagation, tenant filtering, and authorization boundaries around grounding data
7. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation — claim-level diagnosis of retrieval and generation failures across modular RAG systems
8. NIST AI 600-1: Generative Artificial Intelligence Profile — risk documentation, data-flow evaluation, content lineage, and deployment measurement guidance