Building RAG agents with LLMs: a production-first guide

Building RAG agents with LLMs is usually presented as a framework tutorial: connect a vector store, decorate a search function as a tool, add a model, and watch it decide when to retrieve. That proves the loop can run. It says little about whether the loop should exist, whether it searches the right material, or whether its final answer is supported.

A dependable build starts one step earlier. First make a fixed retrieval-augmented generation pipeline work on the actual questions and documents. Then add only the decisions that the fixed path cannot make reliably. Each new branch needs state, a test, a budget, and a fallback.

This article focuses on that build sequence. The separate agent RAG architecture guide covers components and trust boundaries in depth, while RAG versus agentic RAG helps decide whether the added control loop is worth its cost. Here the goal is narrower: turn a useful retrieval baseline into a bounded RAG agent that a team can test and release.

Prove that another retrieval decision can help

Start with a set of questions that a fixed pipeline handles poorly for a reason an agent can change. Good candidates have a conditional information path. The system may need to choose among several sources, split a request into dependent subquestions, inspect the first result before forming the second query, or decline retrieval when the answer belongs in the conversation state.

Microsoft's current agentic RAG guidance makes the same distinction. A single query against one index is usually better served by standard RAG. Agentic control becomes useful when source selection, query decomposition, or iterative refinement must happen at request time.

Do not use an agent to conceal a weak index. If the baseline misses a policy because the document was never ingested, no planner can retrieve it. If access metadata is absent, model routing cannot reconstruct it safely. If chunks remove table headings or effective dates, a second search may produce more incomplete fragments rather than a better answer.

Write down the specific decision that is missing. "The agent can reason" is not a testable requirement. These are:

• choose the product catalog before searching maintenance records;
• run a second query only when the first result lacks an effective date;
• route code questions to API documentation and account questions to an authorized customer service;
• stop without retrieval for greetings and for requests outside the product's scope.

Keep a fixed-path control group for those cases. The agent has earned its place only if the additional decision improves an outcome that matters, such as supported-answer correctness or coverage, without unacceptable latency, cost, or security failures. A visually interesting trace is not evidence of improvement.

Write the acceptance contract before choosing a framework

An implementation goes off course quickly when "better answers" is the only requirement. Define the request boundary, allowed evidence, output contract, and failure behavior before writing the graph.

For one initial use case, record:

Contract area	Decision to make before implementation
Users and scope	Which users, tenants, languages, products, and question types are eligible?
Evidence	Which repositories are authoritative, how are versions identified, and what metadata enforces access?
Retrieval	Which source can answer each question class, and what must every result return?
Answer	What fields, citations, uncertainty statements, and no-answer behavior must the response contain?
Loop	Which transitions are allowed, and what call, token, time, and duplicate-result limits end the run?
Safety	Which data classes must not reach a model, which retrieved content is untrusted, and which actions are forbidden?
Release	Which evaluation thresholds, trace fields, fallback rate, and incident controls are required?

The contract should include examples with expected dispositions. Some questions should produce an answer with cited evidence. Others should return "insufficient approved evidence," request clarification, or stay on the fixed path. Include denied and cross-tenant cases from the beginning. A graph trained exclusively on successful retrieval teaches the team almost nothing about containment.

Separate product policy from model instructions. "Never search another tenant" cannot live only in a prompt. The retrieval adapter must receive trusted identity and scope from the application, apply filters at the data boundary, and refuse arguments outside that scope. The prompt may explain the tools available to the model, but it does not grant access.

Choose a trace schema at the same time. At minimum, one run should preserve the workflow version, model and prompt versions, user scope, proposed tool call, validated arguments, result identifiers, evidence grade, transition reason, budgets consumed, final disposition, and cited source IDs. The AI agent observability guide describes how those events fit into a reviewable run.

Build and measure a fixed RAG baseline

The baseline is the shortest complete path from a question to a grounded answer:

question -> authorized retrieval -> reranking -> evidence pack -> answer with citations

It needs the same production corpus, access filters, and output validator planned for the agent. A toy index followed by an agentic rebuild does not provide a fair comparison. The baseline should be good enough that failures can be attributed to a missing decision rather than unfinished retrieval work.

Begin with document inventory. For each source, capture an owner, stable source ID, access class, version or effective date, update process, and deletion path. Preserve structural context during extraction. A paragraph from a procedure may depend on its heading; a table cell may be meaningless without its row and column labels. Test parsing on awkward files as well as clean HTML.

Create representative questions before tuning chunk sizes or search parameters. Each question needs one or more acceptable source records and, where feasible, the claims those sources can support. Add cases with no supporting source. This set will change as the product changes, but it gives the first retrieval experiment something sturdier than a few hand-picked prompts.

Measure retrieval without generation. Inspect whether the authorized source appears, whether irrelevant records crowd it out, and whether returned chunks contain the facts needed to answer. Use query-level diagnostics and review misses by cause: absent document, bad parsing, unsuitable chunk, missing filter, weak query, ranking error, or ambiguous ground truth.

Then evaluate generation from a frozen evidence pack. This exposes answer problems without search variance. Check whether the response answers every requested part, keeps claims within the supplied evidence, maps citations to the right source records, and refuses when support is missing. The RAGChecker paper argues for this modular diagnosis because a single end-to-end score cannot tell a retrieval failure from a generation failure.

Record baseline latency and resource use under realistic document sizes and concurrency. The exact numbers are local facts. They become the reference for every agentic branch added later.

Expose narrow retrieval functions as typed tools

Once retrieval works, wrap it in tool contracts that describe meaningful source boundaries. Avoid one search_all(query) function when the system has different repositories, permissions, freshness rules, or result schemas. A broad tool hides routing errors and makes least-privilege access harder to enforce.

A useful retrieval contract might accept a query, product IDs, locale, and a maximum result count. Trusted application context supplies tenant and user scope. The result should return structured records rather than a prose summary:

SearchResult
  source_id
  source_type
  title
  uri
  version
  effective_at
  access_scope
  excerpt
  retrieval_score

The model may propose a query and product ID. Code validates types, length, allowed values, and scope before the search runs. The adapter adds security filters, enforces timeouts and result caps, and returns stable identifiers. It should not return credentials, raw authorization tokens, or internal connection details.

Tool descriptions deserve evaluation. Give each tool a distinct name, a short statement of what it can answer, its exclusions, and the meaning of its parameters. Build a routing set with questions that look similar but belong to different sources. Measure the selected tool and arguments before judging the final answer.

The current LangGraph agentic RAG tutorial demonstrates the essential pattern: preprocess documents, create a retriever tool, let a model choose whether to call it, grade the returned documents, optionally rewrite the question, and generate an answer. That sequence is a useful reference implementation. A production version still needs identity, structured evidence, hard budgets, and negative tests around each node.

Keep retrieval tools read-only. If the larger product can also create tickets, send messages, or update records, put those functions behind a separate policy and approval boundary. The model's ability to find evidence should not silently grant authority to act on it.

Implement the smallest bounded control graph

Start with one coordinator and a finite set of transitions. Multiple agents are unnecessary for most first versions. They add handoffs and duplicate context before the team has proved that one retrieval loop works.

A compact graph can support six dispositions:

admit -> route -> retrieve -> grade -> synthesize -> validate -> return
             |         |         |
             |         |         +-> rewrite -> retrieve
             |         +-> insufficient evidence
             +-> direct response or out of scope

Use structured state rather than relying on a message transcript. Useful fields include the original question, normalized request, trusted user scope, selected tool, attempted queries, retrieved source IDs, unresolved claims, accepted evidence, retry count, remaining budgets, and stop reason. Prompts can be rendered from this state at the edge. The state itself should remain inspectable.

The ReAct paper established an influential pattern in which a model interleaves reasoning, actions, and observations. That pattern does not require an unconstrained loop. The runtime can expose a small action set and validate each requested transition.

Use deterministic code wherever the condition is already known. Budget exhaustion, an unauthorized tool, a duplicate query, an empty result, schema failure, and deadline expiry do not need model judgment. Model-assisted nodes are useful when the decision depends on language, such as classifying an information need or judging whether a passage covers a subquestion. Their outputs should still follow a schema and have a failure route.

Query rewriting needs special restraint. Keep the original question and compare each rewrite with it. A rewrite should repair retrieval language, not quietly change the user's intent. Limit retries, stop when no new source IDs appear, and record why another search was allowed. Self-RAG provides primary evidence that adaptive retrieval and explicit critique can outperform indiscriminate fixed retrieval in its evaluated tasks. It does not prove that any generic self-critique prompt will be reliable in a different product.

Define the terminal states before the happy path is complete. A run may finish with a supported answer, direct non-retrieval response, clarification request, insufficient evidence, policy denial, timeout, tool failure, validator rejection, or human escalation. A system that reports all of these as success will be impossible to operate honestly.

Preserve evidence through synthesis and citation

The agent should assemble an evidence pack, not paste its entire working transcript into the final prompt. Tool chatter, rejected documents, query rewrites, and model comments can consume context and blur which material was accepted.

For every accepted passage, preserve the source ID, exact excerpt or content digest, version, access scope, retrieval event, and the subquestion it supports. Synthesis receives only eligible evidence plus the response contract. If the answer contains factual claims, map them back to source records before returning it.

Citations need validation beyond URL presence. Check that each citation resolves to a retrieved and authorized source, that the cited passage supports the associated claim, and that the version displayed to the user matches the version used during generation. When a claim has no support, remove it, retrieve under the remaining budget, or return a limitation. Do not attach the nearest source to make the answer look grounded.

Retrieved text is untrusted input. A web page, uploaded PDF, support note, or indexed repository entry can contain instructions addressed to the model. OWASP's prompt injection guidance explicitly includes malicious content placed in a RAG repository. Tell the synthesis model to treat retrieved text as data, but do not rely on that instruction alone. Restrict tools and output channels, sanitize or quarantine risky document types, prevent retrieved content from changing system policy, and test documents with embedded attack strings.

Provenance also helps ordinary debugging. NIST's Generative AI Profile calls for source documentation, performance thresholds, ongoing monitoring, and mechanisms to stop deployment when risk becomes unacceptable. Stable evidence records make those controls practical. Without them, a reviewer sees a fluent answer and a link but cannot reconstruct what the model actually received.

Evaluate retrieval, trajectory, and answer separately

An agent adds a new failure layer between question and answer. Evaluation must cover the trajectory as well as the endpoints.

Use one versioned set of requests with expected evidence and allowed behaviors. Split it into development and held-out portions, then add adversarial and operational cases. Include misspellings, ambiguous requests, unsupported questions, stale records, empty indexes, slow tools, conflicting passages, access denials, prompt injection in documents, and questions that need no retrieval.

Score each layer with a repairable question:

Layer	Evaluation question	Example signals
Admission	Did the request receive the right scope and policy?	tenant and user context errors, denied-case escapes
Routing	Did the model choose an allowed and useful tool?	tool selection accuracy, invalid arguments, unnecessary retrieval
Retrieval	Did the tool return enough eligible evidence?	source recall, source precision, stale or unauthorized results
Loop	Did another step add information?	calls per run, duplicate queries, new source yield, stop-reason accuracy
Synthesis	Are claims complete and supported?	groundedness, completeness, citation support, refusal quality
Operations	Can the product meet its service envelope?	latency percentiles, cost per disposition, timeouts, fallback rate

The Microsoft end-to-end RAG evaluation guide separates groundedness and completeness, then extends evaluation to privacy, content safety, and adversarial threats. Its agentic guidance adds tool selection, retrieval efficiency, total latency, and cost. Treat those as categories rather than a universal scorecard. The product team still has to choose thresholds based on its users and potential harm.

Compare the agent against the fixed baseline on the same requests. Review improvements and regressions by question type. If the loop helps multi-source questions but makes single-source answers slower and less stable, route only the former through it. Agentic behavior does not need to be an application-wide setting.

Human review should expose the question, accepted evidence, trajectory summary, and final answer. Ask reviewers to label the failed layer and severity. Pair model-based graders with periodic expert checks, especially for release gates. A grader that shares the generator's blind spot can make a regression look consistent.

Test determinism where it matters. Repeated runs need not use the same words, but authorization, schema validity, forbidden tool calls, maximum steps, and stop behavior should remain within policy every time.

Secure the complete retrieval data path

Authorization begins before search. The application authenticates the caller and creates trusted scope. The agent sees only tools that make sense for that scope. The retrieval service applies tenant, user, region, product, and document-state filters before results reach the model.

Microsoft's multitenant RAG guidance recommends an API layer that contains access and filtering logic rather than allowing application components to query tenant stores directly. This also makes the rule easier to test: the same boundary governs fixed retrieval, agentic retrieval, and any later client.

Threat-model the ingestion path as well as the query path. Identify who can add or edit documents, how changes are reviewed, when indexes refresh, how deleted data disappears, and whether an attacker can place instructions in a source likely to rank highly. Record source versions so an incident can be tied to the corpus that produced it.

Minimize what enters prompts and traces. Search results may contain personal data, secrets, legal material, or internal identifiers. Redact only with a defined policy and tests. Protect the trace store because it can accumulate the original request, retrieved excerpts, model output, and error payloads in one place.

Run tools with least privilege and short-lived credentials held by the application. Validate destinations for any network retrieval. Apply timeouts, response-size limits, content-type checks, and safe parsing. If retrieval uses an external tool protocol, the boundaries in MCP security architecture are directly relevant: tool metadata is not policy, arguments require validation, and credentials must stay out of model-visible content.

Plan revocation. A team should be able to disable one tool, one source, one tenant, one workflow version, or the entire agentic route without taking down the fixed baseline. That control matters during an incident and during an ordinary bad index update.

Release the loop in stages and keep the baseline

Run the agent in shadow mode first. It receives eligible production-shaped requests and writes traces, but users still receive the fixed pipeline's result. Compare source choices, supported claims, latency, failures, and cost. Protect or synthesize production data as required; "shadow" does not mean exempt from privacy rules.

Next, expose the agent to an internal cohort or a small percentage of low-risk traffic. Keep a deterministic route back to the baseline for timeouts, tool outages, validator failures, and question classes where the loop has not shown an advantage. A fallback should be visible in telemetry so a healthy-looking answer rate does not hide a failing agent.

Gate expansion on written thresholds from the acceptance contract. Review metrics by question class, tenant, source, and workflow version. Averages can hide a small group of looping requests that dominates cost and latency, so inspect the distribution of tool calls and stop reasons.

Version prompts, graph logic, tool schemas, retrieval settings, indexes, and evaluation data. Run the held-out suite before promoting any of them. A model upgrade can change tool choice; a chunking change can alter citation support; a new document source can introduce both recall and prompt injection risk.

Keep a sample of production traces for review under a defined retention and access policy. Turn user feedback and incident findings into new evaluation cases. The goal is to shrink the set of unexplained failures, not to make the graph more elaborate.

The first production RAG agent should look modest: a strong fixed retriever, a few narrow tools, one explicit loop, strict stop conditions, supported citations, and a reliable fallback. That design leaves room to add decomposition, parallel retrieval, or specialized coordinators later. More importantly, it leaves evidence that each addition solved a real problem.

Sources and methodology

This article draws on the primary documentation and research listed below. An editor reviewed the technical claims and wording before publication.

1. Build a custom RAG agent with LangGraph — current official tutorial for document preprocessing, retriever tools, conditional routing, evidence grading, query rewriting, answer generation, and graph assembly
2. Develop an agentic RAG solution — current official guidance on retrieval tool contracts, loop controls, implementation choices, operational metrics, and production tradeoffs
3. Large language model end-to-end evaluation — official separation of groundedness, completeness, safety, privacy, and adversarial evaluation for RAG systems
4. Design a secure multitenant RAG inferencing solution — official identity propagation, tenant filtering, API-layer governance, and data isolation guidance
5. ReAct: Synergizing Reasoning and Acting in Language Models — primary research on interleaving model reasoning with actions and observations from external knowledge sources
6. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — primary research on retrieval on demand and explicit critique of retrieved evidence and generated claims
7. RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation — primary research separating retrieval and generation diagnostics instead of reducing RAG quality to one score
8. OWASP LLM01:2025 Prompt Injection — current threat guidance for direct and indirect prompt injection, including malicious instructions in retrieved documents
9. NIST AI 600-1: Generative Artificial Intelligence Profile — primary risk-management guidance on evaluation thresholds, provenance, monitoring, incident response, and deployment controls