7. Agentic RAG - ReAct Agent with Tool-Calling
Give your RAG system the ability to reason, act, and self-correct. Build a ReAct agent that decides when to retrieve, when to search the web, and when to compute — using LangGraph and custom tools.
Agentic RAG
ReAct Agent with Tool-Calling
Stop pre-wiring your retrieval pipeline. Let the LLM decide what to search, when to dig deeper, and when it has enough evidence - using Ollama's native tool-calling API and a three-phase ReAct loop.
🔍Why Static Pipelines Break
Every RAG pipeline built so far in this series follows the same fixed sequence: receive a question, run a vector search with a hardcoded k, pass the top chunks to the LLM, return an answer. That works well when questions are simple and self-contained.
Real-world questions rarely are. Consider: "Which of our services handles the most concurrent connections, and what does the documentation say about tuning it for high traffic?" A static pipeline runs one search, retrieves four chunks that may or may not cover both halves, and guesses. There is no mechanism to go back for more evidence.
k=4 globally assumes every question needs exactly four chunks. Simple questions waste the context budget on irrelevant text; complex multi-part questions starve for evidence from only one search.search_documents(k=N) and choose N itself based on how confident it is after each retrieval.The deeper issue: in a static pipeline the developer decides how retrieval happens at design time. In an agentic pipeline the model decides at runtime - and it has far more context about what it specifically needs to answer the question it was asked.
🔄The ReAct Pattern
ReAct (Reason + Act) is a prompting strategy that interleaves LLM reasoning with external tool use. Each cycle has three phases that repeat until the model has enough evidence:
Reason
The LLM internally reflects on what it knows so far and decides what action to take next. This happens inside the model's response before any tool call.
Act
The LLM emits a structured tool_call - a function name plus JSON arguments - instead of plain text. Ollama handles this natively.
Observe
The tool runs; its output is appended to the message history as a role: tool message. The loop continues with full context of what was found.
Answer
When no tool calls are emitted, the LLM produces a final text response. Every claim is grounded in evidence retrieved during the loop.
Modern Ollama models (llama3.1:8b, qwen2.5:7b, mistral:7b v0.3+) support tool-calling natively via the tools parameter - no prompt engineering required. Just pass the tool schemas and inspect the response for tool_calls.
🏗Architecture Overview
Three new files extend the Article 3 multi-tenant API:
project/ ├── main.py # add agent router ├── agent_tools.py # NEW - tool schemas + executor functions ├── agent.py # NEW - ReAct loop └── routers/ └── agent.py # NEW - POST /agent/query
from routers import documents, query, agent # add agent app.include_router(agent.router)
The agent is stateless per request - it builds a fresh message list each call. Persistence is ChromaDB's job, not the agent's.
🔧Tool Registry
We expose three tools. Each has a JSON Schema definition passed to Ollama, and an executor function called in Python when the model requests it.
| Tool | What it does | When the agent uses it |
|---|---|---|
search_documents |
Vector-searches ChromaDB; returns [{id, text, score}] up to k results |
First step of every question; re-called with different query angles when initial results are insufficient |
get_chunk |
Fetches full text of one chunk by ID | When a search result was truncated and the model needs the complete passage |
summarise_chunks |
Asks Ollama to condense a list of texts into one paragraph | When the model has gathered many chunks and wants to synthesise before writing its final answer |
The JSON Schema definition for search_documents - this is what Ollama receives as part of the tools parameter:
{
"type": "function",
"function": {
"name": "search_documents",
"description": "Search the knowledge base for relevant chunks.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"k": {"type": "integer", "default": 4},
},
"required": ["query"],
},
},
}
All three tool schemas are collected in TOOL_SCHEMAS - a plain list that gets passed to ollama.chat(tools=...) on every iteration. The dispatcher routes each tool_call to the right Python function and returns the result as a JSON string:
import ollama
def search_documents(collection, query: str, k: int = 4) -> list[dict]:
results = collection.query(query_texts=[query], n_results=k,
include=["documents", "metadatas", "distances"])
return [
{"chunk_id": m.get("chunk_id", i), "text": t, "distance": d}
for i, (t, m, d) in enumerate(zip(results["documents"][0], results["metadatas"][0], results["distances"][0]))
]
def get_chunk(collection, chunk_id: str) -> dict:
result = collection.get(ids=[chunk_id], include=["documents"])
if not result["documents"]:
return {"error": f"chunk not found: {chunk_id}"}
return {"chunk_id": chunk_id, "text": result["documents"][0]}
def summarise_chunks(model: str, texts: list[str], focus: str = "") -> dict:
prompt = (
f"Summarise the following passages" + (f" focusing on {focus}" if focus else "") + " in 2-3 sentences:\n\n"
+ "\n---\n".join(texts)
)
resp = ollama.chat(model=model, messages=[{"role": "user", "content": prompt}],
options={"temperature": 0, "num_predict": 256})
return {"summary": resp["message"]["content"].strip()}
def dispatch(name: str, args: dict, collection, model: str) -> str:
try:
if name == "search_documents":
result = search_documents(collection, args["query"], args.get("k", 4))
elif name == "get_chunk":
result = get_chunk(collection, args["chunk_id"])
elif name == "summarise_chunks":
result = summarise_chunks(model, args["texts"], args.get("focus", ""))
else:
result = {"error": f"Unknown tool: {name}"}
except (KeyError, ValueError) as exc:
result = {"error": str(exc)} # agent adapts gracefully
return json.dumps(result, ensure_ascii=False)
LLMs occasionally hallucinate argument names. Always wrap dispatch() in a try/except and return a {"error": "..."} dict - the model will read the error and correct itself on the next iteration rather than crashing the loop.
🔁The Agent Loop
The loop maintains a single messages list that grows with each iteration. The model always has full context of what it has tried and found - it never repeats a search it already ran:
import ollama
import agent_tools as tools
from dataclasses import dataclass, field
@dataclass
class AgentResult:
answer: str
steps: list["AgentStep"]
iterations: int = 0
def run(question: str, collection, model: str = "llama3.1:8b", max_iterations: int = 6) -> AgentResult:
steps: list[AgentStep] = []
iteration = 0
messages = [
{"role": "system", "content": _SYSTEM_PROMPT},
{"role": "user", "content": question},
]
while iteration < max_iterations:
iteration += 1
resp = ollama.chat(
model=model, messages=messages,
tools=tools.TOOL_SCHEMAS,
options={"temperature": 0, "num_predict": 1024},
)
msg = resp["message"]
tool_calls = msg.get("tool_calls") or []
messages.append({"role": "assistant", "content": msg.get("content", ""),
"tool_calls": tool_calls})
if not tool_calls: # no more tool calls → final answer
return AgentResult(answer=msg["content"].strip(), steps=steps, iterations=iteration)
for tc in tool_calls:
name = tc["function"]["name"]
args = tc["function"]["arguments"]
result = tools.dispatch(name, args, collection, model)
messages.append({"role": "tool", "content": result})
Two design choices matter here:
temperature = 0
- Tool decisions must be deterministic
- Non-zero temperature causes random variation in which tool is called
- Makes debugging and evaluation nearly impossible otherwise
max_iterations guard
- Prevents infinite loops on ambiguous questions
- When limit is hit, inject a "give your best answer now" message
- Graceful degradation - partial answer is better than an exception
Every decision is recorded in an AgentStep dataclass for full observability:
from dataclasses import dataclass, field
@dataclass
class AgentStep:
step: int
type: str # "tool_call" | "tool_result" | "answer"
tool_name: str = ""
tool_args: dict = field(default_factory=dict)
content: str = ""
🚀FastAPI Endpoints
from typing import Annotated
import dataclasses
from fastapi import APIRouter, Query
from pydantic import BaseModel
import chromadb
import agent as ag
router = APIRouter()
_chroma_client = chromadb.PersistentClient(path="./chroma_data")
def _get_collection(tenant_id: str):
return _chroma_client.get_or_create_collection(f"tenant_{tenant_id.replace('-', '_')}")
class StepOut(BaseModel):
step: int
type: str
tool_name: str = ""
tool_args: dict = {}
content: str = ""
class QueryResponse(BaseModel):
answer: str
iterations: int
steps: list[StepOut]
@router.post("/agent/query", response_model=QueryResponse)
async def agent_query(
q: Annotated[str, Query(description="Question to answer")],
tenant_id: Annotated[str, Query()] = "default",
model: Annotated[str, Query()] = "llama3.1:8b",
max_iterations: Annotated[int, Query(ge=1, le=10)] = 6,
):
collection = _get_collection(tenant_id)
result = ag.run(question=q, collection=collection,
model=model, max_iterations=max_iterations)
return QueryResponse(
answer=result.answer, iterations=result.iterations,
steps=[StepOut(**dataclasses.asdict(s)) for s in result.steps],
)
POST /agent/query
- Runs the full ReAct loop
- Returns
answer,steps(trace),iterations - Caller can render a reasoning timeline from the steps list
GET /agent/models
- Lists Ollama models available on the host
- Lets UIs populate a model selector dynamically
- Returns
[{name: "llama3.1:8b"}, ...]
📋Reasoning Trace in Action
Here is what a real multi-step trace looks like for: "What are the main scaling bottlenecks and how are they mitigated?"
{"query": "scaling bottlenecks performance limits", "k": 5}
{"query": "mitigation strategies horizontal scaling caching replicas", "k": 4}
{"texts": [...9 chunks...], "focus": "bottlenecks and mitigations"}
Three tool calls, three iterations, one grounded answer. The model searched with two different query angles then synthesised - something a hardcoded k=4 pipeline cannot do.
✅Production Checklist
When to choose agentic over static RAG:
| Scenario | Static RAG | Agentic RAG |
|---|---|---|
| Simple factual lookup | ✅ Fast, predictable | ⚠️ Overkill - 2-5 s overhead per tool call |
| Multi-part questions | ❌ Single search misses parts | ✅ Searches separately for each part |
| Ambiguous phrasing | ❌ One embedding vector | ✅ Re-queries with different phrasings |
| Comparative questions | ❌ Needs multiple doc evidence | ✅ Multiple searches build full picture |
| High-stakes accuracy | ⚠️ No self-correction | ✅ Can verify with follow-up searches |
text[:400]) and expose get_chunk for full text only when the model specifically asks for it by ID.Verified tool-calling models: llama3.1:8b, llama3.1:70b, qwen2.5:7b, mistral:7b (v0.3+). Check the model's Ollama page for the Tools tag before deploying - older versions silently ignore the tools parameter.
From Pipeline to Agent
Three files, three tools, one loop. The model goes from a passive chunk consumer to an active retrieval participant - deciding what to look for, how many results it needs, and when it has enough to answer. No external orchestration framework required.
Next up: Corrective RAG (CRAG) - building a system that scores its own retrieved context and decides whether to use it, rephrase the query, or fall back to a broader search strategy.
→ Continue to Article 8: Corrective RAG (CRAG)