AI Agent Architecture: Layers, Patterns and Trade-offs

Why architecture matters more than the model

Most AI agent demos look alike: a prompt, a model, a couple of function calls, a chat window. The difference between a demo and a system you can put in front of customers is almost never the model. It is everything around it: how requests arrive, how the agent decides what to do next, which systems it may touch, what it remembers, how it is stopped from doing something harmful, how a human takes over, and how you find out that quality slipped last Tuesday.

Models also change every few months. An architecture that hard-wires one provider's SDK into every tool and prompt becomes expensive to upgrade. A good agent architecture treats the LLM as a replaceable reasoning component and puts durable engineering effort into the parts that carry your business logic, data and risk. That is the lens this guide uses, and it is how our AI agent development team scopes new builds.

Reference architecture diagram

The diagram below shows a single-agent deployment that serves several channels. Multi-agent variants replace the single orchestrator box with a router and specialist agents, but the surrounding layers stay the same.

The nine layers, one by one

1. Channels and channel adapters

Customers reach an agent through a web widget, WhatsApp Business, a phone line, email, Slack or an internal tool. Each channel has its own constraints: WhatsApp has template rules and message windows, voice needs sub-second turn-taking and barge-in handling, email is slow but long-form. A channel adapter converts every inbound event into one internal message format (user identity, session ID, text or transcript, attachments, channel metadata) and converts the agent's reply back into what the channel supports, such as buttons, list messages, SSML or plain text.

Keep identity resolution here. If the same customer writes on WhatsApp and later on the website, the adapter layer is where you decide whether those are the same person and whether you are allowed to join the histories.

2. Orchestrator and the agent loop

The orchestrator is the program that runs the loop: build context, ask the model what to do, validate the proposed action, execute it, feed the result back, repeat until the model produces a final answer or a limit is hit. In production we treat it as a small state machine rather than an open-ended loop. It enforces a step budget (for example, a maximum number of tool calls per turn), a wall-clock timeout, and explicit states such as collecting details, awaiting approval and handed off.

A useful rule: anything that must always happen the same way (checking a customer is verified before showing an order, writing an audit record) belongs in orchestrator code, not in the prompt. The model decides what to do; code decides whether it is allowed and how it is recorded.

3. The LLM layer

Put every model call behind an internal gateway. The gateway handles provider credentials, retries with backoff, timeouts, fallbacks to a second provider, token accounting and response caching where appropriate. It also enables model routing: a small, fast model classifies intent or extracts fields, while a larger model handles multi-step reasoning. Provider prices and model line-ups change often, so treat per-token cost as a configuration input, not an architectural constant. Our LLM and RAG cost calculator lets you plug in current prices and volumes.

4. Retrieval (RAG)

Retrieval grounds the agent in your documents, policies, catalogue and knowledge base. A production pipeline covers ingestion, chunking, embeddings, a vector index (often combined with keyword search), metadata filters, reranking and citation of sources. The retrieval layer should apply access filters before results reach the model, so a customer never gets content from another tenant or an internal-only document. We cover the full pipeline in production RAG architecture.

5. Tools and integrations

Tools are how an agent acts: look up an order, check room availability, create a CRM lead, issue a refund. Design them like a public API for a junior employee: narrow, well-named, strongly typed, with clear error messages. Prefer get_order_status(order_id) over a generic run_sql(query). Every tool should declare its risk level, required permissions and whether it is idempotent. Integrations with CRMs, ERPs and payment gateways are frequently the longest part of a build; see AI integrations for how we approach them.

6. Memory

Memory ranges from the recent turns of a conversation to a rolling summary, a structured customer profile and searchable history of past interactions. Each type has different storage, cost and privacy implications, and mixing them up is a common source of both bugs and compliance risk. Our dedicated guide to AI agent memory covers the options in detail.

7. Guardrails and policy

Guardrails sit on both sides of the model. Input checks detect prompt injection attempts, strip or mask personal data that the model does not need, and block out-of-scope requests. Output checks validate structured responses against schemas, confirm that tool arguments are within allowed ranges, and screen replies for leaked secrets or disallowed content. A policy layer decides which tools a given user, tenant or channel may call. For the full threat model, read AI agent security.

8. Human handoff and approvals

Two different mechanisms share this layer. Handoff moves a conversation to a live person when the agent is stuck, the customer asks, or sentiment turns negative; the human needs the full transcript and a summary. Approval gates pause a specific risky action, such as a refund above a threshold, until a person approves it, then resume the agent. Approvals need durable state, because the approver may respond hours later. We walk through an implementation in building an agent with tool calling and human approval.

9. Observability and evaluations

Log every step as a trace: the context sent, the model's proposed action, the tool call, its result, latency and tokens. Without this, you cannot debug a bad answer or explain a cost spike. On top of tracing, maintain an evaluation set of realistic conversations with expected outcomes and run it on every prompt, model or tool change. The practical side of that is covered in testing and evaluating AI agents.

Single agent vs router vs multi-agent

There are three common topologies. They are not maturity levels; many excellent production systems are single agents.

Single agent with tools

One orchestrator, one system prompt, a set of tools. Simplest to build, test and debug. It works well while the tool count stays manageable (roughly a dozen to a couple of dozen well-described tools is a typical comfort zone) and the instructions for all tasks fit together without contradicting each other.

Router plus specialist agents

A lightweight classifier or small model routes each request to a specialist, such as billing, bookings or technical support. Each specialist has its own prompt, tool set, permissions and evaluation set, and can use a different model. This pattern scales to more domains, lets separate teams own separate agents, and limits blast radius: the bookings agent simply has no refund tool. The cost is a routing step that can misclassify, and handoffs between specialists that need shared context.

Multi-agent collaboration

Several agents work on one task: a planner breaks work into steps, workers execute, a reviewer checks results. This suits long-running, research-like or document-heavy workflows, such as preparing a tender response or reconciling records across systems. It multiplies model calls, latency and failure paths, and makes evaluation harder, so reserve it for work that genuinely benefits from decomposition.

Indicative global build ranges follow the same shape: task-specific agents are typically $15k–$45k, reasoning agents with several integrations $50k–$150k, and multi-agent systems $150k–$400k+. If you are still deciding whether you need an agent at all, compare the options in AI agent vs copilot vs RAG chatbot.

Synchronous vs asynchronous, event-driven agents

Synchronous agents answer while the user waits: chat, voice, a support widget. The design constraint is latency. Stream partial responses, run independent tool calls in parallel, cache retrieval for common questions, and use a small model for classification steps. For voice, the whole turn usually needs to feel close to conversational pace, which often rules out long reasoning chains mid-call.

Asynchronous agents are triggered by events: a new email in a shared inbox, an order stuck in a status, a nightly batch of invoices, a webhook from a CRM. They run on a queue, can take minutes, and report results to a system or a person. Here the constraints are durability and idempotency. Use a job queue or workflow engine with retries, make every write tool safe to repeat, store checkpoints so a crashed run can resume, and apply a dead-letter queue for jobs that fail repeatedly.

Many real systems mix both. A customer chat can start a synchronous conversation that enqueues an asynchronous job, such as generating a quote that needs approval, and later notifies the customer on WhatsApp.

Where the data lives

Buyers often assume that an AI agent needs a copy of their business data. It usually should not. A clean separation looks like this:

• Systems of record stay put. Orders, bookings, invoices and customer records remain in your CRM, ERP, PMS or database. The agent reads and writes them through scoped tools, using service credentials limited to the operations it needs.
• The agent platform owns operational data. Conversation state, memory, approval requests, traces and audit logs live in databases you control, in the region your compliance requirements dictate, with defined retention periods.
• The search index is derived data. The vector and keyword index is built from source documents and can be rebuilt. Each chunk carries metadata (tenant, document ID, access level, version) so you can filter and delete precisely.
• The model provider sees the minimum. Send only the context needed for the current step, mask identifiers the model does not need, and check the provider's data retention and training terms. If those terms do not fit, a self-hosted open-weight model is an option.

For multi-tenant products, decide early whether tenants share an index with mandatory filters or get separate indexes. Shared is cheaper; separate is simpler to reason about for regulated clients.

Hosting options

The orchestrator itself is ordinary backend software. Containers on a managed service, serverless functions for bursty async work, or a small Kubernetes deployment are all reasonable. Choose what your operations team already runs well.

Decision table

Use this as a starting point in architecture discussions. It reflects how our solution architects usually frame the first design review.

Common failure modes

These are the problems we see most often when reviewing agent systems, whether built in-house or inherited:

• Unbounded loops. The model keeps calling a tool that returns an error. Fix with step budgets, error messages the model can act on, and a fallback to handoff.
• God tools. A single tool that runs arbitrary queries or API calls. It makes injection attacks and accidental damage far more likely. Split it into narrow operations.
• Business rules in prompts only. "Never refund more than the order value" written in a prompt is a suggestion. Enforce it in the tool.
• Context bloat. Every turn resends the full history, all retrieved chunks and every tool description. Latency and cost climb while accuracy falls. Summarise, retrieve selectively and load tools per task.
• Retrieval that ignores permissions. The index is searched first and filtered later, or not at all. Apply tenant and access filters inside the query.
• No idempotency. A retry creates two bookings or sends two payment links. Use idempotency keys on every write.
• Silent quality drift. A provider model update or a knowledge base change degrades answers and nobody notices for weeks. Run evaluation sets on a schedule, not only at release.
• Handoff without context. The human agent receives "customer needs help" and asks the customer to repeat everything. Pass a summary, the transcript and the actions already taken.
• Cost surprises. No per-tenant or per-conversation token limits. Add budgets and alerts in the LLM gateway.

Architecture choices also shape payback. If you are building the business case alongside the design, our guide to AI agent ROI shows how to connect volumes, handling time and running costs. More reading is collected on the AI agent resources hub.

Frequently asked questions

What are the core components of an AI agent architecture?

A production agent usually has nine parts: channel adapters, an orchestrator that runs the agent loop, one or more LLMs, a retrieval (RAG) layer, tools and integrations, a memory store, guardrails and policy, a human handoff path, and observability with evaluations. Small agents collapse several of these into one service, but each concern still needs an owner.

Should we start with a single agent or a multi-agent system?

Start with a single agent and a well-designed tool set. Move to a router with specialist agents when one prompt can no longer hold the instructions for every task, or when different tasks need different models, permissions or owners. Full multi-agent collaboration is worth its extra cost and debugging effort only for genuinely long, multi-step work.

Which LLM should the architecture be built around?

None in particular. Put the model behind an internal interface so you can route simple steps to a smaller, cheaper model and hard reasoning to a larger one, and so you can switch providers or move to a self-hosted open-weight model later without rewriting the orchestrator, tools or evaluations.

Where should customer data live in an AI agent system?

Systems of record stay where they are, such as your CRM, ERP or database, and the agent reads them through scoped tools. The agent platform owns only what it needs: conversation state, memory, the vector index and audit logs, all in your own cloud region with retention rules. The LLM provider should receive the minimum context required for each call.

How much does it cost to build an agent with this architecture?

As indicative global ranges, a task-specific agent is typically $15k to $45k, a reasoning agent with several integrations $50k to $150k, and a multi-agent system $150k to $400k or more. A production RAG pipeline typically adds $20k to $45k. Running costs depend on traffic, model choice and how much context each call carries.