AI agent architecture: models, harnesses, tools, memory and controls

AI agent architecture is the complete system around a model. Instructions define the job, a harness manages the execution loop, tools provide external capabilities, memory carries useful state and controls bound cost and authority. A stronger model can improve the system, but it does not replace the other layers.

AI agent architecture is a system, not a model

An agent receives a goal, decides what to do next, uses a tool when external evidence or action is required, observes the result and continues until it reaches a stopping condition. The architecture determines what the agent can see, what it can do and how every step is constrained and reviewed.

OpenAI’s practical guide to building agentsdescribes the foundation as a model, tools and instructions. Anthropic’s guide to building effective agentsseparates fixed workflows from agents that direct their own tool use. Both support the same practical rule: use the simplest design that can complete the job reliably.

The five layers of AI agent architecture

1. Instructions: the outcome, process, evidence standard, authority and stopping rules.
2. Model: the reasoning engine that interprets the task and selects the next step.
3. Harness: the runtime that assembles context, invokes the model, calls tools and records state.
4. Tools and memory: bounded external operations plus the state needed across steps or sessions.
5. Controls: permissions, budgets, approvals, evaluation, observability and revocation.

These are logical layers, not a required vendor stack. A small agent can implement them in one process. A larger system can separate them into services as scale, security or team ownership demands.

The harness owns the execution loop

The harness turns model output into controlled execution. It decides which instructions and history enter the context, validates a requested tool call, handles timeouts and errors, returns the observation to the model and stops when the task is complete or a boundary is met.

Keep this loop inspectable. A human operator should be able to reconstruct the task, selected tools, material decisions, cost and result without relying on the model’s final summary. That makes the system easier to test and diagnose when a plausible answer hides a bad route.

Tools and memory need boundaries

Give each tool a clear purpose, structured input and observable result. Load a detailed contract only when it can change the agent’s decision. Scrollport uses a small stable control surfaceso an agent can discover, inspect and run a growing catalog without placing every definition into every conversation.

Treat memory as selected state, not an unlimited transcript. Save facts that will improve a later decision, attach their source and lifetime, and provide a way to correct or remove them. Never use memory to bypass the current permission, price or tool contract.

Start simple and add complexity from evidence

Begin with one agent, one bounded outcome and the smallest useful tool set. Add routing, specialised subagents or long-term memory only when evaluations show a repeated failure the new component can address. Multi-agent designs introduce more handoffs, context boundaries and failure states, so complexity must earn its place.

AI agent architecture checklist

• Can one person state the workflow outcome and owner?
• Does every tool have a narrow purpose, validated contract and failure state?
• Can the harness stop on cost, time, uncertainty or required human approval?
• Is saved memory sourced, scoped and removable?
• Can operators trace a result back to the decisions and evidence that produced it?
• Does an evaluation show that each added component improves the real task?

Next, use the AI agent evaluation guideto turn these architecture choices into testable acceptance criteria.