AI agent observability: what to trace, measure and review
Trace AI agent decisions, tool calls, cost and outcomes so operators can find failures, protect sensitive data and improve real workflows.
By Scrollport
In brief
AI agent observability connects the final business outcome to the decisions and tool calls that produced it. Capture the workflow identity, model and tool activity, latency, cost, result state and human interventions without exposing secrets or unnecessary payloads. Use those signals to investigate failures, select evaluation cases and decide whether a workflow is ready to expand.
AI agent observability connects the business outcome to the model decisions, tool calls and human interventions that produced it. It helps operators answer three questions: what happened, why did it happen and what should change before the next run?
Observe the whole job, not only the model
A successful model response can still produce a failed workflow. The agent may select the wrong tool, use stale evidence, retry an expensive operation or stop before the acceptance criteria are satisfied. Start with a stable workflow and run identity, then join each model, tool, policy and human event to that job.
OpenAI’s Agents SDK tracing guidetreats an end-to-end workflow as a trace containing generations, tool calls and other spans. The OpenTelemetry generative AI conventionsprovide vendor-neutral attribute names for many of the same signals.
Five layers of AI agent observability
- Outcome: whether the human’s acceptance criteria were met.
- Route: the decisions, handoffs and stopping condition used.
- Tool activity: the selected operation, result state, latency and retries.
- Model activity: model identity, token use, latency and structured decisions.
- Control activity: permissions, approvals, policy blocks, spend and human overrides.
Outcome is the top layer because a fast, inexpensive trace has little value if the user still has to redo the work. Technical signals explain the outcome; they do not replace it.
Trace tool decisions, cost and outcomes
Record the capability the agent needed, the tool it chose, the inspected contract and price, the run result and whether that evidence was accepted. Keep retries and fallbacks explicit. This reveals whether a failure belongs to planning, tool selection, provider execution or the final synthesis.
For paid work, record the estimate, approval boundary, amount settled and principal that authorised it. Scrollport exposes price before execution and uses a prepaid wallet so agent spending can be bounded as part of the workflow rather than reconstructed from a monthly bill.
Protect sensitive data in traces
Observability is not permission to collect every prompt, credential or returned payload. Prefer identifiers, structured states and redacted summaries. Classify fields before ingestion, limit access, define retention and make deletion possible. Sample detailed payloads only when the diagnostic value justifies the exposure.
Turn production signals into review
Review failure clusters, expensive routes, repeated human overrides and outcome drift on a fixed cadence. Promote representative failures into the evaluation set, assign an owner and verify the proposed fix against both the failed case and normal cases. This closes the loop between observability and improvement instead of producing a dashboard nobody acts on.
Use the AI agent testing guide to define the checks that should run before a change reaches production.