Observability
Recording enough state and activity to inspect a run afterwards — the model calls, retrievals, tool executions, approvals, costs, errors and external effects that made up a job.
Updated 15 September 2026
In practice
After a customer complains, an operator reconstructs the job: which policy version applied, which sources the agent read, what it proposed, who approved it, what the CRM returned and when the analyst rejected the draft.
The distinction
Observability records what happened. Evaluation judges it, and security constrains it. A complete trace can sit alongside a wrong result, because a trace shows the path rather than the correctness of the destination.
The join is the point
Stable identifiers are what make a trace useful:
job_id
-> run_id and parent_run_id
-> model, runtime, tool and policy versions
-> identity, authority and approval decisions
-> trace, spans, logs and costs
-> external resource receipts and side effects
-> delivered output
-> acceptance, rework and incident state
Without the join, a team optimises token cost while losing acceptance, reduces latency while increasing unsafe retries, or closes an incident without identifying every affected record.
More logs create a second risk
Prompts, retrieved documents, tool arguments and outputs can contain confidential, personal or regulated data. Capturing everything improves investigation and expands storage, access and retention exposure at the same time.
Practical systems separate metadata that is always recorded, content that is sampled or redacted, secrets that are never logged, and high-impact events that require full protected evidence. Storing a hash or a reference can prove which version influenced a run without copying the content into every platform.
Portable, but not yet standard
Shared telemetry conventions reduce exporter lock-in and are still in development for agent-specific spans. They also do not supply a data model for the business job — the objective, the acceptance test and the accountable owner are the organisation's to add.
Sources & context
Semantic conventions for GenAI agent and framework spans
OpenTelemetry · Development
Defines spans for model, retrieval, memory and tool operations; the agent and framework conventions are explicitly marked Development.
Source checked 2026-09-13
Observe your agent applications
AWS · 2026
Documents OpenTelemetry-compatible telemetry with sessions, latency, duration, token use and error rates.
Source checked 2026-09-13