Evaluation, observability and security for AI agents

On this page

Chapter 10 of The Hitchhiker's Guide to the Agentic Economy

Report hub · Previous: identity, authority and policy · In this chapter: what to measure · market map · control loop · evidence card · Next: discovery, marketplaces and offers

Uninspected execution as unhedged commercial risk

In commercial operations, execution that cannot be independently audited cannot be insured, legally defended, or recognized as revenue.

An observability trace shows which model calls, database queries, and network requests executed. On its own, a trace cannot prove whether the business deliverable was factually accurate, legally authorized, safe from prompt injection, or commercially accepted by a paying client. These are five distinct operational questions that fail independently.

Managing autonomous software requires synchronizing five layers of evidence around every job: component performance, workflow trajectory, operational telemetry, formal business acceptance, and security containment.

The technical tools supporting this discipline—distributed tracing engines, automated evaluation datasets, policy enforcement gateways, and AI runtime security platforms—are maturing rapidly. The primary friction in enterprise deployment is organizational: defining explicit acceptance tests, capturing real-world database side-effects, establishing hard stop conditions, and granting designated operators the operational authority to intervene.

Key Takeaways

• Evaluate the whole job as well as the model response. Tool selection, arguments, side effects, recovery and acceptance can fail independently.
• Observability records activity. Evaluation judges it. Security controls prevent, detect and contain harmful effects. None substitutes for the others.
• The emerging control stack links traces, approvals, resource events, costs and acceptance through stable job and run identifiers.
• Prompts, model versions, tools, policies, datasets and evaluators are becoming versioned production inputs.
• A high average pass rate can hide a low-frequency severe failure. Release gates need consequence-weighted thresholds and hard stops.

Five operational layers of verification

A single synthetic benchmark score cannot capture the operational risk profile of an autonomous system. Production governance requires five distinct verification layers:

1. Component-level evaluation: Did individual models, retrieval pipelines, classifiers, or tool schemas satisfy specific technical benchmarks?
2. Trajectory and workflow evaluation: Did the agent plan, sequence, and execute an acceptable multi-step path without circular reasoning or redundant tool loops?
3. Operational observability: What specific compute resources, network endpoints, and database tables ran, how long did execution take, what did it cost, and where did exceptions occur?
4. Business acceptance: Did the responsible business owner or corporate client inspect and accept the delivered output, and how many minutes of manual human correction were required?
5. Security and audit evidence: Did the workload operate strictly within authorized spending and data boundaries, resist adversarial prompt injection, and log sufficient evidence to support post-incident analysis?

A foundation model can generate an articulate summary using illegally scraped context. A workflow can execute every planned step perfectly while delivering an output that is useless to the client. A runtime security filter can intercept malicious prompt injections while an unconstrained agent sends confidential records to an incorrect email recipient.

Maintaining operational control requires explicit technical definitions:

• Evaluation: Measuring system behavior and outputs against explicit test assertions.
• Observability: Capturing runtime state, execution spans, and system metrics necessary to reconstruct execution.
• Security: Implementing preventive, detective, and containment controls that mitigate adversarial manipulation and operational failure.
• Business acceptance: The formal human or rule-based verification confirming that a deliverable satisfies contractual specifications.

The unified correlation key

Linking these evidence layers requires maintaining an unbroken chain of correlation identifiers:

job_id
  -> run_id and parent_run_id
  -> agent, model, runtime, tool and policy versions
  -> identity, authority and approval decisions
  -> trace, spans, logs and costs
  -> external resource receipts and side effects
  -> delivered output
  -> acceptance, rework and incident state

Without this correlation chain, engineering teams optimize token consumption while increasing client rejection rates, reduce latency by cutting safety validations, or close security incident tickets without identifying which downstream databases were corrupted.

Structuring evaluation around business outcomes

Evaluating agent workflows begins with defining explicit operational test cases rather than relying on generic LLM benchmarks. An enterprise evaluation suite must model realistic business distributions: standard transactions, complex edge cases, adversarial prompt attacks, unauthorized tool attempts, and network recovery scenarios.

Different evaluation methodologies carry distinct failure modes:

Evaluation approach	Operational utility	Inherent technical limitation
Deterministic assertions	Validating JSON schemas, parameter formats, permission checks, and hard spend caps	Incapable of evaluating semantic nuance, writing style, or qualitative analysis
Ground-truth comparison	Scoring outputs against verified datasets, expected SQL queries, and tool calls	Inflexible when tasks allow multiple valid execution trajectories
Human expert review	Assessing professional judgment, legal accuracy, and high-consequence edge cases	Expensive, slow, and prone to subjective variance without strict rubrics
LLM-as-a-judge	Evaluating semantic relevance, tone, and policy adherence across high-volume samples	Subject to model bias, instruction drift, and potential prompt injection vulnerability
Invariant testing	Verifying that core business policies hold constant when irrelevant inputs vary	Complex to engineer; may fail to anticipate real-world data distribution shifts
Adversarial red-teaming	Stress-testing goal hijacking, privilege escalation, credential exfiltration, and tool abuse	Passing a static test set does not guarantee defense against novel attack vectors

OpenAI’s trace grading architecture structures evaluation by assigning programmatic labels and scores across multi-step execution traces, diagnosing where an agent succeeded or failed rather than evaluating only the final output string (OpenAI trace grading). LangSmith decouples offline testing on curated datasets from online evaluation running asynchronously against production traffic (LangSmith evaluation).

Evaluating external side-effects

Evaluating multi-step trajectories can identify redundant tool calls or missing context. However, for consequential business workflows, evaluation must verify real-world database and resource state changes directly:

• The target database record was created or modified exactly once.
• No unauthorized records or adjacent tenant rows were touched.
• Email recipients, attachments, and messaging channels matched authorized digests.
• Financial transaction amounts, currencies, and recipient accounts conformed to signed mandates.
• Software deployments passed automated smoke tests and security scans.
• Cooperative task cancellations halted all downstream execution processes.
• Third-party API receipts match trace event logs.

An autonomous agent can complete a customer update, report success to the user, and accidentally create duplicate records in the enterprise CRM. Trajectory and output evaluations miss this error; verifying external resource state catches it immediately.

This is where the Agent Run Contract, Service Access Contract, and Agent Authority Record function as automated verification assertions.

Observability across agent execution

Traditional observability captures system metrics, application logs, and distributed network traces. Autonomous agent architectures require capturing higher-level semantic execution events: model reasoning iterations, dynamic context retrieval, memory reads, tool selection rationale, and cross-agent task hand-offs.

The OpenTelemetry project maintains semantic conventions for GenAI, standardizing span attributes for model inference, vector search retrieval, memory operations, and tool executions (OpenTelemetry GenAI conventions). OpenTelemetry's agent conventions define span structures for autonomous planning, multi-turn loops, and execution graphs, currently maintained in active development status (OpenTelemetry agent conventions).

Open telemetry standards prevent vendor lock-in. However, standard telemetry must be augmented with business context: mapping distributed spans back to the enterprise job ID, customer tenant ID, and authorized policy version, while strictly redacting credentials and sensitive personal data.

Privacy and data governance in observability logging

Full execution logging captures user prompts, retrieved documents, tool payloads, and model outputs. In regulated enterprise environments, comprehensive logging introduces significant data privacy and security risks.

Production architectures enforce strict data segmentation:

• Operational metadata (job IDs, model version, latency, token counts, HTTP status codes) is logged continuously.
• Customer prompts and retrieved documents are sampled, masked, or referenced via cryptographic hashes.
• API credentials, session keys, and database passwords are automatically stripped prior to trace export.
• High-impact decisions retain encrypted, access-controlled audit trails subject to formal retention policies.
• Access to raw execution traces is restricted by role-based access control and audited continuously.
• Data retention, automated deletion schedules, and statutory legal-hold procedures apply to all telemetry databases.

Who supplies evaluation, observability and security infrastructure

The market spans hyperscaler monitoring suites, specialized AI observability platforms, open telemetry standards, and enterprise AI security gateways:

Market layer	Representative platforms	Commercial model	Documented capability	Operational boundary
Integrated platform evaluation	OpenAI Evals and trace grading, Google Vertex AI agent evaluation, Microsoft Foundry observability	Bundled with foundation model and hyperscaler cloud consumption	Automated test datasets, LLM-as-a-judge rubrics, trace scoring, and native cloud dashboards	Proprietary data formats; tightly coupled to the vendor’s hosting environment and model lifecycle
Specialized AI observability	LangSmith, Arize Phoenix, Langfuse	Open-source core with hosted SaaS and enterprise VPC tiers	Distributed execution tracing, prompt versioning, offline experiment comparison, and online evaluation	Telemetry ingestion and visualization; application must define its own business acceptance tests
Cloud infrastructure operations	Amazon Bedrock AgentCore Observability, Azure Monitor, Google Cloud Operations	Metered ingestion, log storage, and dashboard monitoring	Metrics, logs, traces, and automated alarms integrated into corporate cloud infrastructure	Infrastructure telemetry captures compute and network health, not domain-level business deliverable quality
Telemetry instrumentation standards	OpenTelemetry GenAI conventions, OpenInference	Open-source Apache-2.0 vendor-neutral standards	Shared semantic attributes for spans, metrics, and events across models, tools, and agent runtimes	Standardizes data transport; several agent-specific conventions remain in development status
Risk and evaluation frameworks	NIST AI RMF, NIST TEVV-Athlon draft, MITRE ATLAS	Public government and non-profit guidance	Structured risk management functions, adversarial threat matrices, and evaluation methodologies	Provide governance frameworks and threat models; do not automate testing on production systems
Agent security standards	OWASP Top 10 for Agentic Applications 2026, ASD agentic AI guidance	Public security guidance	Threat categorizations, architectural attack surfaces, and defensive design principles	A defensive checklist does not guarantee protection in an unvetted deployment
AI runtime security platforms	Palo Alto Networks Prisma AIRS, Microsoft Defender for Cloud AI posture	Enterprise cybersecurity software licensing	Automated red-teaming, runtime prompt-injection detection, data exfiltration filters, and IAM auditing	Runtime detection latency, false-positive blocking rates, and coverage must be validated on live traffic

AWS AgentCore Telemetry exports OpenTelemetry-compliant metrics, spans, and logs directly into CloudWatch, tracking sessions, latency, token burn, and error rates (AWS AgentCore observability). Microsoft Foundry provides end-to-end tracing and evaluation across agent lifecycles, supporting third-party frameworks. Google Vertex AI evaluates agent execution against structured quality, tool precision, hallucination, and safety rubrics.

Open-source and independent platforms like Arize Phoenix, Langfuse, and LangSmith provide multi-cloud flexibility, allowing enterprises to monitor agent workloads across heterogeneous model providers and private VPCs (Arize Phoenix docs; Langfuse docs).

Security across the agent attack surface

Securing autonomous software requires defending an expanded attack surface: system prompts, dynamic context retrieval, persistent memory, tool parameters, network credentials, and external systems of record. Protecting only the input prompt and output text ignores the surfaces where real harm occurs.

The OWASP Top 10 for Agentic Applications (2026) categorizes critical architectural vulnerabilities: goal hijacking, tool misuse, identity abuse, supply-chain risks, untrusted code execution, memory poisoning, insecure multi-agent communication, cascading failures, trust exploitation, and rogue autonomous behavior (OWASP Top 10 for Agentic Applications).

The Australian Signals Directorate emphasizes that prompt injection has no fully reliable model-only solution. The ASD recommends defense-in-depth: sandboxing execution runtimes, enforcing least-privilege tool access, validating parameter schemas, isolating sensitive data, and mandating human approval before committing external actions (ASD, "Agentic AI harnesses"; "Careful adoption of agentic AI services").

Prompt screening can identify known attack patterns, while a deterministic policy engine intercepts unauthorized database writes. The first control detects the threat; the second prevents the operational damage.

Decoupling prevention, detection and containment

A resilient security posture separates controls across four functional phases:

Control phase	Operational mechanism	Mandatory audit evidence
Prevent	Least-privilege IAM, tool allow-lists, strict JSON validation, container sandboxes, human approvals	Active policy configuration hashes, denied-action test logs, deployment records
Detect	Anomaly detection, unauthorized API invocation alerts, cost spikes, data exfiltration flags	Security alerts, execution trace logs, anomaly detector version, triage notes
Contain	Automated kill switches, token revocation hooks, task queue cancellation, network egress blocking	Containment trigger timestamp, affected job graph, access denial logs
Recover	Transaction compensation routines, database rollbacks, customer notifications, task re-execution	Pre-failure and restored database state, authorization records, incident post-mortem

A model guardrail that screens input text cannot revoke an exposed OAuth token, halt an active background worker, or roll back a corrupted database record. Organizations must build containment and recovery workflows before granting agents write permissions.

The six-stage operational control loop

Production systems require an ongoing operational control loop:

1. Specify   -> Task scope, acceptance criteria, budget caps, and severe failure definitions
2. Instrument -> Job IDs, trace spans, model/prompt versions, tool receipts, and costs
3. Pre-test  -> Offline evaluations, regression suites, adversarial red-teaming, recovery drills
4. Monitor   -> Real-time telemetry, latency SLAs, policy violations, and human review minutes
5. Contain   -> Incident response, automated kill-switches, token revocation, customer alerts
6. Re-assess -> Updating evaluation suites with real production failures before version upgrades

1. Specification: Defining the business deliverable, verifiable acceptance criteria, authorized budget limits, and explicitly prohibited actions.
2. Instrumentation: Tagging every execution step with stable job, run, and tenant identifiers, capturing version hashes for models, prompts, tools, and policies.
3. Pre-release verification: Stress-testing the workflow across normal, edge, and adversarial test cases; high-consequence failure modes must act as blocking gates.
4. Runtime observability: Continuously monitoring execution latency, token costs, policy denial rates, and manual human correction minutes.
5. Incident containment: Providing designated operators with the technical tools to pause workflows, revoke credentials, and isolate compromised containers immediately.
6. Continuous re-evaluation: Automatically incorporating real production failures into offline evaluation test suites prior to deploying updated model weights or prompts.

The National Institute of Standards and Technology’s TEVV-Athlon (Test, Evaluation, Verification, and Validation) framework establishes this progression, emphasizing that AI evaluation must connect lab testing to real-world operational and economic impact (NIST TEVV-Athlon Framework).

Evaluating control and observability claims

Analyzing vendor observability and security claims requires matching evidence types to the operational conclusions they support:

Evidence source	What it establishes	Unresolved operational question
Threat framework (OWASP, MITRE)	Structured taxonomy of known risks and attack techniques	Actual prevalence, vulnerability, or control effectiveness within a specific deployment
Vendor platform documentation	Supported tracing features, dashboard metrics, and alert APIs	Completeness of instrumentation, deployment configuration, or net reduction in error rates
Offline benchmark scorecard	Relative performance across curated test datasets and rubrics	Real-world reliability across messy enterprise inputs or novel prompt injection vectors
Distributed execution trace	Granular chronology of model calls, tool executions, and latency	Factual accuracy of reasoning, data authorization, or client business value
LLM-judge evaluation	Programmatic quality scores assigned across sample outputs	Ground truth reliability, judge model hallucinations, or subtle compliance failures
Database state verification	Proof that an external record was created, updated, or deleted	Verification that the client accepted the overall deliverable as complete
Adversarial penetration test	Resilience against specific tested red-team attack vectors	Guarantee that unpredicted prompt structures cannot bypass system guardrails
Incident drill post-mortem	Evidence that operators contained a simulated failure	Operational performance during unpredicted third-party infrastructure outages
Production job record	Empirical proof that an authorized run satisfied business tests	Safety and performance across unrelated business workflows

Vendor documentation confirms tool supply. Enterprise reliability requires empirical testing, verified state changes, and auditable delivery records.

Unified architecture: five linked control views

The analytical frameworks developed across Part II function as linked views of a single underlying business job, bound by immutable correlation keys:

Agent Control Evidence Card
  -> job record: objective, owner, consequence and acceptance
  -> Agent Run Contract: one execution lifecycle
       -> Context Contract: permitted information boundary
       -> Service Access Contract: one capability and effect boundary
       -> Delegation Envelope: one parent-to-child agent edge
       -> Agent Authority Record: principal, action, limits and revocation
  -> resource receipts, trace, evaluation, acceptance and incident records

Operational parameter	Authoritative system of record	Referenced by downstream systems
Objective, scope, and acceptance criteria	Business Job Record (Sponsoring Business Unit)	Every contract, execution trace, and evidence card
Execution state, worker lease, and recovery	Agent Run Contract (Orchestration Engine)	Distributed traces, tool calls, sub-agent tasks, incident logs
Permitted context, retrieval, and memory	Context Contract (Enterprise Data Platform)	Execution runs, evaluation suites, and compliance audits
API schemas, idempotency, and receipts	Service Access Contract (Integration Gateway)	Execution runs, authority decision engines, acceptance ledgers
Sub-agent delegation, scope, and lineage	Delegation Envelope (Task Coordinator)	Child runs, authority mandates, and final evidence cards
Principal, spending caps, and revocation	Agent Authority Record (Enterprise IAM / Policy Engine)	Every external tool execution and sub-agent delegation
Evaluation results and scorecards	Evaluation Ledger (Quality Assurance Team)	Release approval records and model upgrade reviews
Distributed traces and execution metrics	Telemetry Store (Observability Platform)	Incident investigations, unit cost accounting, error triage
Incident containment and recovery actions	Incident Log (Security Operations Center)	Regulatory breach notifications, client remediation, post-mortems
Deployment release and expansion gates	Release Governance Record (Enterprise Risk Committee)	Production deployment pipelines and monitoring cadences
Data classification, residency, and retention	Data Governance Ledger (Chief Privacy Officer)	Context stores, tool connectors, trace logs, retention audits
Commercial contracts, billing, and SLAs	Enterprise Commercial Record (Procurement / ERP)	Vendor sub-delegations, customer invoices, dispute records

When two systems disagree, the stricter control boundary must govern. An active execution run cannot widen permissions beyond the current Agent Authority Record. A protocol reporting completed status cannot override a business acceptance rejection. A distributed trace cannot override confirmed database state.

Explicit governance decision rights

A production workflow requires unambiguous assignment of operational decision rights:

Decision right	Accountable organizational role	Mandatory evidence required
Authorize external action	Business Sponsor & Resource Owner	Valid Agent Authority Record matching exact parameter digests
Accept business deliverable	Client or Designated Business Reviewer	Delivered artifact meeting documented acceptance criteria
Halt execution (Kill switch)	Designated Stop Authority (Security / Ops)	Immediate credential revocation and worker container termination
Reconcile ambiguous state	Technical Reconciliation Owner	Direct database query confirming external write status
Command incident response	Incident Commander	Formal containment, evidence preservation, and stakeholder notification
Govern data and privacy	Chief Privacy Officer / Data Owner	Verification of data processing regions, retention, and deletion
Manage commercial recourse	Contract Owner / Legal Counsel	Service credit claims, contractual indemnification, and dispute resolution

The Agent Control Evidence Card

Every production agent workflow requires a structured evidence card linking release decisions to operational audit trails:

Governance field	Mandatory verification evidence
Task scope & risk	Explicit business deliverable, client identity, acceptance criteria, and catastrophic failure definitions
Version registry	Immutable version hashes for model weights, system prompts, tool schemas, data indexes, and policy rules
Authority boundary	Sponsoring principal ID, business sponsor, spending limits, and emergency revocation endpoints
Integrity metadata	Evidence card ID, version number, issuing authority, and cryptographic links to source records
Evaluation test suite	Benchmark datasets, edge-case distributions, adversarial attack fixtures, and coverage blind spots
Evaluator methodology	Deterministic assertion rules, ground-truth test sets, human review rubrics, and judge-model calibrations
Pre-release results	Pass/fail rates, severe failure counts, latency distributions, and formally accepted exceptions
Observability configuration	OpenTelemetry span attributes, sampling ratios, credential masking rules, and log retention schedules
Data governance record	Processing jurisdictions, zero-retention agreements, sub-processor registers, and deletion proofs
External effect receipts	Database transaction receipts, idempotency key logs, and status query reconciliation endpoints
Unit economic return	Delivered volume, first-pass acceptance rates, manual rework minutes, and fully loaded cost per accepted job
Security audit results	Policy denial counts, prompt-injection defense logs, sandbox containment tests, and incident post-mortems
Operational ownership	Designated business sponsor, technical lead, security lead, and emergency stop authority
Regulatory & dispute recourse	Mandatory stakeholder notifications, human dispute channels, contractual SLAs, and refund liabilities
Scheduled review triggers	Mandatory audit dates and vendor events (e.g., model deprecation) triggering full re-evaluation

Governance release criteria

• Production release: Permitted only when all catastrophic failure gates pass with zero exceptions, and baseline task completion meets documented thresholds.
• Workflow expansion: Permitted only after production telemetry demonstrates stable first-pass acceptance, manageable human review overhead, and zero uncontained security incidents.
• Immediate termination: Triggered when an agent exceeds spending or data boundaries, uncontained security anomalies are detected, audit logs fail, or unit operating costs exceed the commercial value of accepted deliverables.

Catastrophic failure modes must remain visible as discrete, un-averaged metrics. Diluting a severe data exfiltration or unauthorized payment into an aggregate "98% accuracy score" conceals the exact risks that make autonomous software commercially unviable.

The trajectory of evaluation, observability and security

Evaluation, observability, and security tooling are converging around the unified job record. Independent software categories will continue to exist, but enterprise buyers will demand unified platforms connecting foundation model reasoning, external tool side-effects, corporate authority mandates, business deliverable acceptance, and financial unit economics.

Standardized telemetry conventions like OpenTelemetry will make execution traces portable across cloud platforms. However, operational competitive advantage will center on domain-specific evaluators tied to real business outcomes, deterministic policy gateways that intercept actions at the execution boundary, and immutable audit evidence that withstands regulatory scrutiny.

Part II established the technical foundation required to execute, bound, observe, and accept autonomous work within an enterprise. Part III moves outside the enterprise boundary: finding and pricing external services, executing digital transactions, settling payments, managing ongoing commercial supplier relationships, and designing sustainable business models around autonomous trade.

Sources and method

• Primary sources: Technical documentation and specifications from OpenAI (Evals, trace grading), Google Cloud (Vertex AI evaluation), Microsoft (Azure Foundry, Defender for Cloud), AWS (Bedrock AgentCore), LangChain (LangSmith), Arize AI (Phoenix), Langfuse, OpenTelemetry, W3C, NIST (AI RMF, TEVV-Athlon draft), and OWASP (Top 10 for Agentic Applications 2026) checked as of 13 September 2026.
• Analytical frameworks: The five-layer measurement model, the six-stage control loop, the unified business state vocabulary, and the Agent Control Evidence Card are original analytical structures developed to evaluate agent reliability and security governance.
• Evidence boundaries: Provider documentation establishes declared feature sets and API parameters. Production fault tolerance, prompt-injection defense effectiveness, and unit economics require empirical testing across enterprise workloads.
• Institutional references: Cyber-security guidance from the Australian Signals Directorate (ASD) establishes operational baselines for securing agentic harnesses and managing indirect prompt injection.
• Editorial record: Published by Agentic Economy, Perth, Western Australia. Updates and corrections are logged via the editorial record.

Back to the report hub · Next: discovery, marketplaces and offers.