Evaluation, observability and security for AI agents

On this page

Chapter 10 of The Hitchhiker's Guide to the Agentic Economy

Report hub · Previous: identity, authority and policy · In this chapter: what to measure · market map · control loop · evidence card · Next: discovery, marketplaces and offers

Our view

An agent trace can show what ran. It cannot by itself show that the job was correct, authorised, safe or worth paying for. Organisations need to join five kinds of evidence around the same job: component behaviour, workflow execution, operational telemetry, business acceptance and security consequence.

Product supply is converging in shape around traces, datasets, evaluators, policy controls and security products. The cited documentation establishes that feature pattern, not adoption or control effectiveness. The unresolved work is organisational. Teams still have to define acceptable outcomes, capture resource effects, decide when to stop the system and give someone authority to respond.

Key Takeaways

• Evaluate the whole job as well as the model response. Tool selection, arguments, side effects, recovery and acceptance can fail independently.
• Observability records activity. Evaluation judges it. Security controls prevent, detect and contain harmful effects. None substitutes for the others.
• The emerging control stack links traces, approvals, resource events, costs and acceptance through stable job and run identifiers.
• Prompts, model versions, tools, policies, datasets and evaluators are becoming versioned production inputs.
• A high average pass rate can hide a low-frequency severe failure. Release gates need consequence-weighted thresholds and hard stops.

This chapter uses standards, government guidance and provider documentation checked on 13 September 2026. Several agent-specific conventions and products are still in development or preview. The status is part of the finding.

Sources and scope

Agentic Economy compared current official guidance, open specifications and product documentation. No listed product was independently benchmarked for this chapter. Capability records therefore establish documented supply only. About Agentic Economy explains the publication. Corrections can be submitted through the contact page.

Evaluation is becoming a multi-layer market

The market is separating model evaluation, agent evaluation, observability and agent security into related product categories. Model providers publish benchmark and system-card results. Evaluation platforms test responses and trajectories. Observability products capture traces, costs and tool activity. Security vendors add runtime controls, scanning, policy enforcement and incident detection.

These categories are already converging inside platforms, but they continue to answer different questions. A trace records what ran. An evaluator judges a result against a criterion. A security control prevents, detects or contains a harmful effect. We expect the market to consolidate around joined operational records, while specialised evaluators and security products remain important for high-consequence domains.

One job needs several kinds of proof

Five measurement layers answer different questions:

1. Model and component evaluation: did a model, retrieval step, classifier or tool call meet a defined criterion?
2. Trajectory and workflow evaluation: did the system choose and execute an acceptable sequence of steps?
3. Operational observability: what ran, how long did it take, what did it cost and where did it fail?
4. Business acceptance: did the responsible customer or operator accept the resulting work, and with how much correction?
5. Security and control evidence: did the system remain within authority, resist tested attacks and allow containment and investigation?

A model can produce a correct answer after an unauthorised data lookup. A workflow can follow the intended trajectory and still deliver no customer value. A security filter can block a malicious prompt while the agent sends a message to the wrong customer through a valid tool. Measurement preserves these distinctions.

Evaluation is measurement of behaviour against a defined test. Observability is the recording of enough state and activity to inspect a run. Security is the set of controls that constrains, detects, contains and recovers from harmful behaviour. Business acceptance is the decision that the delivered result met the promised outcome.

The minimum join key

Stable identifiers connect the records:

job_id
  -> run_id and parent_run_id
  -> agent, model, runtime, tool and policy versions
  -> identity, authority and approval decisions
  -> trace, spans, logs and costs
  -> external resource receipts and side effects
  -> delivered output
  -> acceptance, rework and incident state

Without the join, a team can optimise token cost while losing acceptance, reduce latency while increasing unsafe retries, or close an incident without identifying every affected resource.

Evaluation starts with a decision, not a generic score

An evaluation set represents the jobs, boundaries and failures the organisation cares about. It includes normal cases, hard cases, prohibited actions, adversarial inputs and recovery cases. The expected result can be a correct output, an acceptable trajectory, a refusal, a hand-off or an unchanged resource.

Evaluator types have different failure modes:

Evaluator	Useful for	Main limitation
Deterministic rule	Schema, exact values, permissions, tool arguments, resource state and hard limits	Cannot judge many contextual or qualitative outcomes
Reference comparison	Known answers, expected records and required tool events	Reference can be incomplete or allow several valid routes
Human review	Consequence, usefulness, professional judgement and ambiguous cases	Costly, variable and difficult to scale without calibration
Model judge	Relevance, quality or policy rubrics across larger samples	The judge has its own errors, bias, drift and data exposure
Metamorphic or invariant test	Behaviour that should remain stable as irrelevant input changes	Requires careful design and may miss realistic distribution shifts
Adversarial test	Goal hijack, tool misuse, credential abuse, data leakage and cascading failure	A passed test set does not prove the absence of other attacks

OpenAI's current trace grading documentation defines trace grading as assigning structured scores or labels to the end-to-end record of decisions, tool calls and reasoning steps. It positions trace evaluation as a way to find where an agent succeeded or failed, rather than treating the final response as a black box. LangSmith similarly separates offline evaluation on curated datasets from online evaluation on production traces (LangSmith, "Evaluation concepts").

Judge the effect as well as the trajectory

Trajectory evaluation can detect a wrong tool or missing step. It can also punish a different valid path. For consequential work, attach deterministic checks to the external effect:

For example, a retried CRM write may produce the right visible answer and still create two customer records. The effect check catches what a response score misses.

• the intended record changed once;
• no unauthorised record changed;
• the recipient and attachment matched approval;
• the payment amount and merchant matched the mandate;
• the deployment passed the required tests;
• the cancellation stopped later steps; and
• the resource receipt can be reconciled with the trace.

This is where the Agent Run Contract, Service Access Contract and Agent Authority Record become test inputs rather than documentation.

Observability needs agent semantics and ordinary operations

Traces, metrics and logs remain the foundation. Agent systems add a need to represent model calls, retrieval, memory, planning, tool execution, hand-offs and long-running workflow state.

The OpenTelemetry GenAI semantic conventions currently define spans for model, retrieval, memory and tool operations, and a separate agent-and-framework document covers agent creation, invocation, workflows and planning. The agent conventions are explicitly marked Development, which makes telemetry mapping versioning and change management part of the implementation (OpenTelemetry, "Semantic conventions for GenAI agent and framework spans").

Open standards can reduce exporter lock-in. They do not create a complete data model for the business job. The organisation's layer adds low-cardinality job, tenant, workflow, version and outcome references without placing secrets or unnecessary personal data into telemetry.

More logs can create a second risk

Prompts, retrieved documents, tool arguments and model outputs may contain confidential, personal or regulated data. Full capture can improve investigation while expanding storage, access and retention risk.

The market separates:

• metadata that is always recorded;
• content that is sampled or redacted;
• secrets and credentials that are never logged;
• high-impact events that require full protected evidence;
• who may inspect raw content;
• retention, deletion and legal-hold rules; and
• how an incident responder obtains the necessary evidence.

Hashing or storing a reference can prove which version influenced a run without copying the full content into every observability platform.

Who supplies evaluation, observability and security

Layer	Representative supply	Buyer and business model	Documented capability	Boundary
Integrated agent evaluation	OpenAI Evals and trace grading, Google Vertex AI agent evaluation, Microsoft Foundry observability	Builders already using the model or cloud platform; usage and platform pricing	Datasets, graders, trace or tool evaluation, dashboards and platform integration	Convenient but tied to provider data models, regions and lifecycle
Specialist evaluation and observability	LangSmith, Arize Phoenix, Langfuse and other specialist platforms	AI application teams; open-source, hosted usage and enterprise plans vary	Tracing, datasets, experiments, human or automated evaluators and production monitoring	Feature support and telemetry portability differ; product use does not define the buyer's acceptance test
Cloud operations integration	Amazon Bedrock AgentCore Observability, Azure Monitor/Application Insights and Google Cloud operations	Cloud platform and operations teams; metered monitoring and storage	Metrics, traces, logs, dashboards and alerts integrated with cloud resources	Operational telemetry may not contain domain acceptance or external resource evidence
Telemetry standards and instrumentation	OpenTelemetry GenAI conventions and OpenInference	Platform, framework and observability vendors; open standards and libraries	Shared span, metric and event vocabulary for model and agent operations	Several agent-specific conventions remain in development
Risk and evaluation frameworks	NIST AI RMF, NIST TEVV-Athlon draft, MITRE ATLAS	Risk, assurance and security teams; public guidance	Risk functions, evaluation structure and adversarial technique knowledge	Frameworks guide programmes; they do not test a deployed system automatically
Agent security guidance	OWASP Top 10 for Agentic Applications 2026 and ASD agentic AI guidance	Builders, defenders, executives and assurance teams; public guidance	Threat categories, architectural risks and mitigations	A checklist completion is not proof that controls work in the deployed route
AI security platforms	Palo Alto Networks Prisma AIRS, Microsoft Defender AI agent posture and other AI security vendors	Enterprise security teams; licences or subscriptions	Discovery, posture, red teaming, runtime inspection, policy and alerts vary by product	Detection coverage, latency, platform support and false decisions require independent testing

This is a functional market map, not a vendor ranking. Large cloud, model, framework, observability and security providers increasingly span several rows. Their evidence paths and integration boundaries are more revealing than feature counts.

Product convergence is real, proof remains local

AWS documents AgentCore telemetry in OpenTelemetry-compatible form and stores metrics, spans and logs in CloudWatch. Its built-in measures include sessions, latency, duration, token use and error rates (AWS, "Observe your agent applications"). Microsoft Foundry documents evaluation, monitoring and tracing across its agent lifecycle and supports tracing for several external frameworks. Google Vertex AI exposes agent-specific rubric metrics such as final-response quality, tool-use quality, hallucination and safety.

Specialist platforms cover similar loops with different deployment and commercial models. Phoenix is open source and uses OpenTelemetry with OpenInference. Langfuse supports hosted and self-hosted operation, online trace scoring and offline experiments. LangSmith connects traces, datasets, offline tests and online evaluators. These documented features establish available product supply. They do not show which product produces a lower incident rate or higher accepted-job rate for a buyer's workflow.

Security is a property of the connected system

The attack surface crosses model, context, memory, runtime, tools, identities, messages and resource systems. Protecting only the prompt and response misses many of the consequences.

OWASP's 2026 agentic list groups risks around goal hijack, tool misuse, identity and privilege abuse, supply-chain vulnerabilities, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation and rogue behaviour. It is a community risk framework, not a prevalence ranking (OWASP, "Top 10 for Agentic Applications 2026").

The Australian Signals Directorate makes the control boundary unusually clear. Its September 2026 guidance defines the harness as the layer connecting the model with context, memory, tools, permissions and the execution loop. It says no harness is inherently secure and that prompt injection has no fully reliable model-only mitigation. The recommended response is defence in depth, least privilege, controlled tools and data, validation, human oversight, monitoring, supply-chain assurance and phased deployment (ASD, "Agentic AI harnesses"; "Careful adoption of agentic AI services").

This guidance is globally useful even though it comes from Australia. It changes the technical conclusion: agent security belongs in the harness and surrounding systems, not in a promise that the model will follow instructions.

For instance, prompt screening may identify a hostile instruction while a separate policy gate prevents the proposed database write. The first control explains the threat. The second limits the effect.

Separate prevention, detection and containment

One control rarely performs all three jobs.

Control objective	Example controls	Evidence to retain
Prevent	Least privilege, allowlisted tools, schema validation, network isolation, approval and resource policy	Configuration, policy version, denied-action tests and deployment record
Detect	Trace anomalies, unauthorised tool attempts, cost spikes, data leakage signals and acceptance drift	Alert, trace, resource event, detector version and triage result
Contain	Kill switch, token revocation, queue cancellation, sandbox termination and egress block	Trigger time, affected graph, successful denials and residual work
Recover	Compensating action, rollback, restore, customer notification and reprocessing	Old and restored state, approvals, communications and reconciliation
Learn	Incident case added to evals, threat model and policy tests	New test, fixed version, comparison result and review owner

A model guardrail that classifies prompts may prevent some known inputs. It does not revoke a leaked credential, cancel a queued action or restore a changed customer record. Design the response path before granting the capability.

The control loop has six stages

1. Specify

The specification contains the job, acceptance criteria, authority, prohibited effects, severe-failure classes, performance target and accountable owner. If “good” and “stop” cannot be stated, a generic evaluator will not discover them.

2. Instrument

Instrumentation assigns job and run identifiers; captures model, prompt, tool, policy, data and runtime versions; records approvals and external effects; and distinguishes protected, sampled, referenced and excluded content.

3. Test before release

Pre-release evidence spans representative jobs, historical failures, adversarial cases and recovery drills. It covers normal completion, refusal, approval change, timeout, duplicated request, revoked credential and downstream outage. Severe failure classes remain hard gates rather than inputs to an average score.

4. Observe production

Production observation covers technical SLOs, cost, authority denials, unusual tool sequences, acceptance, rework and incident signals. Outputs are sampled where lawful and useful, and real failures become future test cases.

5. Respond

A named operator needs authority to pause the workflow, revoke credentials, cancel queued work, contain the environment, inspect evidence and notify affected owners. That response path is an operating capability, not a policy sentence.

6. Re-evaluate change

A change to a model, prompt, tool, connector, memory policy, runtime, evaluator or resource system reopens the relevant evaluation. Comparable cases reveal whether the new version warrants broader scope.

NIST's AI RMF remains a voluntary cross-sector risk framework, and its 2024 Generative AI Profile adds technology-specific considerations. NIST's August 2026 TEVV-Athlon publication is an initial public draft covering models, agentic systems and other AI technologies; it aims to connect evaluation to real-world impact and outcomes. Treat it as an emerging framework and track the final version rather than labelling it settled guidance (NIST, "The TEVV-Athlon Framework for Evaluating AI Systems").

What the sources establish

Evidence	What it establishes	What it does not establish
Framework or threat taxonomy	A reusable set of risks, functions or test considerations	Prevalence or control effectiveness in one deployment
Provider documentation	Available tracing, evaluation, security or response features	Complete instrumentation, correct configuration or outcome improvement
Offline evaluation	Performance on the defined cases, versions and evaluators	Production distribution, authority or incident rate
Production trace	Recorded execution path and captured attributes	Correctness, complete causality or business value
Online evaluator	A score or label over selected production records	Ground truth unless independently established
Resource-state test	Whether the intended external effect occurred	Whether the customer accepted the broader result
Security test	Behaviour against the tested attack and environment	Absence of untested vulnerabilities
Incident exercise	Whether named people and controls handled the scenario	Performance under every real incident condition
Accepted-job record	Whether delivered work met a stated business test	Safety or authorisation unless those are included in the acceptance boundary

Current evidence establishes a broad supply of instrumentation, evaluation workflows, security taxonomies and runtime controls. It does not establish a universal agent reliability score, a complete observability standard or independently measured superiority among providers.

One job record, five linked control views

The Part II artefacts are Agentic Economy proposals, not standards or certification evidence, and they are not separate sources of truth. One authoritative business job record references the layer-specific records by stable identifier. The Agent Control Evidence Card is an index used for release and operation; it links to evidence rather than copying mutable policy or resource state.

Agent Control Evidence Card
  -> job record: objective, owner, consequence and acceptance
  -> Agent Run Contract: one execution lifecycle
       -> Context Contract: permitted information boundary
       -> Service Access Contract: one capability and effect boundary
       -> Delegation Envelope: one parent-to-child agent edge
       -> Agent Authority Record: principal, action, limits and revocation
  -> resource receipts, trace, evaluation, acceptance and incident records

Fact	Authoritative owner	Referenced elsewhere by
Objective, customer, consequence and acceptance test	Business job record; business owner	Every contract and the evidence card
Run state, attempt, deadline and recovery	Agent Run Contract; runtime owner	Trace, service call, child task and incident
Permitted source and memory boundary	Context Contract; data owner	Run, evaluation and evidence records
Operation, schema, idempotency and resource receipt	Service Access Contract; integration or resource owner	Run, authority decision and acceptance
Parent, child, scope and artefact acceptance	Delegation Envelope; parent-task owner	Run, authority and evidence card
Principal, actor, purpose, action, policy and expiry	Agent Authority Record; business sponsor and resource owner	Every consequential effect and delegated edge
Evaluation result and evaluator version	Evaluation result record; assurance owner	Evidence card, release decision and next version review
Trace and event chronology	Telemetry records; operations owner	Evidence card, evaluation, reconciliation and incident
Incident state, containment and recovery	Incident record; incident commander	Evidence card, affected-party notice, remedy and next version review
Release, expansion or stop decision	Release decision record; named release owner	Evidence card, governance and operations
Data classes, region, retention, deletion and lawful hold	Data control record; data or privacy owner	Context, service access, delegation, telemetry and incident records
Counterparty, service level, billing, liability and remedy	Commercial record; contract or service owner	Delegation, effect, acceptance, incident and dispute records

When two records disagree, the stricter current boundary governs until the source of truth is resolved. A copied permission in a run cannot widen the current Authority Record. A protocol's completed state cannot override the business acceptance record. A trace cannot override the external resource's confirmed state.

Name decision rights, not only owners

Each referenced record carries an immutable ID, version, issuer and effective time. The job also names the decision rights below; the same person may fill several roles for a small deployment, but no role remains implicit.

Decision right	Accountable role	Required authority or evidence
Authorise an effect	Business sponsor and resource owner	Current Authority Record bound to the exact operation, target and effect representation
Accept the outcome	Customer or business acceptance owner	Delivered result and acceptance test; independent of provider completed state
Stop work	Named stop authority	Ability to pause runs, revoke credentials, cancel queued work and block the resource boundary
Resolve conflicting state	Reconciliation owner	Resource receipt or authoritative resource read, deadline and escalation route
Command an incident	Named incident commander	Scope, containment decisions, evidence preservation, notification duties and recovery owner
Control protected evidence	Data or privacy owner	Permitted data classes, access, region, retention, deletion, legal hold and proof of disposal
Provide recourse	Contract, service or regulated-entity owner	Correction, restoration, rework, refund, liability, notice and dispute path
Answer affected people	Named accountable organisation	Notice, explanation, contest or appeal route and applicable jurisdictional obligations

If roles disagree, the stop authority may narrow or halt work; it cannot widen authority or declare acceptance. The accountable organisation remains responsible for resolving the conflict and providing any required notice or remedy across its suppliers.

One identifier and state vocabulary

The minimum record contains job_id, run_id, attempt_id, effect_id, approval_id, authority_id, service_access_id, any delegation_id and child task_id, each artefact_id, trace_id, resource receipt and acceptance_id. IDs are immutable, tenant-and-job scoped, export-stable and distinct for every repeated effect, attempt, approval, child task, artefact and acceptance decision. Systems may use different field names, but the joins and event order must survive export and incident review. An append-only sequence or transition ID sits beside event time because clocks alone cannot order distributed events reliably.

Business states sit above provider-specific states:

job_opened -> authorised -> executing
  -> waiting_for_input | awaiting_approval | retry_wait
  -> outcome_unknown -> reconciling
  -> delivered -> accepted | rework | rejected

side exits: approval_denied | expired | failed | cancellation_pending |
            cancelled | compensating | compensated

Provider and protocol states map into this vocabulary rather than redefining the job. Cancellation has its own ladder: requested, acknowledged, execution stopped, effect prevented, or effect already present and reconciled or compensated. Cancelled is not a complete business outcome until the last permitted or residual effect is known.

Business state	Runtime view	Service-access view	Delegation or protocol view	Authority and evidence view
job_opened	Run created, not executing	Route selected, no call	Candidate or parent task identified	Principal, owner and acceptance test recorded
authorised	Current run may enter the next step	Operation, target and effect are within the access contract	Delegation envelope accepted for this edge	Current policy allows the represented action; approval present if required
executing	Worker owns a lease and a step is running	Request or browser action is in progress	A2A or local task is working	Credential and policy remain current; trace is active
waiting_for_input	A required field or user response blocks progress	Provider or route needs specified input	Protocol reports input-required or application wait	Permitted responder, input boundary and expiry remain visible
awaiting_approval	Exact proposed effect is ready but not authorised	Route is paused before the effect boundary	Protocol or application awaits approval	Canonical target, operation, material fields, timing and approval digest are recorded; denial or expiry cannot become approval through retry
retry_wait	A classified retry and bounded timer block progress	Rate limit or declared transient failure is pending	Application waits outside or around protocol state	Attempt budget, same logical effect and safe-replay evidence remain visible
approval_denied or expired	Run stops or follows a predeclared narrower path	No unapproved effect may be sent	Parent, child or protocol wait closes without widening scope	Denial or expiry reason, timestamp and any residual effect are recorded
outcome_unknown	Run cannot prove the effect	Timeout, disconnect or ambiguous response after a possible action	Parent and child state or callback evidence disagree	No success or cancellation claim; reconciliation owner and deadline required
delivered	Execution produced an output and can stop running	Resource receipt or output is attached	Protocol may report completed and return an artefact	Business acceptance remains open
accepted, rework or rejected	Job reaches its business terminal decision	Effect and receipt are reconciled	Parent accepts, returns or rejects the child artefact	Evidence Card records the decision, cost, incident and accountable owner

Provider spellings remain intact inside their native records. For example, an A2A TASK_STATE_CANCELED value can map to the business cancellation ladder without being rewritten in the protocol log.

An illustrative joined job

Consider a customer enquiry that produces a researched proposal and then updates the CRM. This example explains the joins; it is not an observed deployment or evidence that the controls work.

1. The job record names the customer, expected proposal, CRM update and human acceptance owner.
2. The Context Contract permits the submitted brief and approved research sources, but not another customer's records.
3. The Run Contract creates one run and one logical CRM effect. The Service Access Contract supplies the direct API schema, idempotency rule and status query.
4. If a specialist agent is used, the Delegation Envelope allows only a research artefact, prohibits subdelegation and sets a deadline and acceptance test.
5. The Authority Record lets the workload draft the CRM change but requires a person to approve the exact account, fields and customer message.
6. If the worker disappears after the CRM accepts the write, the run moves to outcome_unknown. The operator uses the effect ID to reconcile the resource before any retry.
7. The Evidence Card links the trace, approval, resource receipt, delivered proposal, rework, cost and final acceptance. If authority was revoked during the run, it also records the last permitted and first denied effect.

The example makes the operating model concrete without manufacturing a benchmark. A future Agentic Economy Lab note will instantiate this path against real systems and publish the environment, versions, fault injections, results and limitations.

An Agent Control Evidence Card

Each production workflow version has one control card linked to its runs. The card records the release decision and immutable references; evaluation results, traces, incidents, resource receipts and acceptance decisions remain authoritative in their own source records.

Field	Evidence required
Job and consequence	Objective, customer, acceptance test and severe-failure classes
Version set	Agent, model, prompt, runtime, tools, data, memory and policy versions
Authority	Principal, actor, sponsor, approval rule, credential and revocation path
Record integrity	Evidence-card ID, version, issuer, effective time and immutable references to current source records
Evaluation set	Population, cases, sources, prohibited actions and known coverage gaps
Evaluators	Rule, reference, human or model judge, including version and calibration
Release result	Passes, failures, severe-failure count, variance and accepted exceptions
Runtime evidence	Trace fields, logs, metrics, sampling, redaction and retention
Data and privacy evidence	Permitted classes, region, subprocessors, raw-content access, deletion or legal-hold evidence
External effects	Resource receipts, state checks, idempotency and reconciliation
Business result	Delivered, accepted, rework, elapsed time and cost per accepted job
Security result	Denials, alerts, incidents, containment and recovery evidence
Owner and response	Service owner, security owner, business sponsor and stop authority
Public impact and recourse	Affected parties, notice, contest or appeal, remedy trigger, response target, cross-provider escalation, correction, restoration, refund and dispute owner
Review trigger	Scheduled date and events that force re-evaluation

Release, expand or stop

• Release only when every severe-failure gate passes and the normal-case acceptance result meets the stated threshold.
• Expand authority, users or volume only after production evidence shows stable acceptance, manageable review and no unresolved material incident.
• Stop when authority is exceeded, a severe effect cannot be contained, evidence is missing, or the workflow's total cost exceeds the value of accepted work.

A business job remains open while a required reconciliation, affected-party notice or remedy remains unresolved. A technical compensation event may change a resource; it does not settle contractual, financial or legal recourse unless the governing record says it does.

Catastrophic failures remain visible beside the ordinary performance distribution. Averaging them into a reassuring overall score destroys the information that matters most.

What the market has not resolved

Evaluation, observability and security products increasingly share the same traces, datasets and runtime hooks. The commercial boundary between them remains fluid. An observability platform can add evaluators; an evaluation product can monitor production; a security product can inspect the same tool calls and trajectories for harmful behaviour.

We expect platform consolidation around common telemetry, with continued specialisation in domain evaluation, adversarial testing and runtime enforcement. The unresolved question is whether a small number of control planes will own the joined record or whether enterprises will continue to assemble independent tools around their existing security and operations stack.

Where we see the market going

Evaluation, observability and security are converging around the job record. Separate products will remain, but buyers will demand one trace that joins model behaviour, tool effects, authority, business acceptance, cost and incident consequence.

Shared telemetry semantics may make traces portable; they will not make judgement universal. Differentiation will move towards evaluators tied to real workflow decisions, controls that act at the effect boundary and evidence that survives provider changes and disputes.

Four questions remain open:

1. Will OpenTelemetry or another shared vocabulary make agent traces portable enough for independent assurance across platforms?
2. Which evaluation designs best predict accepted business work rather than benchmark or judge-model scores?
3. Can runtime security products demonstrate useful detection and blocking performance without unacceptable latency, false decisions or data exposure?
4. Which organisations can join agent activity, resource effects, incidents and financial outcomes well enough to manage the system as an operating capability?

The chapter's conclusion would change if a portable standard joined trace, authority, resource effect and acceptance across platforms without substantial custom work. It would also change if independent studies showed that one compact metric reliably predicted both business acceptance and severe operational failure.

About the author and editorial record

Agentic Economy is the accountable publisher of this report. Joel Chan founded the publication in Perth to research the infrastructure, companies and operating choices shaping the agentic economy. This chapter is source-led market analysis, not a sponsored ranking or proof that every listed product has been independently tested.

Previous: identity, authority and policy · Next: discovery, marketplaces and offers.