AI models and inference infrastructure for agents

On this page

Chapter 4 of The Hitchhiker's Guide to the Agentic Economy

Report hub · Previous: the agentic economy market map · In this chapter: provider map · cost · deployment · Australia · route comparison

The economics of model procurement

Foundation model capability is metered, rate-limited, and subject to regular vendor depreciation cycles.

The underlying model provides the core reasoning engine within an agent workflow. However, procuring inference is not a one-time architectural decision. An enterprise purchases a dynamically changing bundle of task capability, latency, token throughput, tool-calling reliability, and data residency guarantees. Models are billed predominantly by the token—a granular metering unit reflecting processed text and multimodal inputs—and lineups are revised every few months.

In a commercial setting, model selection is an empirical trade-off: identifying the specific model and deployment route that completes an operational task reliably at an acceptable total unit cost.

Model inference represents the computation required to run input context through trained model weights to generate a response or select an external tool action. The surrounding inference platform controls where that computation occurs, how requests are queued and rate-limited, which API versions remain accessible, and what operational telemetry is captured.

Compute infrastructure encompasses the physical accelerators, networking fabrics, cooling, electrical power, and serving runtimes supporting that computation. It can be accessed as a fully managed cloud API or hosted directly on customer-managed hardware.

Key Takeaways

• Model selection is a job-level decision, not a permanent vendor ranking.
• Per-token price is one cost. Retries, tool calls, latency, review and failed jobs often matter more.
• Direct model APIs, multi-model clouds, dedicated deployments and self-hosting create different control and operating burdens.
• Open weights expand deployment choice, but they transfer evaluation, serving, patching and capacity work to the operator or hosting provider.
• An Australian cloud region does not automatically mean every model request stays in Australia; the model, endpoint and routing policy jointly determine the route.

Four emerging inference delivery markets

The market for model inference is dividing into four distinct delivery structures: direct provider APIs, multi-model cloud platforms, dedicated managed deployments, and customer-hosted open weights. Each model sells a different balance of capability access, guaranteed capacity, regional sovereignty, operational overhead, and switching flexibility.

Concurrently, vendor boundaries are blurring. Frontier lab providers are embedding native tool execution, persistent memory stores, and orchestration runtimes directly into their developer platforms. Cloud hyperscalers offer proprietary, partner, and open-weight models through unified billing and IAM control planes. Specialized inference engines are simplifying the operation of open weights on private clusters. While access to raw model intelligence is democratizing, production differentiation is shifting toward intelligent routing, operational reliability, data privacy controls, and fully loaded task economics.

What the model supplies in an agent system

In an agentic architecture, the foundation model contributes the learned capacity to parse ambiguous instructions, reason through multi-step dependencies, and select external actions. Six operational characteristics govern enterprise performance:

1. Domain reasoning capability: Can the model execute the necessary reasoning, programming, extraction, multimodal interpretation, or analytical synthesis?
2. Tool-calling precision: Can the model reliably parse JSON schemas, construct valid parameter arguments, ingest tool return payloads, and terminate execution cleanly when the goal is achieved?
3. Instruction and schema adherence: Does the model consistently respect system prompts, negative constraints, and structured output formats required by downstream enterprise software?
4. Context retrieval and retention: Can the model process long document histories and conversational context without suffering from instruction drift or overlooking key constraints?
5. Latency and concurrency limits: Can the provider endpoint deliver sufficient token throughput under peak concurrent load to satisfy business latency SLAs?
6. Governance boundaries: Are the model versioning policies, processing regions, data retention terms, and enterprise access controls compliant with organizational standards?

The model does not operate in isolation. An orchestration runtime must catch and retry malformed JSON outputs. A retrieval pipeline can inject poisoned or obsolete context. A policy gateway can intercept unauthorized write requests. Human analysts can catch hallucinations, but at the cost of labor overhead and operational delays. The foundation model represents one component within a broader execution pipeline.

The limits of synthetic benchmarks

Standard academic benchmarks compare models against static test datasets using automated scoring rubrics. Deploying an agent into an enterprise workflow changes the operational test entirely. Real-world performance depends heavily on system prompt design, dynamic tool configurations, database context, permitted API actions, and the business cost of unhandled errors.

A model achieving frontier scores on coding benchmarks can fail in enterprise production if it cannot navigate a proprietary repository, exhausts context windows during complex refactoring, or invokes destructive database commands without authorization. Conversely, a compact, lower-cost model can deliver superior enterprise ROI on structured document extraction where schemas are fixed and exceptions are routed to human operators.

Model evaluation must be conducted against the target business task, holding tool availability, contextual inputs, and acceptance standards constant.

Who supplies models and inference

The market consists of model creators, multi-model cloud hosts, specialized serving runtimes, accelerator manufacturers, and data-center operators:

Frontier and specialized model platforms

The current provider landscape as of September 2026 demonstrates clear operational specialization:

Provider	Documented model families	Access model	Agent-specific capabilities
OpenAI	GPT-5.6 Sol, Terra and Luna; specialised realtime, computer-use, image and cyber models	Direct API and Microsoft Foundry	Frontier models support native function calling, web browsing, file search, and computer use
Anthropic	Claude Fable 5.1, Opus 5, Sonnet 5 and Haiku 4.5	Direct API, AWS Bedrock, Google Cloud, and Microsoft Foundry	Structured tiering separating frontier reasoning, balanced agent execution, and high-throughput utility; requires active lifecycle management
Google	Gemini model families plus specialised realtime, media and robotics endpoints	Direct Gemini API and Vertex AI	Deep native multimodal integration; specialized tools such as Search, Maps, and code execution carry separate billing schedules
Mistral AI	Mistral Large, Medium, Small and Ministral families; specialized OCR models	Direct API, self-hosted weights, and partner clouds	Provides flexible commercial licensing across hosted endpoints and private enterprise VPC deployments
Cohere	Command, North, Embed, Rerank, Parse and Transcribe families	Cohere platform, Model Vault, and cloud marketplaces	Decouples generative reasoning from specialized retrieval, semantic reranking, and document parsing

Model lineups evolve rapidly. Anthropic’s deprecation documentation explicitly distinguishes between active, legacy, deprecated, and retired models, committing to at least 60 days of advance notice prior to retiring public endpoints (Anthropic, "Model deprecations"). Managing model migrations is an ongoing operational requirement rather than an occasional engineering event.

Multi-model platforms and serving runtimes

Inference delivery options span managed cloud marketplaces to optimized open-weight serving runtimes:

Infrastructure layer	Representative platforms	Supplied capability	Customer operational responsibility
Managed multi-model cloud	Amazon Bedrock, Microsoft Foundry, Google Vertex AI Model Garden	Consolidated billing, IAM integration, model catalog, and hosted inference across proprietary and open models	Specifying regional routing policies, monitoring quota allocations, and managing model version deprecations
Direct provider API	OpenAI, Anthropic, Google, Mistral, Cohere	Immediate access to newest model releases and native tool features	Managing vendor concentration, negotiating data retention terms, and engineering multi-provider fallback logic
Dedicated managed inference	Cloud provisioned throughput, Cohere Model Vault, NVIDIA NIM Certified	Reserved GPU capacity delivering predictable latency and isolated workloads	Managing capacity utilization, paying for idle compute, and configuring VPC network peering
Open-source serving software	Hugging Face TGI, vLLM, SGLang	High-performance inference serving on customer-controlled infrastructure	Provisioning GPUs, tuning batching algorithms, managing driver updates, and maintaining uptime
Packaged inference containers	NVIDIA NIM and hyperscaler container images	Pre-compiled, hardware-optimized model containers with standard health endpoints	Managing container orchestration (Kubernetes), hardware licensing, and enterprise support agreements

Microsoft Foundry’s deployment documentation reflects these commercial distinctions: serverless endpoints charge per token or via provisioned throughput with configurable geographic processing boundaries, while managed compute provisions dedicated hardware billed by accelerator hour (Microsoft, "Deployment overview for Microsoft Foundry Models").

NVIDIA similarly distinguishes standard day-zero model releases from NIM Certified containers that bundle broad hardware validation, enterprise vulnerability patching, and formal support SLAs (NVIDIA, "NIM offerings"). Enterprises buy operational support and security posture alongside raw compute performance.

Token pricing versus unit economic cost

Published token pricing provides an incomplete baseline for forecasting operational expenditure. In autonomous agent workflows, the per-token price represents only a fraction of total cost per accepted deliverable.

List prices vary widely across performance tiers. As of September 2026, OpenAI lists GPT-5.6 Luna at US$0.20 per million input tokens and US$1.20 per million output tokens, while frontier GPT-5.6 Sol is priced at US$4.00 and US$20.00 respectively (OpenAI model documentation). Anthropic lists Claude Haiku 4.5 at US$1.00 / US$5.00, Claude Sonnet 5 at US$2.00 / US$10.00, and frontier Claude Fable 5.1 at US$10.00 / US$50.00 (Anthropic model overview). Mistral lists Mistral Small 4 at US$0.15 / US$0.60, and Mistral Large 3 at US$0.50 / US$1.50 (Mistral pricing).

These rates reflect raw input and output metering, not total task efficiency. Context window sizes, prompt caching discounts, output token volume, and multi-turn reasoning overhead differ substantially across providers. A more expensive model can deliver lower net cost per accepted task if its superior reasoning eliminates execution retries, reduces recursive tool loops, and minimizes manual human correction.

The Model-to-Outcome Cost Stack

Calculating the true cost of an autonomous workflow requires aggregating expenses across the full execution lifecycle:

model input and output
+ built-in and external tools
+ retrieval and data
+ runtime and hosting
+ retries and fallback models
+ human review and correction
+ incident and support cost
= total cost of delivered work

total cost of delivered work / accepted jobs = cost per accepted job

A single customer job may trigger twelve intermediate LLM calls, three search queries, and two automated retries following schema validation errors. If an unhandled hallucination requires forty minutes of manual analyst correction, human labor will dominate the cost structure regardless of how cheaply the model processed tokens.

Evaluating workload consumption profiles

Forecasting inference expenditure requires modeling the complete workload profile:

• Maximum and average context window size per turn.
• Prompt caching reuse ratios for system instructions and persistent context.
• Frequency and pricing of external tool and search invocations.
• Average reasoning iterations and recursive retry rates per task.
• Workload routing splits between lightweight classification models and frontier reasoning engines.
• Peak concurrency, request latency constraints, and provisioned throughput floors.
• Human exception-handling rates and manual review minutes per completed deliverable.

Google's Gemini pricing highlights this complexity: endpoints are differentiated across standard, batch, flex, and priority tiers, with specialized tools like Search and Code Execution carrying dedicated fee schedules (Google, "Gemini Developer API pricing"). Selecting the cheapest token tier can fail if its queueing delays breach operational latency requirements.

Four commercial inference models

Enterprises select among four operational deployment models, frequently combining several across a single workflow:

1. Direct provider APIs

Direct API access provides immediate availability of frontier model weights, native tool-calling features, and prompt caching optimizations. The provider manages physical infrastructure, billing for consumed tokens or provisioned concurrency.

This route minimizes engineering setup for teams capable of operating under the provider's standard data-retention and geographic terms. Organizations must implement their own rate-limit handling, multi-provider failover, and model version pinning.

2. Multi-model cloud platforms

Cloud platforms (AWS Bedrock, Azure Foundry, Google Vertex AI) host third-party and open models within existing corporate cloud tenancies, integrating directly into enterprise billing, VPC security, and IAM controls.

However, feature availability is not uniform. Model versions, latency profiles, and regional support vary significantly across hyperscalers. Amazon Bedrock’s cross-region inference profiles route requests across multiple geographical data centers to balance capacity; AWS explicitly advises organizations with strict data residency mandates to audit all destination regions within a selected profile (AWS, "Supported Regions and models for inference profiles").

3. Dedicated managed deployments

Dedicated cloud deployments allocate isolated accelerator hardware or reserved throughput capacity exclusively to a single customer. This approach eliminates multi-tenant queueing delays and ensures deterministic inference latency.

The primary commercial risk is capacity utilization. Reserved compute incurs fixed capital costs whether fully utilized or sitting idle. Organizations must weigh this premium against the latency volatility and rate-limiting inherent in shared public endpoints.

4. Self-hosted and customer-managed inference

Deploying open or licensed model weights onto private cloud VPCs, on-premise clusters, or edge devices gives enterprises total control over data residency, weights inspection, and runtime environments. Modern serving engines (vLLM, SGLang, Hugging Face TGI) provide continuous batching, tensor parallelism, and optimized KV-cache management (Hugging Face, "Serve CLI").

Self-hosting transfers substantial operational overhead to the customer: hardware procurement, GPU cluster maintenance, driver compatibility, security patching, and capacity planning. Open weights eliminate per-token vendor licensing fees, but replace them with fixed hardware depreciation and specialized infrastructure engineering labor.

The inference pipeline as an operational queue

A production inference request traverses multiple operational subsystems before and after the model processes tokens:

job arrives
  -> authenticate and apply quota
  -> choose model, version and region
  -> assemble and cache context
  -> queue and batch work
  -> run on accelerators
  -> stream or return output
  -> validate tool call or schema
  -> record usage, latency and errors
  -> retry, escalate or accept

This pipeline explains why identical model weights exhibit divergent performance across different cloud providers. Hardware generations, batching algorithms, quantisation levels, network latency, and memory interconnects directly impact throughput and time-to-first-token.

The pipeline also produces distinct operational failure modes:

• Requests rejected at the gateway due to rate limits or quota exhaustion.
• Latency timeouts occurring while waiting in provider batch queues.
• Execution failures within the accelerator hardware.
• Malformed JSON outputs that fail downstream schema validation.
• Valid tool calls directed to unavailable third-party APIs.
• Outputs that complete technically but fail business acceptance tests.

Classifying all these failures as generic "model errors" obscures where remediation must take place.

Matching capacity models to workload requirements

Commercial selection depends on the operational profile of the task:

Workload profile	Recommended deployment model	Critical performance metrics
Exploratory pilot	Shared direct provider or cloud API	First-pass acceptance rate, total token cost, rate-limit occurrences
High-volume data extraction	Smaller specialized model via batch API or reserved throughput	Throughput volume, schema validation pass rate, queue wait times
Interactive user support	Low-latency shared endpoint or provisioned throughput	Time-to-first-token, end-to-end response time, human escalation rate
Long-running autonomous agent	Frontier reasoning model with stateful orchestration and fallback	Task completion duration, context growth rate, tool error frequency
Regulated corporate workload	Dedicated VPC or regional deployment with strict data isolation	Auditable processing regions, zero-retention logs, IAM security policies
Air-gapped or edge deployment	Quantized open-weight model deployed on local hardware	Memory footprint, power consumption, local hardware compatibility

Upstream constraints: physical compute and power

Model inference relies on physical hardware supply chains: specialized accelerators, high-bandwidth memory (HBM), networking switches, liquid cooling, and electrical grid capacity. Model providers abstract this infrastructure behind REST endpoints, but they cannot escape its economic constraints.

Key hardware providers include NVIDIA, AMD, and custom silicon programs from hyperscalers (Google TPU, AWS Trainium). Data-center physical capacity and utility power interconnects determine where compute clusters can be constructed.

Australia's 2025 Electricity and Energy Sector Plan highlights official forecasts from the Australian Energy Market Operator (AEMO), projecting data-centre electricity consumption to reach approximately 12 TWh by 2029–30 (6 per cent of National Electricity Market grid demand) and 34 TWh by 2049–50 (12 per cent) (Australian Government, "Electricity and Energy Sector Plan 2025"). These projections underscore that digital inference infrastructure is increasingly constrained by national energy and physical utility planning.

The Australian infrastructure landscape

Australian enterprises access frontier model platforms via direct APIs and regional cloud availability zones. AWS operates regions in Sydney and Melbourne with three Availability Zones each (AWS Regions and Availability Zones). Google Cloud maintains infrastructure regions in Sydney and Melbourne, while Microsoft Azure operates Australia East, Southeast, Central, and Central 2 (Google Cloud locations; Azure global infrastructure).

Physical cloud regions confirm infrastructure presence, but they do not guarantee that every specific model or feature is hosted locally. Global API gateways frequently route inference requests to international data centers to balance server load. Partner model integrations, specialized safety classifiers, and backup storage can reside in different jurisdictions.

An Australian enterprise production deployment requires verifying eight operational parameters:

1. The exact model name, release date, and version identifier.
2. The specific cloud service and regional endpoint executing the request.
3. The designated geographic regions for active inference and data storage.
4. Whether dynamic cross-region load balancing is enabled by default.
5. Contractual terms governing prompt caching, data retention, and training exclusions.
6. Identity access controls, private VPC peering, and encryption keys.
7. Secondary fallback models and their associated data routing paths.
8. Retained audit logs documenting transaction costs, operational errors, and data deletion.

Evaluating model and inference evidence

Evaluating model infrastructure requires matching evidence types to the operational claims they support:

Evidence source	What it establishes	Unresolved operational question
Provider model card	Technical specifications, context window, and modality support	Performance and reliability on specific enterprise workflows
Published pricing schedule	Metering units and nominal token prices as of a checked date	Net fully loaded cost per accepted business deliverable
Cloud model catalog	API availability within a specific platform and region	Active enterprise customer volume, production SLAs, and quota limits
Academic benchmark	Relative performance across standardized, isolated test suites	Reliability when integrated with real enterprise tools and dirty data
Infrastructure documentation	Supported deployment architectures, batching modes, and VPC controls	Achieved latency, queueing delays, and error rates under peak load
Customer case study	A reported implementation by an early enterprise adopter	Commercial transferability, counterfactual labor costs, and long-term ROI
Task-level evaluation	Empirical pass rates against client-specific acceptance tests	Broader cross-industry performance and market adoption

Managing model lifecycles in production

Foundation model providers compete aggressively on reasoning benchmarks and nominal token costs. However, enterprise production systems differentiate on broader operational controls:

Governance domain	Operational specification
Task definition	The exact business deliverable produced, the intended recipient, and formal acceptance criteria
Risk profile	Failure modes, maximum financial liability, and regulatory reporting obligations
Modal requirements	Requirements for text, code execution, structured JSON, document vision, or audio
Tool boundaries	Explicit allow-lists for external APIs, read/write permissions, and human approval triggers
Data boundaries	Permitted data inputs, approved processing jurisdictions, and contractual retention limits
Latency targets	Required execution speed across interactive, near-real-time, and batch processing modes
Workload volume	Anticipated query volume, peak concurrent jobs, context size distribution, and throughput quotas
Lifecycle policies	Deprecation notice requirements, version pinning commitments, and migration schedules

For the reference job AE-JOB-001 (the competitor briefing), an enterprise evaluation requires tracking metrics across production runs:

configuration, eligible_jobs, delivered_jobs, accepted_jobs,
first_pass_accepted, severe_incidents, median_latency, p95_latency,
input_tokens, output_tokens, tool_cost, inference_cost, runtime_cost,
human_minutes, total_cost, cost_per_accepted_job

Production deployment principles

1. Strict governance boundaries: Hard data residency, privacy, or security breaches immediately eliminate a model configuration, regardless of benchmark reasoning performance.
2. First-pass acceptance: High first-pass success rates matter more than raw speed; re-prompting loops and human analyst correction quickly erode operating margins.
3. Unit economic cost: Net cost per accepted job is the controlling metric; low token pricing does not compensate for frequent execution failures.
4. Model portfolio architectures: Production systems increasingly route simple classification and extraction steps to smaller, low-cost models, reserving expensive frontier models for multi-step planning and final validation.
5. Fallback risks: Automated failover routes must be configured with identical data residency, security, and context-handling guarantees as primary endpoints.
6. Continuous migration: Frequent vendor deprecation schedules make regression testing and model cutover a routine operational process.

The model release control card

Before promoting an agent workflow into enterprise production, organizations establish a release record:

Control parameter	Governance requirement
Accountable owners	Designated business owner, lead technical architect, and enterprise risk authority
Configured route	Exact model version ID, cloud endpoint, regional boundary, and designated fallback path
Verification tests	Representative job evaluations, adversarial prompt testing, schema validation tests, and kill-switch trials
Operational thresholds	Minimum first-pass acceptance rates, latency ceilings, review time limits, and unit cost caps
Review cadence	Mandatory re-evaluation prior to release, following vendor updates, and on scheduled monitoring cadences
Audit evidence	Immutable evaluation datasets, scorecards, trace logs, and security sign-off records
Incident escalation	Formal operational procedures for halting execution, activating manual fallbacks, and notifying stakeholders

The trajectory of model inference

Model intelligence will continue advancing at the frontier, while inference delivery fragments across direct APIs, hyperscaler marketplaces, specialized hosting providers, and customer-managed clusters. As model weights become more interchangeable, switching costs will concentrate in the surrounding software harness: prompt engineering, context pipelines, tool integrations, and evaluation test suites.

Commercial pricing will increasingly shift up the value chain. While cloud providers continue billing for raw tokens and compute seconds, enterprise buyers will evaluate vendors on total cost per accepted business outcome.

A model can reason only across the context it is provided. Chapter 5 examines how agent systems assemble information: real-time context retrieval, persistent memory stores, and the data provenance required to make autonomous decisions defensible.

Sources and method

• Primary sources: Technical documentation, pricing schedules, regional infrastructure maps, and model lifecycle policies from OpenAI, Anthropic, Google, Mistral, Cohere, Microsoft, AWS, and NVIDIA checked as of 13 September 2026.
• Analytical frameworks: The Model-to-Outcome Cost Stack, the six-parameter model evaluation framework, and the Model Release Control Card are original analytical structures developed to evaluate inference procurement.
• Evidence boundaries: Provider documentation establishes published pricing, feature sets, and regional availability. Production performance, throughput stability, and unit economics require empirical validation on customer workloads.
• Infrastructure context: Australian grid electricity projections cited from the 2025 Electricity and Energy Sector Plan reflect AEMO forecasts for aggregate data-center power demand; agent-specific compute is not reported separately.
• Editorial record: Published by Agentic Economy, Perth, Western Australia. Updates and corrections are logged via the editorial record.

Back to the report hub · Next: context, memory and data.