AI models and inference infrastructure for agents

On this page

Chapter 4 of The Hitchhiker's Guide to the Agentic Economy

Report hub · Previous: the agentic economy market map · In this chapter: provider map · cost · deployment · Australia · route comparison

Our view

The model is the reasoning engine inside many agents, but it is neither the whole agent nor a permanent architectural choice. An organisation buys a changing bundle of capability, latency, throughput, tool support, data handling and lifecycle risk. The correct model is the one that completes a defined job dependably at an acceptable total cost.

Model inference is the process of running a trained model against new input to produce output or select a next action. The surrounding inference service decides where that computation runs, how requests are admitted and billed, which versions remain available, and what operational evidence the customer receives.

Compute infrastructure is the hardware, networks, storage, power and serving software used to run that inference. It may be hidden behind an API or operated directly by the customer.

Key Takeaways

• Model selection is a job-level decision, not a permanent vendor ranking.
• Per-token price is one cost. Retries, tool calls, latency, review and failed jobs often matter more.
• Direct model APIs, multi-model clouds, dedicated deployments and self-hosting create different control and operating burdens.
• Open weights expand deployment choice, but they transfer evaluation, serving, patching and capacity work to the operator or hosting provider.
• An Australian cloud region does not automatically mean every model request stays in Australia; the model, endpoint and routing policy jointly determine the route.

This chapter uses provider documentation checked on 13 September 2026. Product names, prices, regions and lifecycle states can change. They are dated records, not enduring recommendations.

Sources and scope

Agentic Economy compiled the provider, price, region and deployment records from the primary sources cited in this chapter. The research method separates documented supply from independent evidence of use or business outcomes. Product records were checked on 13 September 2026; the stated limitation travels with every comparison. About Agentic Economy explains the publication and its Australian focus. Readers can submit source corrections through the contact page, where the editorial record can be checked before an update is made.

Four inference markets are developing

The model market is separating into direct provider APIs, multi-model cloud platforms, dedicated managed deployments and self-hosted inference. Each route sells a different combination of model capability, capacity, regional control, operational support and switching freedom.

At the same time, the boundaries are blurring. Model companies are adding tools, memory and agent runtimes. Clouds are offering first-party, partner and open models through one control plane. Serving companies are making open weights easier to operate. We expect model access to broaden while production differentiation moves towards routing, reliability, data boundaries and the economics of the complete workload.

What the model supplies

A model contributes the learned capability to interpret inputs, generate outputs and choose among possible steps. For agentic work, six characteristics matter most:

1. Task capability: can the model perform the reasoning, coding, extraction, vision, speech or language work required?
2. Tool behaviour: can it select tools, construct valid arguments, use the result and stop when the job is complete?
3. Instruction and format reliability: does it follow the operating policy and produce output another system can consume?
4. Context handling: can it work with the relevant documents and task history without losing the important constraints?
5. Latency and throughput: can the service return enough work within the customer's time boundary?
6. Operational boundary: are the model, region, retention terms, access controls and lifecycle suitable for the organisation?

The model does not decide these questions alone. A runtime may retry malformed calls. Retrieval can improve or poison context. A policy layer may block actions. Human review can catch errors while adding cost and delay. The Delegated Work Model treats the model as one contributor to a longer job.

Capability is conditional

Published benchmarks can compare models on a defined dataset and scoring method. An agent changes the test. Performance now depends on the prompt, tools, context, action space, stopping rule, runtime and consequences of error.

A model that scores well on a coding benchmark may still fail an internal change request because it cannot inspect the repository, loses state during a long task or calls the wrong deployment tool. A smaller model may be the better production choice for invoice extraction when the schema is fixed, the document types are known and exceptions go to a person.

The practical comparison unit is the delegated job. Comparable results hold tools, instructions and acceptance tests constant across the whole route.

Who supplies models and inference

The market contains model creators, multi-model platforms, serving software, hardware suppliers and infrastructure operators. Large companies span several roles. The records below are representative rather than a complete market census.

Model platforms

Provider	Current documented families or roles	Access model	Agent-relevant boundary
OpenAI	GPT-5.6 Sol, Terra and Luna; specialised realtime, computer-use, image and cyber models	Provider API and selected partner platforms	Current frontier families document function, web, file-search and computer-use tools, with capability and price varying by model
Anthropic	Claude Fable 5.1, Opus 5, Sonnet 5 and Haiku 4.5	Claude API, AWS, Google Cloud and Microsoft Foundry for supported models	The current line separates frontier, complex-agent, balanced and high-throughput roles; model IDs and lifecycle status need active management
Google	Gemini model families plus specialised realtime, media and robotics endpoints	Gemini API and Vertex AI	Model and tool availability varies by endpoint; search, Maps, code execution and other tools can have separate charges
Mistral AI	Mistral Large, Medium, Small and Ministral families; OCR and other specialised models	Mistral API, deployable products and partner clouds	Publishes broad price tiers and tool-use guidance; deployment and support differ between hosted and open-weight paths
Cohere	Command, North, Embed, Rerank, Parse and Transcribe families	Cohere platform, dedicated Model Vault and partner clouds	Separates generation from retrieval, ranking, document parsing and speech; those components can serve different steps in one agent job

The table describes current supply. It does not compare quality across jobs or prove production use. Each provider also changes its lineup over time. Anthropic's lifecycle page, for example, distinguishes active, legacy, deprecated and retired models and says publicly released models receive at least 60 days' notice before retirement (Anthropic, "Model deprecations"). Model migration is becoming normal operations rather than an exceptional project.

Multi-model platforms and serving layers

Layer	Representative participants	Supplied capability	Unresolved customer layer
Managed multi-model cloud	Amazon Bedrock, Microsoft Foundry, Google Vertex AI Model Garden	A catalogue, access control, billing and hosted inference across first-party and partner models	Exact model, region, endpoint type, data path, quotas and migration plan
Provider-hosted API	OpenAI, Anthropic, Google, Mistral, Cohere and other model creators	Fast access to the provider's current models and native tools	Vendor concentration, retention terms, regional routing, fallback and exit plan
Dedicated managed inference	Cloud-managed compute, Cohere Model Vault, NVIDIA NIM Certified and specialist inference providers	Isolated or reserved capacity with more predictable control and performance	Capacity commitment, utilisation, patching boundary and total infrastructure cost
Open serving software	Hugging Face Transformers/TGI, vLLM, SGLang and related projects	Software to expose compatible model endpoints on chosen hardware	Model licence, weights, GPUs, scaling, monitoring, security and support
Packaged inference runtime	NVIDIA NIM and cloud-specific containers	Validated model and runtime combinations, health endpoints and deployment tooling	Supported hardware, release track, model coverage and enterprise support terms

Microsoft's current Foundry documentation illustrates the commercial split. Its serverless API can charge by tokens or provisioned throughput and offers regional, data-zone or global processing. Managed compute runs selected open and partner models on dedicated accelerators and bills by accelerator time; the current managed-compute product is in preview and carries no production service-level agreement (Microsoft, "Deployment overview for Microsoft Foundry Models").

NVIDIA makes a similar distinction between rapid model availability and production lifecycle support. Its current NIM documentation separates functionally validated releases from NIM Certified releases with broader compatibility, security updates and enterprise support (NVIDIA, "NIM offerings"). The product being bought is not only inference speed. It is an operating and support boundary.

Tokens are not the unit of business value

Token prices are useful for estimating one part of a workload. They are poor substitutes for cost per accepted job.

The spread in current list prices is wide. On 13 September 2026, OpenAI listed GPT-5.6 Luna at US$0.20 per million input tokens and US$1.20 per million output tokens, while GPT-5.6 Sol was US$4 and US$20 respectively (OpenAI model documentation). Anthropic listed Claude Haiku 4.5 at US$1 and US$5, Claude Sonnet 5 at US$2 and US$10, and Claude Fable 5.1 at US$10 and US$50 (Anthropic model overview). Mistral listed Mistral Small 4 at US$0.15 and US$0.60, and Mistral Large 3 at US$0.50 and US$1.50 (Mistral pricing).

Those figures are not a quality ranking. Tokenisers, context limits, output behaviour, tools, caching, batch rates and service terms differ. A more expensive model may cost less per accepted job if it completes the task in fewer turns, needs fewer retries or reduces review. That is a testable hypothesis, not a result established by the price table. A cheap model can be an excellent router, extractor or verifier without being the right model for every step.

Agentic Economy's Model-to-Outcome Cost Stack

Version 1.0 of the cost stack measures the whole job:

model input and output
+ built-in and external tools
+ retrieval and data
+ runtime and hosting
+ retries and fallback models
+ human review and correction
+ incident and support cost
= total cost of delivered work

total cost of delivered work / accepted jobs = cost per accepted job

The denominator matters. Ten model calls might support one customer job. A retry remains attached to that job. A failed attempt that consumes tokens and analyst time belongs in its cost even if the final result is rejected.

Price the workload shape

Before comparing providers, estimate:

• typical and worst-case input size;
• expected output and reasoning usage;
• cacheable instructions or context;
• number and price of tool calls;
• average turns and retries;
• proportion routed to a larger model;
• peak requests, concurrency and latency target;
• review minutes and exception rate.

Google's pricing page shows why this wider view matters. It publishes separate modes for standard, batch, flex and priority processing, and separate charges or allowances for tools such as Search and Maps (Google, "Gemini Developer API pricing"). The cheapest token price can be attached to a mode that does not meet the job's response-time or availability requirement.

Four ways to buy inference

The market offers four main operating models. Organisations often use more than one.

1. Direct provider API

A direct API is the fastest route to a provider's current models and native features. The provider operates the service and usually charges for tokens, media, tools or reserved capacity.

This route suits teams that want rapid access and can accept the provider's account, data and regional terms. The organisation still needs usage controls, evaluation, fallback and version migration. A provider alias can simplify upgrades, while a dated model ID can improve reproducibility. The choice depends on whether stability or automatic access to the newest model matters more for that job.

2. Multi-model cloud platform

A cloud platform places several model providers behind the customer's existing cloud account, identity, network and billing controls. It can reduce procurement friction and make models easier to compare inside one operating environment.

The abstraction is incomplete. Models differ by region, API, feature, pricing and lifecycle. Amazon Bedrock explicitly documents model-by-model API and regional compatibility. Its cross-region inference profiles can route requests across several destinations; AWS advises customers with residency constraints to inspect every destination in the chosen profile (AWS, "Supported Regions and models for inference profiles").

3. Dedicated managed deployment

A dedicated deployment gives one customer reserved or isolated capacity without requiring the customer to run the full inference stack. It can support predictable throughput, private networking and stronger workload isolation.

The commercial risk is utilisation. Reserved accelerators or provisioned throughput cost money while idle. Their premium sits against queueing, rate-limit and noisy-neighbour costs on shared infrastructure. A dedicated endpoint also leaves application-level data access, agent permissions and outcome monitoring with the customer.

4. Self-hosted or customer-managed inference

Open or licensed model weights can run in a customer's cloud, data centre or edge environment. Serving systems handle batching, sharding, streaming, quantisation and compatibility with application APIs. Hugging Face describes its lightweight transformers serve route as suitable for evaluation and moderate load, while directing large production deployments towards engines such as vLLM or SGLang (Hugging Face, "Serve CLI").

Self-hosting can improve control over weights, runtime and data location. It also makes the operator responsible for hardware capacity, model licensing, serving performance, security patches, observability and upgrades. Open weights does not mean zero cost, unrestricted licensing or operational independence.

The inference path is a queue, not a single calculation

A production inference request passes through several controls before and after the model runs:

job arrives
  -> authenticate and apply quota
  -> choose model, version and region
  -> assemble and cache context
  -> queue and batch work
  -> run on accelerators
  -> stream or return output
  -> validate tool call or schema
  -> record usage, latency and errors
  -> retry, escalate or accept

This path explains why the same model can perform differently across providers and deployment modes. Hardware, quantisation, serving engine, batching, load and network distance affect latency and throughput. Tool schemas and runtime validation affect whether an output becomes a valid action.

The path also creates separate failure states. A request can be rejected by quota, time out in a queue, fail during inference, produce invalid structured output, call an unavailable tool or complete technically while failing the customer's acceptance test. Counting all of these as “model errors” hides the layer that needs fixing.

Capacity changes the model decision

Shared, on-demand inference converts infrastructure into a variable cost. Provisioned throughput or dedicated GPUs convert part of it into a capacity commitment. Batch processing can lower price where the deadline is flexible. Priority modes can buy lower queueing risk for time-sensitive work.

The commercial choice follows the workload:

Workload	Likely starting mode	Evidence to collect
Irregular pilot	Shared provider or cloud API	Accepted-job rate, token and tool cost, rate-limit events
High-volume extraction	Smaller model, batch or reserved throughput	Throughput, schema-valid rate, cost and queue delay
Interactive support	Low-latency shared or provisioned endpoint	Time to first token, total response time, escalation rate
Long-running agent	Capable model with durable runtime and fallback	Completion time, retries, context growth, tool failures, review
Sensitive internal workload	Regional or dedicated deployment after a data-flow review	Actual processing regions, retention, access, logs and incident path
Edge or disconnected system	Deployable model sized for local hardware	Quality loss, update process, power, memory and safety controls

External constraint: compute extends below the cloud

Inference depends on accelerators, networking, storage, power, cooling and data-centre capacity. Model providers can hide that machinery from an API customer, but they cannot remove its economics.

Hardware suppliers include NVIDIA, AMD, Google and cloud providers with their own accelerator families. Serving stacks decide how efficiently models use that hardware. Data-centre and energy operators determine where capacity can be built and connected. These markets affect model availability, reserved-capacity prices and which organisations can self-host.

No public source in this research supports separating “agent compute” cleanly from other AI or data-centre workloads. Data-centre growth therefore does not establish agent adoption.

Australia's 2025 Electricity and Energy Sector Plan cites AEMO forecasts of data-centre electricity consumption reaching about 12 TWh by 2029-30, or 6 per cent of grid-supplied electricity in the National Electricity Market, and about 34 TWh by 2049-50, or 12 per cent (Australian Government, "Electricity and Energy Sector Plan 2025"). These are data-centre forecasts under uncertainty. They show that digital infrastructure is becoming material to energy planning; they do not measure AI-agent demand.

The Australian infrastructure question

Australian organisations can access global model platforms through direct APIs and local cloud infrastructure. AWS documents Sydney and Melbourne regions with three Availability Zones each (AWS Regions and Availability Zones). Google Cloud identifies Sydney and Melbourne infrastructure regions, and Azure lists Australia East, Southeast, Central and Central 2 among its Australian regions (Google Cloud locations; Azure global infrastructure).

Those facts establish infrastructure presence. They do not establish that every model, endpoint or processing mode is available locally. A global endpoint may route a request outside Australia. A regional cloud project may call a partner model served elsewhere. Logging, safety review, backups and support records can have separate locations.

An Australian production route is defined by:

1. the exact model and version;
2. the product and endpoint used to access it;
3. the request-processing and storage regions;
4. whether routing can change under load;
5. retention and provider-access terms;
6. identity, network and encryption controls;
7. the fallback model and its data path;
8. the evidence supplied for usage, incidents and deletion.

Data location is one design input rather than a complete risk decision. The job's data sensitivity, professional obligations, customer promises, recovery needs and benefit of the system still need to be weighed together. See control and capability for the narrower authority question.

What the sources establish

The model and inference market is easy to observe at the supply layer and harder to compare at the outcome layer.

Evidence	What it establishes	Missing commercial evidence
Provider model card	Documented capability, context, modalities and limits	Performance on the buyer's job
Price page	Published charging basis at a checked date	Total cost per accepted job
Model catalogue	Availability through one platform and region	Eligible access, quota and production use
Benchmark or system card	Performance under a named method	Reliability in the buyer's tools and workflow
Deployment documentation	Supported serving and control options	Achieved latency, capacity and incident record
Customer case	A reported implementation and outcome	Transferability, counterfactual and complete costs
Job-level evaluation	Performance against the buyer's acceptance test	Wider market adoption

The strongest current claim is narrower: documented model and inference supply is broad. Public documentation can show prices, tool support, deployment paths and regions. It does not prove that an Australian buyer is eligible for a specific endpoint, that the route satisfies its data and support terms, or that lower inference prices create profitable agent services.

How the market differentiates inference routes

Model providers compete on benchmark capability and unit price. Production routes separate on a broader set of job-level variables. Comparable evaluation requires the same representative work across each configuration; twenty jobs or four weeks is a practical observation window, not a population-level statistical threshold.

The production boundary

Field	Decision
Job and customer	What accepted result is being produced, and for whom?
Consequence	What can go wrong, and what is the maximum tolerable loss?
Required modalities	Text, image, audio, video, code or structured data
Tool boundary	Which tools can the model select, and which actions need approval?
Data boundary	Which information enters the model, where may it be processed and retained?
Time boundary	Interactive, same-day, overnight batch or long-running
Volume	Typical and peak jobs, input size, output size and concurrency
Lifecycle	Required support period, version pinning and migration window

Observable outcomes

configuration, eligible_jobs, delivered_jobs, accepted_jobs,
first_pass_accepted, severe_incidents, median_latency, p95_latency,
input_tokens, output_tokens, tool_cost, inference_cost, runtime_cost,
human_minutes, total_cost, cost_per_accepted_job

For AE-JOB-001, a defensible comparison would contain one row per attempted brief and a summary by configuration, including the exact model version, route, fallback, source pack and analyst decision. No comparative result belongs in this report until those jobs have actually been run.

The production decision

• Hard security, data and incident boundaries eliminate otherwise capable configurations.
• First-pass acceptance and severe failure matter more than average quality.
• Cost per accepted job is commercially more informative than price per token.
• Smaller models increasingly handle routine steps while stronger models handle planning or exceptions.
• Fallback routes carry their own cost and data path.
• Model lifecycles make migration testing part of the production route.

A hard data or authority failure ends the comparison for that configuration. Severe incidents remain separate from average quality, and the prior process remains the fallback until the new route has survived failure, fallback and migration tests.

The result may be a portfolio rather than one winner: a small classifier, a specialist retrieval or ranking model, a capable planner, and a separate verifier. That is a legitimate architecture when each hand-off improves the accepted job enough to justify its complexity.

Commitment and exit economics

The full inference commitment spans low, expected and peak volume; currency movement; tax; minimum commitments; priority or batch charges; data transfer; idle capacity; support; internal operations; re-evaluation; and a migration reserve. Break-even volume matters because a lower unit price does not necessarily produce a lower total cost.

The production commitment includes the exact model and endpoint, eligible regions, routing and retention, quota, service level, support response, change notice, audit evidence, training use, deletion, export, termination and fallback path. These fields expose the real route behind the catalogue entry.

The model release control card

Field	Required record
Owners	Business owner, technical owner, control owners and risk-acceptance authority
Route	Exact model, version, endpoint, region, fallback and data path
Tests	Matched jobs, adversarial cases, tool and schema failures, fallback, rollback and kill switch
Thresholds	Acceptance, severe failure, latency, review burden and cost boundaries
Cadence	Before release, after material provider change and on a defined monitoring schedule
Evidence	Evaluation set, result rows, policy decisions, route logs, approvals and incident record
Escalation	Stop condition, containment owner, manual fallback and customer or regulator notification route

A model catalogue entry proves neither approval nor production fitness. A release decision needs evidence for the exact configured route.

Where we see the market going

Model capability will continue to concentrate at the frontier while inference delivery fragments across direct APIs, cloud platforms, specialist hosts and customer-managed deployments. The model itself will become easier to substitute than the evaluation, tool, policy and data work around it. Switching cost will increasingly sit outside the token price.

Pricing will also move up the stack. Tokens and reserved capacity remain supplier meters, but buyers will compare routes by accepted-job cost, latency under load, review effort and release risk. Providers that make model changes observable and portable will weaken lock-in; providers that bundle unique tools, context and operating controls will strengthen it.

Three market questions remain open:

1. Does model capability concentrate while inference delivery becomes interchangeable?
2. Do open serving standards reduce switching costs in practice, or do tools, prompts and managed features keep applications coupled to providers?
3. Does the economically important unit move from tokens and reserved capacity to accepted tasks and outcomes?

The economic conclusion would change if matched-job records showed that token price predicts total accepted-job cost after retries, review and incidents. The infrastructure conclusion would change if real switching records showed that prompts, tools, policies and managed features move cleanly between providers with little re-evaluation or migration work.

About the author and editorial record

Agentic Economy is the accountable publisher of this report. Joel Chan founded the publication in Perth to research the infrastructure, companies and operating choices shaping the agentic economy in Australia. This chapter is source-led market analysis, not a sponsored ranking or a claim that every listed product has been independently tested.

Back to the report hub · Next: context, memory and data.