Inference Economics: Tokens, Latency and the Cost of an Agent
UIT University 365 Institute of Technology
Series AI Economics
Duration 25 minutes
IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.
[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]
In this Lecture
Inference Economics: Tokens, Latency and the Cost of an Agent
The Invoice That Arrives After the Answer
A support team puts an agent into production in March. It reads the ticket, looks up the order, decides the remedy, drafts the reply and files the disposition. Everyone is pleased with it until the June platform invoice arrives at four times the pilot figure. Nobody doubled the traffic. Three things happened instead: the context grew, the model got chattier, and a retry loop nobody watches doubled the steps on a fifth of the tickets.
This lecture is about that number. The price per million tokens is public and takes a minute to read. The cost of finishing one piece of work is a different quantity, and it is the one your finance reviewer will ask about. By the end you can take a workload you actually run, count its tokens, price it on the November 2026 board, and defend the monthly line with arithmetic instead of adjectives.
Three ideas carry the argument. A price per token is a procurement number; a cost per completed task is an operating number, and only the second one belongs in a budget. Every step of an agent re-reads everything that came before it, so steps multiply money almost one for one. Latency and price are bought separately: the batch route halves the bill and takes up to a day, caching cuts the input bill and punishes gaps between calls.
Two companion lectures own the neighbouring ground. AI Cost Governance: Budgeting Tokens, Seats and Agents owns seats, caps, owners and the governance cadence. Reasoning Models in Practice: Test-Time Compute and When It Pays owns how much deliberation to buy. This lecture owns the meter.
Inference Economics: Tokens, Latency and the Cost of an Agent
What the Meter Counts
Every commercial model since 2023 sells the same unit: the token, a fragment of text roughly four characters of English long. You are billed per million of them, in two directions, at different prices. Input tokens are what you send. Output tokens are what the model writes. That asymmetry is the first thing to internalise, because a page of instructions costs a fraction of what a page of answer costs, and every design decision that changes the output length changes the bill more than the same decision on the input side.
The unit hides three complications. Text is not the only thing that becomes tokens: an image, a PDF page, a minute of audio and a tool result all arrive as token counts, so a workload you think of as "one document" can be 40,000 tokens before anyone writes a sentence. Long context is sold as capacity but behaves as spend, because you pay for it on every call that carries it. And the model's own deliberation is billed: Gemini 4's price page states that output price includes thinking tokens, and the other two vendors meter reasoning the same way. Thinking is the most expensive text in the building.
One accounting fact follows from this. The meter counts what you send, not what the model reads, so a 4,000-token policy manual sent on every call is bought on every call unless it is cached. The cache is the only place where repetition becomes cheaper instead of more expensive.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Five Lines of an Inference Invoice
An inference bill has five line items, and each one is set by a different decision. Read a real invoice and you will find all five, even when the vendor collapses them into a summary.
Line | What it charges | What sets it |
Input | Uncached prompt tokens | How much you send, every call |
Cache write | Storing a prefix for reuse | Prefix length, cache lifetime |
Cached read | Prompt tokens served from cache | How often the prefix repeats |
Output | Generated answer tokens | Answer length, format, stop rules |
Reasoning | Deliberation tokens, at the output rate | Effort level, task difficulty |
Two charges live outside the token meter and still belong in the budget. Cache storage is billed by time on some routes, so a cache you keep alive and never read is a slow leak. And the platform that runs the agent, the observability tool and the evaluation suite are fixed costs that do not move with tokens but do move the cost per task once the volume is small.
The five lines explain most invoice surprises. A bill that doubles while traffic is flat is usually output and reasoning: a model that thinks longer, a prompt that invites a longer answer, a format that asks for a table when a sentence would do. A bill that grows faster than traffic is usually input and cache write: longer conversations, more retrieved documents, a prefix that changes on every call and never hits the cache.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Price Board in November 2026
Prices below are per million tokens, verified on 14 November 2026 against the OpenAI pricing and models pages, the Anthropic models overview and the Google Gemini API pricing page. They are the same figures recorded in this month's Model Recruitment lecture, which keeps the ledger.
Band | Model | Input | Cache read | Output |
PhD | GPT-6 Astra | 10.00 | 1.00 | 50.00 |
PhD | Claude Fable 5.2 | 8.00 | 0.20 | 40.00 |
PhD | Gemini 4 Carbon | 2.00 | 0.10 | 10.00 |
Master | GPT-6.1 Sol | 2.00 | 0.10 | 10.00 |
Master | Claude Opus 5.5 | 4.00 | 0.20 | 20.00 |
Master | Gemini 4 Argon | 2.00 | 0.10 | 10.00 |
Bachelor | GPT-6.1 Terra | 1.00 | 0.05 | 5.00 |
Bachelor | Claude Sonnet 5.5 | 2.00 | 0.10 | 10.00 |
Bachelor | Gemini 4 Hydrogen | 1.50 | 0.075 | 7.50 |
Associate | GPT-6 Luna | 0.10 | 0.01 | 0.50 |
Associate | Claude Haiku 5.5 | 0.10 | 0.01 | 0.50 |
Associate | Gemini 3.8 Flash | 0.75 | 0.075 | 3.75 |
The range is one hundred to one between the cheapest input price and the most expensive: $0.10 against $10.00. No other number on this page matters as much, and none is as easy to get wrong, because the cheapest model is not interchangeable with the most expensive one. A band difference is a capability decision before it is a cost decision. Evaluating AI Systems is how you prove which band holds your task; this lecture assumes you have done it and takes the tokens as given.
Two models on this board carry dated introductory prices that double on 1 December. The last section of this lecture prices that change.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Five-to-One Rule and the Arithmetic of One Task
Read the output column against the input column and a pattern appears. On all twelve models above, output costs exactly five times input: $2.00 and $10.00 on GPT-6.1 Sol, $4.00 and $20.00 on Claude Opus 5.5, $10.00 and $50.00 on GPT-6 Astra. Cache reads sit at 5 to 10 percent of the input price. The board is therefore one scalar per model, and a task costs one multiplication:
cost = input price / 1,000,000 x (input tokens + 5 x output tokens)
The bracket is the useful quantity. Call it effective tokens: the input you sent, plus five times the output you asked for. A task's effective token count tells you what it costs on any model in one step, and the ratio between input and output inside that bracket tells you what kind of business you are in.
Workload | Input | Output | Effective tokens | Output share |
Chat turn | 500 | 800 | 4,500 | 89% |
Draft from a brief | 2,000 | 1,200 | 8,000 | 75% |
RAG answer | 20,000 | 600 | 23,000 | 13% |
One agent step | 15,000 | 250 | 16,250 | 8% |
Five-step agent task | 75,000 | 1,320 | 81,600 | 8% |
This table is the reason two teams with the same invoice have opposite problems. A chat product is an output business: at 89 percent of the bill, the lever is answer length, not retrieval. An agent is an input business: at 8 percent, shortening the answer is rounding, and the lever is how much context each step carries. Neither team should copy the other's optimisation list. Compute your own row before you choose.
Cache reads do not follow the five-to-one rule, which is the whole point of them: at 5 percent of input, a cached token is twenty times cheaper than a fresh one, and the arithmetic above is the uncached case.
Inference Economics: Tokens, Latency and the Cost of an Agent
A Worked Invoice: One Support Ticket, Twelve Thousand Tokens
Take one named task and price it exactly: resolve one inbound support ticket, meaning read the thread and the account record, decide the remedy, draft the reply. On GPT-6.1 Sol, at the prices on the board, sent as a single call.
The prompt is 12,000 tokens: a 4,000-token stable prefix (system policy, response rules, six worked examples) and 8,000 tokens of ticket and account history. The prefix is cached after the first call of the day, so it is billed at the cache-read rate. The answer is 800 tokens, of which roughly 250 are reasoning.
Component | Tokens | Rate per 1M | Cost |
Cached prefix | 4,000 | $0.10 | $0.0004 |
Ticket and history | 8,000 | $2.00 | $0.0160 |
Output, including reasoning | 800 | $10.00 | $0.0080 |
One call, total | 12,000 | $0.0244 |
Without the cache the same call is $0.0320, so caching the prefix saves 24 percent before you have changed anything else. The latency, on assumptions you will replace with your own measurements, is 8.0 seconds of prefill at 1,500 tokens per second and 8.9 seconds to write 800 tokens at 90 tokens per second: 16.9 seconds for the reader to see a complete answer, 8.3 of them before the first token appears.
Copy this table and replace the numbers. What makes it defensible is the structure: a rate per line, a token count per line, a multiplication per line, a total. A reviewer can check every row; a slogan cannot be checked at all.
Inference Economics: Tokens, Latency and the Cost of an Agent
Context Growth: Why Step Five Costs More Than Step One
The single call is not how the work gets done. An agent resolves the ticket in steps: classify, retrieve the order, decide the remedy, draft, file. That structure has a cost consequence that surprises almost everyone the first time they measure it, and it is not the number of calls. It is that a model has no memory between calls. Every step re-sends the whole conversation so far, so the prompt grows with the work.
Watch the input column. Step one sends the prefix and the ticket: 12,000 tokens. Step two adds the order record and the classification, and re-sends everything: 13,500. Step three adds the policy lookup: 15,000. Step four adds the decision: 16,500. Step five adds the draft: 18,000. Five steps, 75,000 input tokens, for a task whose information content is one 12,000-token prompt.
That is the agent multiplier, and it has three components worth naming separately, because each is managed differently. Steps multiply the bill: five steps, about five times the input. Fan-out multiplies it again: a step that calls four tools in parallel and pastes four results into the next prompt adds four times the payload, not one. And retries multiply it a third time, because a retried step re-sends the context that made it fail.
The practical consequence: when someone asks why the agent costs five times the demo, the answer is not that the model is expensive. It is that the agent read the ticket five times. Caching the prefix softens it, because the prefix is the part that repeats identically. Nothing softens re-reading the ticket, except sending a shorter ticket.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Same Ticket as a Five-Step Agent
Here is the same task, priced step by step on the same model and the same prices. Each step carries the 4,000-token cached prefix, and adds about 1,500 tokens of tool output to everything already in the prompt.
Step | What it does | Input | Output | Cost |
1 | Classify and route | 12,000 | 120 | $0.0176 |
2 | Retrieve order and policy | 13,500 | 150 | $0.0209 |
3 | Decide the remedy | 15,000 | 300 | $0.0254 |
4 | Draft the reply | 16,500 | 600 | $0.0314 |
5 | File the disposition | 18,000 | 150 | $0.0299 |
Five-step task, total | 75,000 | 1,320 | $0.1252 |
The multiplier is 5.13. Five steps, 5.1 times the money, and that is with the prefix cached on every step: without caching the same run is $0.1632, or 6.7 times the single call. The wall clock moves too, from 16.9 seconds to 66.5, which is the subject of the latency sections below.
Note what the table does not include, because the next section does. Six percent of these runs fail at step three, when the remedy decision comes back in the wrong schema. A failed run has already paid for three steps. The retry pays for all five again.
Two readings of this table are both correct and both useful. Per call, the agent is cheap: $0.025 on average, because most steps are short. Per task, it is five times the one-shot draft. Report the number that matches the decision in front of you, and always label which one it is.
Inference Economics: Tokens, Latency and the Cost of an Agent
Retries, Reflections and the Cost of Being Wrong
A failed run at step three has spent 40,500 input tokens and 570 output tokens before it stops: $0.0639. The retry then costs a full $0.1252, so the pair costs $0.1891 to produce one completed task. At a 6 percent failure rate, the expected cost per completed task is $0.1290 rather than the $0.1252 of a clean run.
Reflection is the same arithmetic with better manners. A self-check step that reviews the draft and rewrites it if needed is a sixth step: another 18,000 to 20,000 input tokens and a few hundred output tokens, roughly $0.03. Buy it when the cost of a wrong answer exceeds three cents of tokens plus the latency; skip it when a human reviews the output anyway. What you must not do is add reflection as a habit and then report cost per call, because the habit disappears from the average and reappears in the monthly total.
Retries also cluster. A prompt change that breaks the output schema does not fail 6 percent of runs evenly; it fails the same 6 percent every hour until someone notices. Alert on the retry rate itself, not on spend: the retry rate moves first.
Inference Economics: Tokens, Latency and the Cost of an Agent
Cost per Call Is Not Cost per Completed Task
Cost per call is a property of a request. Cost per completed task is a property of the system, and it is the only figure that survives contact with a budget. The difference is the failures.
Quantity | Value |
Clean run, five steps | $0.1252 |
Failed run at step three | $0.0639 |
Failure rate | 6% |
Expected cost per completed task | $0.1290 |
Naive attempt cost divided by success rate | $0.1332 |
The naive division overstates here by 2.7 percent, because a failed run is cheaper than a successful one: it stops early. It understates in the opposite case, which is common: a task that fails at the last step, or one whose failure triggers a human callback, costs more than the average run and the division hides it. Use the expected-cost form, and put the cost of a failure in the numerator even when it is not a token: the agent's time, the customer's second message, the credit you offer.
The formula you can defend in a meeting:
cost per completed task = (sum of all attempt costs + cost of failures) / completions
Scale it and the distinction stops being academic. At 12,000 tickets a month, the gap between $0.1252 and $0.1290 is $46, which is noise. Raise the failure rate to 20 percent, or price a failure at a $40 credit, and the same two numbers diverge by four figures. Budget the completed task, so the denominator has to be completions.
Inference Economics: Tokens, Latency and the Cost of an Agent
Latency Is a Separate Purchase Decision
Latency is not one number. Three quantities decide what a reader experiences, and each is bought differently.
Time to first token is queue wait plus prefill: how long the request sat in a queue, then how long the model took to read the prompt before writing. Tokens per second is the decode rate, and it sets how long the answer takes to finish once it starts. Total latency is both plus the round trips to your tools. A model can be quick to start and slow to finish, or the reverse, and the two failures feel completely different to a user.
You also buy throughput, which is not latency. Throughput is how many requests the system completes per second, and it is governed by concurrency: by Little's law, the number of requests in flight equals the arrival rate multiplied by the latency. To serve five requests a second at ten seconds each, you need fifty requests in flight. If your tier allows twenty, your real ceiling is two requests a second, whatever the token rate says. Queueing is where latency and throughput meet, and it is why a system that benchmarks well can still time out under load.
The purchase decision follows. An interactive surface needs a low time to first token, and streaming moves the first visible token earlier without changing the bill by one cent. An overnight pipeline needs throughput and does not care about latency, which is exactly the trade the batch route sells. A voice agent needs both, so it buys the small, fast band even when the large model would answer better.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Latency Budget You Can Measure
Write the budget down with the four inputs you can measure, then do the arithmetic once and keep it. The assumptions below are labelled as assumptions; replace them with your provider's numbers and the structure still holds.
Assume prefill runs at 1,500 tokens per second, decode at 90 tokens per second, and a tool round trip takes 0.6 seconds.
Path | Prefill | Decode | Tool time | Total |
Single call, 12,000 in, 800 out | 8.0 s | 8.9 s | 0 | 16.9 s |
Five-step agent, 75,000 in, 1,320 out | 50.0 s | 14.7 s | 1.8 s | 66.5 s |
The agent is 3.9 times slower and 5.1 times more expensive, and the two multipliers do not have the same shape. Money scales with total tokens. Time scales with total tokens divided by the concurrency you can hold, which is why an agent that feels fine in a demo collapses when ten users arrive: the per-request arithmetic did not change, the queue did.
Caching shows up on this table as well as on the invoice. A cached prefix does not need prefill, so the 20,000 cached tokens in the agent run stop costing seconds as well as dollars. On the assumptions above, moving 20,000 of 75,000 tokens out of prefill removes about 13 seconds of waiting across the five steps. Latency and cost usually move together; this is one of the few levers that improves both.
Instrument time to first token and total time per call from day one. Averages hide the tail, so keep the 95th percentile: the reader who waits four times the median is the reader who stops using the product.
Inference Economics: Tokens, Latency and the Cost of an Agent
Batch, Off-Peak and the Price of Waiting
Every large vendor sells the same second price for the same model: a batch route at 50 percent of the interactive price, on both input and output, in exchange for a completion window measured in hours rather than seconds. OpenAI's Batch API states a 24-hour completion window, Anthropic's Message Batches expire if not finished within 24 hours and usually finish within one, and Google's batch mode targets 24 hours. All three were read on 14 November 2026.
Fifty percent is the largest single discount on the board, and it is a throughput purchase, not a latency one. The same ticket at $0.1290 becomes $0.0645, so 12,000 tickets a month fall from $1,548 to $774. Nothing about the model changed. You agreed to wait.
Three conditions decide whether you can take it. The work must be deferrable: overnight classification, backfills, evaluation runs, bulk extraction, yes; a customer waiting in a chat window, no. The model must be eligible: eligibility is decided per model, not per vendor, so check the list before you build a pipeline around the discount. And the results arrive unordered, so every request needs your own identifier and your code must match on it.
The clock is a price on some routes too. DeepSeek publishes an off-peak rate at half the peak price on a published UTC window, so the same tokens cost half as much at the right hour. If your workload is deferrable, the schedule is part of the invoice; read the current window rather than trusting last quarter's figure.
Inference Economics: Tokens, Latency and the Cost of an Agent
Caching: The Only Discount That Rewards Repetition
Caching turns a repeated prefix into a cheaper token, and the write costs something, so the first question is how many times you will read it. On the November 2026 board, a cached read costs 5 to 10 percent of the input price: $0.10 against $2.00 on GPT-6.1 Sol, $0.20 against $8.00 on Claude Fable 5.2.
The write is where the routes differ. OpenAI charges no cache write fee, so a cached token pays from its first reuse. Anthropic charges a 5-minute write at 1.25 times input and a 1-hour write at 2 times input, which puts the break-even at two uses for the short cache and three for the long one. Google caches implicitly and also charges storage by time. Those three policies produce three different designs for the same workload.
The lifetime is the trap. A 5-minute cache read by a queue that runs every ten minutes never hits, so you pay the write every time and the discount never arrives. If your calls are more than five minutes apart, buy the 1-hour cache, or restructure so the prefix is read in a burst. Measure the hit rate before you claim the saving: a cache write with no read is a 25 percent surcharge on the prompt, which is the opposite of a saving.
What to cache is a design rule: the stable prefix only, meaning system policy, response rules, worked examples and the schema. Never cache what changes with the request, because a changing prefix invalidates everything after it. On the ticket agent, 20,000 of 75,000 input tokens are the cached prefix, and that 27 percent of tokens produces a 23 percent cut in the bill, $0.1252 instead of $0.1632, plus the prefill seconds noted above.
Inference Economics: Tokens, Latency and the Cost of an Agent
Where the Numbers Have to Come From
Every number in this lecture came from a rate card and a token count you can read from the response itself. That is the discipline. Estimates are how a budget becomes a surprise.
The providers return the breakdown. OpenAI reports prompt tokens, completion tokens, cached tokens and reasoning tokens in the usage object. Anthropic reports input tokens, output tokens, cache creation tokens and cache read tokens. Google reports usage metadata with prompt, candidate, cached and thought token counts. Check the field names against your SDK version, then log them per call. Four numbers plus a task identifier is enough.
Build the ledger per task, not per day. One row per step: task id, step, model, input, cached input, output, reasoning, tool calls, retry flag, start and end time, outcome. Roll it up nightly into cost per completed task, and keep the raw rows for a month. When the invoice disagrees with the ledger, the raw rows are how you find the untracked traffic: batch jobs, embeddings, evaluation runs and the free-text experiments someone ran on a Friday.
Reconcile to the invoice monthly. A gap of a few percent is normal, from rounding and from traffic you did not instrument. A gap of 20 percent means a workload is outside the ledger, and the ledger is the only thing that lets you attribute the bill to a product, a customer or a team. Without attribution, every optimisation argument ends in opinion. And a cost dashboard with a hard-coded rate from March is worse than no dashboard, because it reports a confident number that is wrong: read the rate from configuration, stamp it with a date, and fail the report when the stamp is stale.
Inference Economics: Tokens, Latency and the Cost of an Agent
Four Levers, Ranked by What They Save
Take the ticket agent at 12,000 completed tasks a month: $1,548 on GPT-6.1 Sol. Four levers move that number, and their sizes are not close. Each figure below is the monthly bill after the change, with everything else held constant.
Lever | Monthly bill | Saving | What it costs you |
Move down a band to GPT-6.1 Terra | $774 | 50% | capability, unless your evaluation says otherwise |
Trim the ticket from 8,000 to 3,000 tokens | $930 | 40% | some context the model no longer sees |
Cache the prefix (already in the base) | $1,548 | 23% versus no cache | a write fee and a lifetime to manage |
Cap output at 700 tokens per task | $1,472 | 5% | shorter replies, less explanation |
The order is the lesson. The model band is worth more than everything else combined, and it is the lever that must pass an evaluation before you pull it, which is why the evaluation lecture comes before this one. Context hygiene is second and costs nothing but discipline: sending the last five messages instead of twenty, and the account summary instead of the account dump, removes 40 percent of the bill without touching the model. Caching is third, and note that it is already baked into the base figure, so its real value is what you would lose without it. Output discipline is last at this workload shape, which is the exact opposite of the ranking a chat product would produce.
Raising effort is not on the list because it buys quality with money, deliberately, which is the subject of the reasoning lecture. Switching vendors changes the rate card, not the workload, and is worth doing only after the first three levers are pulled.
Inference Economics: Tokens, Latency and the Cost of an Agent
The Budget Line You Can Defend
A defensible line has four parts and fits on one page. The unit, the volume, the multiplier and the headroom.
The unit is the cost per completed task, computed as above, with the model and the token counts named. The volume is completions per month, from the business, not from the platform. The multiplier is what the unit does under load: retries at the measured rate, plus the price changes already announced. The headroom is 15 percent, which is not padding; it is the observed variance between a forecast and a month in which one prompt change doubled the retry rate for a week.
Then set three alerts, and set them on the right quantities. A per-task ceiling at three times the median catches the loop that never terminates, which is the failure mode that produces a five-figure invoice from a four-figure forecast. A daily spend alert at 110 percent of run rate catches the volume surprise. A retry-rate alert catches the quality regression before it reaches the invoice at all.
Write the sentence a reviewer can quote: this workload costs 13 cents a completed task on GPT-6.1 Sol at November 2026 prices, we run 12,000 a month, that is $1,548, and the same work on Terra is $774 if it passes the evaluation. Four numbers, one date, one alternative. That sentence survives scrutiny because every part of it can be checked, and it dies immediately if any part is a guess.
Inference Economics: Tokens, Latency and the Cost of an Agent
What Changes, and the Cliff That Never Moves
Two things move the numbers after this lecture is published, and one of them is already dated.
The dated one is the Gemini 4 introductory price. Carbon and Argon are $2.00 input and $10.00 output through 30 November 2026, then $4.00 and $20.00 from 1 December. The same workload on Gemini 4 Carbon costs $0.1290 a task in November and $0.2581 in December, so a monthly line of $1,548 becomes $3,097 with no change to traffic, prompts or model. Any budget written this month on those two models needs a December line, and a review date of 1 December.
The cliff that never moves is the prompt-length boundary. Anthropic's published batch table lists Claude Haiku 5.5 at $0.05 input and $0.25 output for prompts up to 100,000 tokens, and at $0.25 and $1.25 for prompts over that line. Batch is half the standard rate, so the standard list steps from $0.10 to $0.50 per million input at the boundary. A 99,000-token prompt costs $0.0099 of input. A 110,000-token prompt at the higher rate costs $0.0550, which is 5.6 times as much for 11 percent more text. Trim to just under the line and you buy the same answer for a fifth of the input price.
Both facts point at the same habit. Put the price review on a schedule, keep the rate card in configuration with a date stamp, and re-read the vendor page in the first week of every month. The monthly Model Recruitment lecture exists for exactly this reason: prices move between quarters, and a budget is a claim about a future that has a date on it.
Inference Economics: Tokens, Latency and the Cost of an Agent
Six Mistakes That Survive Every Finance Review
Budgeting tokens instead of tasks. A token budget cannot be spent or compared with a supplier's quote. Convert before the number leaves your team.
Averaging cost per call across models. The mean of $0.003 and $0.30 is meaningless when the models do different work. Report per task, per model.
Assuming every attempt succeeds. The failure rate is part of the unit cost and moves faster than any price.
Forgetting the retry loop. A loop that runs twice on 5 percent of tasks adds 5 percent to the bill, no feature to the product, and nothing to a per-call average.
Hard-coding a price. November prices are not December prices, and the two models with announced increases are the two most likely to sit in a pilot.
Counting the cache write and not the lifetime. A 5-minute cache behind a 10-minute queue is a surcharge, not a discount.
Each of these is arithmetic, which is why a review that asks for the four numbers above will surface all six.
Inference Economics: Tokens, Latency and the Cost of an Agent
Feynman Summary: Explain It Like You Are 12
Imagine a taxi that charges by the word. You pay for the address you give the driver, and five times as much for every word the driver says back. Now imagine the driver forgets everything between trips, so before each leg you have to repeat the whole conversation. One trip is cheap. The same trip in five legs costs about five times as much, because you paid to say the same things five times.
That is an agent: not one clever answer, but five ordinary ones, each carrying the whole story so far. Two things follow. You can hire a cheaper driver, the biggest saving of all, but only if that driver reaches the destination. And you can ask the driver to go in the evening, which halves the fare and delivers tomorrow.
Inference Economics: Tokens, Latency and the Cost of an Agent
Mindmap: The Complete Picture
The map puts the five invoice lines at the centre, because everything else feeds them. The left branch is arithmetic: tokens, rates, effective tokens, the cost of one task. The middle branch is structure: context growth, steps, fan-out, retries, and the gap between a call and a completed task. The right branch is purchase decisions: time to first token, throughput, batch, caching and dated price changes. The bottom branch is the four levers in the order they should be pulled.
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.
[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]
Inference Economics: Tokens, Latency and the Cost of an Agent
Practical Exercise: Price One Task in Ninety Minutes
Do this with a workload you actually run, not with the ticket agent above. You need read access to your provider's usage data and twenty real traces.
Name the task and its completion. One sentence: the system finishes X for Y, N times a month. If you cannot name the completion, you cannot price it.
Pull twenty traces and count tokens per step from the usage object: input, cached input, output, reasoning, plus steps, tool calls and retries.
Build the step table. One row per step, averaged over the twenty traces, at the rate for the model you actually use. Total the input and output columns and multiply by the November 2026 rates.
Compute both units. Cost per call is the mean step cost. Cost per completed task is all attempts plus failures, divided by completions. Write both down and label them.
Price three bands: the model you run today, one band down, two bands down, on the same token counts.
Rank your levers: model band, context hygiene, caching, output discipline, each with a monthly saving beside it.
Write the sentence. "This workload costs X per completed task, we run N a month, that is Y, and the same work on the cheaper band is Z if it passes the evaluation."
Pass condition: a colleague can check every line without asking you a question.
Inference Economics: Tokens, Latency and the Cost of an Agent
Glossary
Term | Definition |
Inference | Running a trained model to produce an answer, as distinct from training it. Inference is what the token meter charges for. |
Token | The unit a model reads and writes, about four characters of English text. Prices are quoted per million tokens. |
Effective tokens | Input tokens plus five times output tokens. On the November 2026 board this single number times the input price gives the cost of a task. |
Cache write | The charge for storing a prompt prefix so later calls can reuse it. Free on OpenAI, 1.25 times input for a 5-minute cache on Anthropic, 2 times for 1 hour. |
Cached read | A prompt token served from cache instead of being read fresh. Priced at 5 to 10 percent of the input rate on the November 2026 board. |
Prefill | The phase in which the model reads the prompt before it writes. Prompt length sets prefill time. |
Time to first token | Queue wait plus prefill: the delay before the first token of the answer appears. |
Decode | The phase in which the model writes output tokens, measured in tokens per second. |
Little's law | Requests in flight equals arrival rate multiplied by latency. It sets the throughput ceiling of a rate-limited tier. |
Batch route | A deferred route priced at half the interactive rate, with a completion window measured in hours rather than seconds. |
Agent multiplier | The growth in cost per task caused by every step re-sending the accumulated context. Five steps cost about five times one call. |
Retry overhead | The share of the bill paid to attempts that produced no usable result, plus the cost of the attempt that replaces them. |
Cost per completed task | All attempt costs plus the cost of failures, divided by completions. The only unit that belongs in a budget. |
Inference Economics: Tokens, Latency and the Cost of an Agent
Quiz: TEST YOUR UNDERSTANDING
1. On the November 2026 price board, output tokens cost how much relative to input tokens, on every one of the twelve models?
A) The same
B) Twice as much
C) Five times as much
D) Between ten and fifty times as much
2. A task sends 12,000 input tokens, of which 4,000 are cached, and produces 800 output tokens on GPT-6.1 Sol. What does the call cost?
A) $0.0080
B) $0.0244
C) $0.0320
D) $0.1252
3. Why does the same ticket cost about five times more as a five-step agent than as a single call?
A) The model charges more per token inside an agent
B) Every step re-sends the accumulated context
C) Tool calls are billed per invocation
D) Agents always need a bigger model
4. Which lever saves the most on the 12,000-ticket workload priced in this lecture?
A) Capping output at 700 tokens
B) Caching the stable prefix
C) Moving down one model band
D) Trimming the ticket from 8,000 to 3,000 tokens
5. What does the batch route actually buy?
A) A lower time to first token
B) Half price in exchange for a completion window of hours
C) A larger context window
D) Cheaper cache writes
Answers: 1-C, 2-B, 3-B, 4-C, 5-B
Related Resources
U365 INSIDE Publications
External Resources
Related U365 Lectures (Coming Soon)
AI Cost Governance: Budgeting Tokens, Seats and Agents, Business AI Series, current cycle
Reasoning Models in Practice: Test-Time Compute and When It Pays, AI Foundations, current cycle
Inference Economics: Tokens, Latency and the Cost of an Agent
U.Copilot for This Lecture
You are a cost analyst for an AI workload. I will paste a table of steps with columns: step, model, input tokens, cached input tokens, output tokens, reasoning tokens. Do this, in order: 1. Restate the rates you use per million tokens, for the model I name, and the date of the rate card. 2. Compute the cost of each step, showing the input, cached and output components separately. 3. Total the task and give me the cost per completed task at the failure rate I supply. 4. Tell me which of these four levers would save most on this workload, with the monthly figure for each: model band, context length, caching, output length. 5. Name every assumption you had to make, and refuse to invent any rate or token count I did not give you.
Inference Economics: Tokens, Latency and the Cost of an Agent
Next Steps
Pick one workload you run weekly and pull twenty real traces with their usage fields.
Build the step table: input, cached input, output and reasoning per step, plus tool calls and retries.
Compute cost per completed task, not cost per call, and write the figure down with the date of the rate card.
Price the same token counts on two cheaper bands and send the cheapest passing band to your evaluation set.
Trim the context before you buy more of it: fewer messages, a summary instead of a dump, shorter tool results.
Set three alerts: per-task cost at three times the median, daily spend at 110 percent of run rate, and the retry rate.
Put the rate-card review on the first working day of each month, and check the batch and cache pages at the same time.
Inference Economics: Tokens, Latency and the Cost of an Agent
IMPORTANT NOTICE
Copyright University 365, Inc. All rights reserved. This lecture is provided for educational purposes and does not constitute legal, financial or professional advice. For enrollment, visit university-365.com/tuition. For permissions or inquiries, contact uda@university-365.com.
Published by the Department of Academics, University 365.
Lecture delivered by the University 365 Institute of Technology (UIT).
Sam Utteker, Dean of Technology, UIT
Signed for the academic year 2026.
Comments