top of page
Abstract Shapes

INSIDE

PUBLICATIONS

The AI Stack 2026: What Every Developer Needs

Updated: 3 days ago

The AI Stack 2026: What Every Developer Needs
The AI Stack 2026: What Every Developer Needs

UIT emblem

UIT University 365 Institute of Technology

Series AI Engineering

Duration 25 minutes

IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation


UNOP isochrone

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)

Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.

[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]

In this Lecture


Back to the TOC

The Stack You Actually Ship


You ship a feature that answers questions from your company's documents. It works in the demo. Three weeks later you cannot explain the bill, you cannot tell whether last Tuesday's prompt change made answers better or worse, and one user's runaway loop burned a month of budget in an afternoon.


None of those failures were model failures. They were architecture failures, and they cluster in the layers that sit around the model. The model is one layer of six. This lecture walks the whole stack, layer by layer, with the decision each layer forces and the symptom that appears when you skip it. By the end you will have a stack you can defend, a short list of what to build first, and the four levers that control almost all of your cost.


The seven layers at a glance


USER App and API layer: auth, rate limits, user experience Model gateway: routing, retries, caching, budgets, fallback Orchestration: prompt assembly, tool calls, agent loops Queues and workers: slow and bursty work off the request path Data: system of record, vector store, cache Observability and evaluation: traces, evals, cost dashboards Guardrails: budgets, content checks, tenancy scoping


Each layer has one job. The seams between them are where you place retries, budgets, and checks. A stack that respects the seams survives a model release; a stack that ignores them needs a rewrite every quarter.

Back to the TOC

Layer 1: The Model Gateway


The seven layers of the 2026 AI application stack, with guardrails spanning every layer and the role of each layer
The seven layers of the 2026 AI application stack, with guardrails spanning every layer and the role of each layer

The most expensive mistake in an AI application is coupling. A codebase that calls one vendor's SDK from fifty places cannot switch when that vendor deprecates a model, changes prices, or has a bad week of uptime.


Put a gateway in front. One layer routes requests to the right model, handles authentication and retries, enforces budgets, applies caching, and fails over to an alternate provider.


app code -> gateway.complete(request) -> provider


Swapping models becomes a configuration change rather than a project. Three responsibilities belong here and nowhere else:


  • Routing. Send the hard reasoning to a frontier model and the classification, extraction, and tagging to a small one. Most production traffic does not need a frontier model, and routing it there is the single largest avoidable cost.

  • Fallback. On a provider timeout or error, try the next provider before failing. Every serious provider has bad minutes.

  • Metering. Count tokens and cost per tenant per request. You cannot control a number you do not record at the point of use.


The symptom when you skip it


Your application code contains provider-specific parameters in twenty files, and a price change becomes a two-week task.

Back to the TOC

Layer 2: Orchestration


Orchestration is where the AI logic lives: prompt assembly, tool calls, agent loops, and multi-step workflows. Keep it explicit and testable, and keep it thin.


Workflows before agents, by default


This is the most consequential choice in the layer. A deterministic chain of model calls is predictable, debuggable, and cheap. A free-roaming agent is powerful and introduces variance and cost that you then have to contain.


Start with the simplest orchestration that solves the problem and add autonomy only where it earns its keep. Most production use cases need a workflow, not an agent: retrieve, assemble, call once, validate, return. Reach for an agent loop when the number of steps genuinely cannot be known in advance, such as a task that must search until it finds an answer.


Keep domain logic out


Do not let the orchestration framework grow to own your business logic. Teams that invert this end up unable to test or debug their AI behaviour without invoking the model, which turns every bug into an experiment and every experiment into a bill. The framework assembles context and calls tools. Your application decides what the rules are.


Tool interfaces as a stable boundary


If your agents call tools, define those tools behind a stable interface. A protocol boundary such as the Model Context Protocol lets the same tool serve multiple models and frameworks, and it keeps your integrations from being rewritten when you change the reasoning layer.

Back to the TOC

Layer 3: Queues and Workers


Where a failure shows up when a stack layer is skipped, with the symptom and the incident each omission causes
Where a failure shows up when a stack layer is skipped, with the symptom and the incident each omission causes

AI work is slow and bursty. A generation can take many seconds, traffic spikes, and provider rate limits throttle you. Doing that work inside a web request is a recipe for timeouts and a poor experience.


Push it to a queue:


request -> enqueue job -> 202 Accepted + job id worker pool -> respects provider rate limits, retries, writes result notify -> push the result when it is ready


Three rules make this layer work:


  • Enqueue, do not block. Accept the request, create a job, return a handle immediately.

  • Rate-limit at the worker. Concentrate provider rate-limit handling in the worker pool, so a traffic spike queues gracefully instead of erroring out.

  • Retry with backoff and a dead-letter queue. Transient provider errors are normal. Route persistent failures somewhere visible instead of losing them.


The symptom when you skip it


Your p95 latency is measured in tens of seconds, and a provider slowdown becomes an outage of your own product.

Back to the TOC

Layer 4: The Data Layer


Three distinct stores serve three distinct jobs, and conflating them causes pain that shows up months later.


System of record: a relational database. Users, jobs, results, conversations, audit, billing. This is the source of truth.


Vector store: embeddings for retrieval. Start with the vector extension of the database you already run. For workloads under roughly ten million vectors, that delivers single-digit-millisecond to low-tens-of-milliseconds p95 latency at a small monthly cost, and it keeps your embeddings in the same transaction as your documents and your permission rows. That last property is the real advantage: "only search what this user is allowed to see" becomes a WHERE clause instead of a distributed consistency problem. Move to a dedicated vector database past roughly fifty to one hundred million vectors, or when you need first-class hybrid search and heavy metadata filtering at scale.


Cache: prompt and response caching, rate-limit counters, session state. Caching is a first-class cost-control mechanism, not an optimization you add later. The stable part of your system prompt and your retrieved context are re-sent on every turn; caching them at the provider or in your own store stops you paying full price for the same tokens repeatedly.


Retrieval before fine-tuning


Feeding your data to the model at query time is cheaper to build, cheaper to maintain, and updates the moment your data changes. Fine-tuning changes behaviour, not knowledge, and it is the expensive option. Start with retrieval. Add fine-tuning only when retrieval cannot fix the behaviour you need.

Back to the TOC

Layer 5: Observability and Evaluation


You cannot operate a non-deterministic system you cannot see. Ordinary logging is not enough, because the failure you care about is a quality regression, not a stack trace.


Traces


Log every model call: prompt, model version, output, token counts, latency, and cost, correlated across the steps of one request. Without request-level correlation you can see that a call was slow but not which step caused it.


Evaluation


Build a fixed set of real inputs with expected outcomes, and run it in CI. Fifty representative cases will catch the regression when a provider silently updates a model behind the same name, and nothing else will. Prompt engineering without evaluations is guessing with extra steps.


Cost dashboards


Break cost down by feature and by tenant. Cost in an AI application is an architecture concern: it is driven by how much context you send, how many steps an agent takes, whether you cache, and which model handles which job. Bolting cost control on at the end is painful, because by then the decisions are structural.


Two policies you need before you log anything


Decide your PII and retention policy for prompts and outputs before the first trace lands in storage: prompts routinely contain customer data. And decide what happens when an evaluation fails, or the eval set becomes a dashboard nobody acts on.


The symptom when you skip it


You learn about quality problems from your users, and you have no way to tell whether last week's change helped.

Back to the TOC

Layer 6: Guardrails


Guardrails sit across the layers and enforce the limits you cannot trust application code to remember.


  • Per-user token and spend budgets, enforced server side. A hard daily cap per account. One scripted user can otherwise spend a month of budget in an afternoon.

  • Tenancy scoping. Every retrieval path must filter by the caller's permissions. In multi-tenant products this is a data-layer property, not a prompt instruction.

  • Input and output checks. Validate structured output against a schema before you act on it. A model that returns prose where you expected JSON will fail somewhere downstream, and you want that failure at the boundary.

  • Confirmation before irreversible actions. Anything that spends money, sends a message, or deletes a record goes through a human decision, or at minimum an idempotency key with an audit trail.


The symptom when you skip it


Your first incident is also your first guardrail.

Back to the TOC

The Four Cost Levers


The four cost levers of an AI application: model routing, context size, batching and step count, with the move for each
The four cost levers of an AI application: model routing, context size, batching and step count, with the move for each

Almost all of your spend is controlled by four decisions. Make them deliberately, and measure each one.


Lever

What it controls

The move

Model routing

Cost per call

Small model for classification, extraction and tagging; frontier model only for hard reasoning

Context size

Tokens per call

Retrieve fewer, better chunks; cache the stable system prompt and retrieved context

Batching

Unit price

Anything not user-facing goes through the batch path, which is typically a flat discount

Step count

Calls per request

Prefer a workflow to an agent; cap agent iterations explicitly


Three numbers to instrument from day one


Record cost per request, tokens per request broken down by feature, and steps per request. When a bill surprises you, one of those three will explain it, and you will find it in minutes instead of days.


Latency follows the same shape


A response that takes four seconds when it should take one is a churn problem. The fixes are the same levers: caching, routing easy calls to smaller models, and cutting context. Latency is an architecture outcome, not a hosting setting.

Back to the TOC

What to Build First: A Starter and a Scale Stack


Two stacks, and most builds migrate between them piece by piece rather than choosing once.


Layer

Starter stack

Scale stack

Infrastructure

Hosted model API plus a serverless app

Multi-provider routing; self-hosted inference where volume justifies it

Data

Relational database with a vector extension

Dedicated data platform plus a vector database for retrieval scale

Model

One hosted model

Routed multi-model: cheap tier for routine calls, frontier for hard ones, fine-tuned where it pays

Orchestration

Direct API calls through a thin gateway

Graph-based workflows with human checkpoints

Application

Standard web app

Multi-tenant, integrated with existing systems

Observability

Basic logging plus a trace tool

Full evaluation, tracing, drift detection, cost dashboards


The sequence that keeps cost and risk down


  • Start with the use case, not the tools. Define the one job the AI does and what "correct" means for it. The stack follows from that definition.

  • Prove the idea with one hosted model. One API, one prompt, no infrastructure. Confirm the model can do the job before you build around it.

  • Add retrieval when the model needs your data. This is where most products find their real value.

  • Add the evaluation layer before you scale. Wire in tracing and automated checks so you can trust the output as volume grows.

  • Move pieces in-house only when the numbers say so. Revisit self-hosting, a dedicated vector store, or fine-tuning once you have real usage data.


The test for any layer


Ask what happens to your application on the day that component is deprecated or triples in price. If the answer is "we change a configuration file", the layer is built correctly. If the answer is "we rewrite the feature", the layer is coupled, and that coupling is the actual defect.

Back to the TOC

The Application Layer: The Seam to the User


The diagram above names the app and API layer as the top of the stack. It is worth its own treatment, because most AI products quietly fail here, and a good demo and a good product differ in this layer and nowhere else.


What belongs in the application layer


  • The state machine of the interaction. Which states can the feature be in, what the user sees in each, and what happens when the model takes longer than the interface expects. An AI feature has at least four states before it has one: waiting, streaming, failed, and done. Design all four.

  • The contract with the layers below. The application never calls a provider directly, never assembles a prompt, and never decides which model handles a job. Those are gateway and orchestration concerns. The application sends a request and receives a result.

  • The honest empty states. What the interface shows when retrieval finds nothing, when a guardrail blocks the answer, or when the answer carries low confidence. A blank box is a bug report waiting to be filed.

  • The feedback capture. Every AI feature should be able to record a thumbs-down with the reason attached. This is the cheapest source of evaluation cases you will ever get, and it costs one button.


The two anti-patterns


The thin demo carried to production. A prototype usually has the model call inline in the request handler, the prompt as a string constant, and no state beyond loading and done. That shape reaches production on many teams and then fails on the first slow provider, the first failed parse, and the first user who reloads the page mid-generation. The failure is not the model; it is the missing seam.


The thick application that owns the AI logic. The other direction is just as common once the first incident lands. The application layer accumulates the routing rules, the retry policy, the prompt versions, and the budget checks, because those were the fastest things to add at the time. The stack then has six layers on paper and one in practice, and no layer can be changed without touching the product code.


The interface that makes the stack visible


A useful discipline: give the application exactly three affordances into the AI layers, and no more.


  • Submit a request and receive a handle immediately.

  • Read a status for that handle, including which step is running.

  • Receive the result, streamed or complete, with a reason attached when it is a refusal.


Anything the application needs beyond those three is a symptom that a lower layer is leaking upward.

Back to the TOC

The Observability You Actually Need: A Trace Walkthrough


Everyone agrees to log model calls. Very few teams can answer the question that matters during an incident: what happened to request 8f3c, step by step, and which step made it slow or wrong? This section is a concrete trace design you can implement in an afternoon.


The eight fields on every step


One trace record per step, not per request, with these fields and nothing else required:


Field

Example

Why it is on the list

trace_id

8f3c-...

Correlates every step of one user request

step_index

3

Where in the chain this step sits

step_kind

retrieve, model, tool, guardrail

Lets you aggregate by step type

model_or_tool

gpt-class-large-v3

Which component ran, by version

input_tokens / output_tokens

4,120 / 380

The only honest way to explain a bill

latency_ms

2,340

Splits queue time from work time

cost_usd

0.0142

Computed at the point of use, never later

outcome

ok, retried, refused, error

The field that turns logs into a graph


The single field most teams omit is step_index combined with trace_id. Without both, you can see that a request was slow but not which of its seven steps consumed the time, and the investigation restarts from scratch every time.


What one good dashboard looks at


Four panels, and each one answers a question you will actually ask:


  • Latency by step_kind, p50 and p95. This tells you whether retrieval or generation is your bottleneck. The answer changes as your corpus grows, so it is worth watching rather than measuring once.

  • Cost per request, by feature, over time. A step change in this panel is either a traffic change or a context change, and you will know which from the tokens panel.

  • Retry and error rate by model_or_tool. A provider having a bad day shows up here before it shows up in the user complaints.

  • Refusal rate by guardrail. A guardrail that refuses a rising share of traffic is either catching a real problem or has become too strict, and only this panel tells you that you have the question.


The eval set that pays for itself


Trace data tells you what happened. It does not tell you whether the answer was right. That is the evaluation set, and the fastest way to build one is to promote failures from the trace into it.


Take every thumbs-down, every refusal a user disputed, and every error, and add the input to a fixed set with the expected output written by a human. Fifty of these, run in CI on every prompt or model change, will catch the regression that a provider's silent model update causes. That is the failure mode that costs the most and is the hardest to notice, because nothing in your own code changed.


The eight fields of one trace step, and the four dashboard panels built from them, latency by step kind, cost per request, retry rate by component and refusal rate by guardrail
The eight fields of one trace step, and the four dashboard panels built from them, latency by step kind, cost per request, retry rate by component and refusal rate by guardrail


The two policies to settle before you store a single trace


Prompts routinely contain customer data, so decide the redaction rule and the retention window before the first trace lands on disk. And decide what a failed evaluation does: block the deploy, or raise a flag. An evaluation that nobody acts on becomes a dashboard within a month, and a dashboard is not a control.

Back to the TOC

Scaling the Stack: From Ten Users to Ten Thousand


The layers do not change as you grow. What changes is which one breaks first, and the order matters because it tells you what to fix next.


The four bottlenecks, in the order most teams meet them


1. Provider rate limits, at the hundreds of requests per minute. The first real ceiling is external. You handle it in the queue and the gateway: rate-limit the worker pool rather than the request path, and add a second provider behind the gateway so a limit is a routing decision rather than an outage.


2. Retrieval latency, at the millions of vectors. A vector search that took 12 milliseconds at 100,000 rows can take hundreds of milliseconds at 10 million rows. This is the point at which a vector extension in your existing database stops being the right answer and a dedicated store starts being one. Measure before you migrate, and remember that the per-tenant filter often costs more than the similarity search itself.


3. Context assembly cost, once caching is the only lever left. When routing is already at its cheapest model and every prompt is already trimmed, the remaining cost is the context you assemble. The step from here is retrieval quality, not retrieval volume: fewer, better chunks at the same token budget.


4. Coordination complexity, past roughly ten people on the system. The last bottleneck is not technical. At some size the prompt catalogue, the eval sets, and the model routing rules need an owner and a review path, because they are now shared artefacts with several authors. A prompt change that quietly improves one feature and degrades another is a coordination failure, and no amount of tracing fixes it.


Three scaling moves that are worth making early


  • Make the queue the only path. Once every model call goes through a queue, you can pause, drain, and reorder work during an incident. A direct call path has none of those options.

  • Version the prompt separately from the code. A prompt is a configuration artefact with its own release cadence. Keeping it in the code repository couples a copy change to a deploy.

  • Keep one tenant's data in one place. A per-tenant routing key from day one turns a future data-residency requirement from a migration into a configuration value.


The scaling test


For each layer, ask what its cost and latency look like at ten times your current volume, and which component breaks first. The layer whose answer you cannot produce is the one to instrument next. Most stacks scale the model path first and the retrieval and coordination paths last, and the order is usually backwards.

Back to the TOC

Choosing the Model Gateway in 2026


The gateway is the layer with the widest choice and the most vendor pressure, so it deserves a decision procedure rather than a preference.


The four shapes a gateway takes


Shape

What it is

Choose it when

Provider SDK, called directly

One vendor's client library in your app code

A prototype, with an explicit plan to replace it

Thin in-house wrapper

One module the app calls, which owns the provider call

One team, one or two providers, and you want to control retries yourself

Hosted gateway service

A third-party routing and metering layer in the request path

You need multi-provider failover and cost reporting before you have the people to build it

Self-hosted gateway

An open-source proxy you run and configure

Regulated data, an on-premise requirement, or fine-grained control over prompt logging


The middle two are where most production systems land. The decision between them is not about features: a hosted gateway can route, retry, and meter on day one, and a self-hosted one can too once you have configured it. The decision is about where your prompts and outputs are allowed to pass. A hosted gateway sees every request, so if your content cannot leave a particular boundary, that option is closed before you compare anything else.


The five capabilities to require


Whichever shape you pick, these five are the ones that pay for the layer:


  • Per-request metering with a tenant dimension. Cost per tenant per request, computed at the call. You cannot reconstruct it later from provider invoices, because they aggregate by key and not by your customer.

  • Failover across at least two providers or two models. A single-provider gateway is a configuration file, not a gateway. The point of the layer is that a provider having a bad hour stops being your outage.

  • Prompt and response caching. The stable part of a system prompt and a retrieved context are re-sent every turn. A cache at the gateway removes that duplicated spend without a change to your application.

  • A hard budget per key and per tenant. A daily ceiling enforced at the gateway, returning an error rather than spending the money. This is the guardrail that prevents the runaway loop from becoming an invoice.

  • Structured request logs you own. The ability to export the trace fields from the previous section into your own store, on your own retention schedule.


The costs people forget to price


  • An extra network hop. A hosted gateway in the request path adds tens of milliseconds. For a conversational product that is part of the latency budget you already published, so measure it as part of the total rather than separately.

  • A second failure domain. The gateway itself can be down, and then every provider behind it is unreachable. Design the failover so that a direct path exists for critical traffic, or accept that this is now a dependency you monitor.

  • A configuration surface that grows. Routing rules, budget rules, and cache rules are code. Keep them in version control with the rest of the system, or they will differ between environments in a way that is invisible until an incident.

  • An exit path. If the gateway is hosted, know before you adopt it how you would remove it. The whole reason for the layer is to avoid coupling, and coupling to the gateway instead is the same mistake one level up.


The test to apply


Ask what happens on the day the gateway is unavailable, and on the day its price triples. If either answer is "we cannot serve requests", the layer is not doing its job. The gateway exists so that the answer to both is "we change a configuration or a provider", which is the same test the stack applies to every other layer.

Back to the TOC

A Worked Cost Model You Can Copy


The four cost levers are easier to apply with numbers. Here is a worked example for one feature, with the arithmetic left visible so you can substitute your own.


The feature


A support assistant that answers a question from a document set of 2,000 pages. It retrieves five chunks, one chunk is about 400 tokens, and it produces a 350-token answer.


The naive build


Component

Tokens

Notes

System prompt and instructions

600

Sent on every request

Five retrieved chunks

2,000

5 x 400

User question and history

300


Output

350


Total per request

3,250

Before any retry


At a frontier-model price of roughly 3 dollars per million input tokens and 15 per million output tokens, that request costs about 0.0090 dollars for input and 0.0053 for output, so a little under 1.5 cents. At 50,000 requests a month that is about 715 dollars.


The three moves, priced


Move 1: route the easy half to a small model. Suppose 60 percent of requests are classification and extraction that a small model handles at a tenth of the price. Those 30,000 requests drop from about 1.43 cents to about 0.14 cents. Saving: roughly 385 dollars a month, from a routing rule and no prompt change.


Move 2: cache the prompt and the retrieved context. The system prompt is stable between requests, and a retrieval for the same question returns the same chunks. Caching the stable prefix at the provider typically bills it at a fraction of the input rate. If half of the input tokens become cached, the input line drops by roughly 40 percent. Saving on the remaining traffic: roughly 90 dollars a month.


Move 3: retrieve fewer, better chunks. Three chunks of 400 tokens instead of five removes 800 tokens per request, about 25 percent of the input. This is a retrieval-quality change, not a cost trick: if the answer quality holds, you have removed spend and latency together. Saving: roughly 120 dollars a month.


Where the model lands


The four moves together take the monthly figure from about 715 dollars to roughly 120 dollars, on the same traffic and the same feature. The routing rule accounts for about two thirds of it, and it is the one move that changes nothing about the product experience.


The three numbers to instrument first


Record cost per request, tokens per request broken down by feature, and steps per request. When a bill surprises you, one of the three will explain it, and you will find it in minutes rather than days. Those three numbers are also what makes the arithmetic above checkable rather than theoretical: substitute your own values for the price, the token counts, and the traffic mix, and you have your own model instead of mine.


The four cost levers applied to one support assistant feature, showing the monthly spend at each step from the naive build to the routed and cached build
The four cost levers applied to one support assistant feature, showing the monthly spend at each step from the naive build to the routed and cached build
Back to the TOC

Feynman Summary: Explain It Like You Are 12


Imagine you open a small restaurant. The recipe is not the hard part. What is hard is having a kitchen that works when forty people arrive at once.


You need someone at the door taking orders quickly, so nobody waits in a line (that is the gateway, it decides which cook handles what). You need a kitchen that follows a written recipe every time, rather than a cook who invents something (that is orchestration; a fixed recipe is easier to trust than free improvisation). You need a ticket rail where orders wait their turn, because you cannot cook everything instantly (that is the queue). You need three kinds of storage: the ledger with the real records, the pantry you reach into while cooking, and a shelf of pre-made sauces so you are not remaking the same base forty times (that is the database, the vector store, and the cache). You need someone tasting the food now and then to check it still tastes right (that is evaluation). And you need a rule that no single table can run up an unlimited tab (that is the guardrail).


Notice that none of those jobs is the recipe. The recipe is just one thing in the kitchen. Most AI products fail because of the kitchen, not the recipe.

Back to the TOC

Mindmap: The Complete Picture


Complete mindmap of the 2026 AI application stack: six layers, the decision each forces, the failure symptom when it is skipped, and the four cost levers
Complete mindmap of the 2026 AI application stack: six layers, the decision each forces, the failure symptom when it is skipped, and the four cost levers

The mindmap puts the request path down the centre and branches to each layer, the decision that layer forces, the symptom of skipping it, and the cost levers that cross all of them.



UNOP isochrone

UNOP Sound (University 365 Neuroscience Oriented Pedagogy)

Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.

[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]

Back to the TOC

Practical Exercise: Map One Request Through the Stack


This exercise takes about 20 minutes on paper, and it is the highest-value planning you can do before writing code.


Step 1: Pick one real feature


Choose a feature you have built or intend to build. Write one sentence describing the single job it does, and one sentence describing what a wrong answer would cost.


Step 2: Trace one request through the seven layers


For each layer, write what it does in your feature, or write "absent" and decide whether the absence is acceptable.


  • App and API: who can call it, and what limits apply per user

  • Gateway: which model, what fallback, where cost is metered

  • Orchestration: how many model calls, and whether the step count is known in advance

  • Queues: which work leaves the request path, and what the user sees while it runs

  • Data: system of record, retrieval source, what is cached

  • Observability: what you log, and the fifty cases you would test on

  • Guardrails: budget cap, tenancy filter, schema validation, what needs confirmation


Step 3: Name the seam


For each boundary between layers, write where the retry, budget, or check lives. A seam with no owner is where the incident will happen.


Step 4: Choose the four cost levers


For your feature, write the model routing rule, the retrieval size, whether anything can be batched, and the cap on steps. Estimate cost per request from those numbers.


What to look for


If you cannot finish Step 2 without inventing components, you have found your build order. If the step count in Layer 2 is unknowable, you have found where an agent is genuinely required, and everywhere else a workflow will do.


Applied AI connection


This is the review you run on any AI feature before it ships, and again after the first month of real traffic. The mapping takes twenty minutes and prevents the rewrite that takes a quarter.

Back to the TOC

Glossary


Term

Definition

Model gateway

A single layer that routes model requests, handles auth and retries, meters cost, and fails over between providers.

Orchestration

The code that assembles prompts, calls tools, and sequences model calls.

Workflow

A deterministic, pre-defined sequence of steps. Predictable and cheaper than an agent loop.

Agent loop

A model-driven loop in which the number of steps is decided at run time. Powerful, variable, and costly.

Queue

A store of pending jobs that moves slow work off the request path so the API returns immediately.

Worker

A process that consumes jobs from a queue, respecting provider rate limits and retrying transient failures.

Dead-letter queue

Where jobs go after their retries are exhausted, so failures stay visible instead of disappearing.

System of record

The relational database that holds users, jobs, results, and audit as the source of truth.

Vector store

A store of embeddings used for similarity search in retrieval.

pgvector

A Postgres extension that adds vector columns and similarity search to an existing relational database.

Prompt cache

A cache of the stable, repeated part of a prompt so you do not pay full price to resend it each turn.

Trace

The recorded path of one request across every model and tool call, with prompt, output, tokens, latency, and cost.

Evaluation set

A fixed set of real inputs with expected outcomes, run in CI to detect quality regressions.

Regression

A drop in answer quality after a change to a prompt, a model version, or retrieved data.

Guardrail

A server-side limit or check that constrains what a request can spend or do.

Token budget

A hard cap on tokens or spend per user or per day, enforced outside the model.

Tenancy scoping

Filtering every retrieval by the caller's permissions so one customer never sees another's data.

Idempotency key

A client-supplied identifier that lets a retried request be recognised instead of executed twice.

Batch API

A non-interactive submission path that trades latency for a lower unit price.

Fine-tuning

Additional training that changes a model's behaviour. It does not reliably add knowledge; retrieval does.

p95 latency

The response time below which 95 percent of requests complete. The number users actually feel.

Trace id

An identifier carried across every step of one user request, so a slow or wrong request can be followed step by step instead of investigated from scratch.

Step index

The position of one step inside a request's chain. Combined with the trace id, it is what makes latency attribution possible.

Prompt version

A prompt treated as a versioned configuration artefact with its own release cadence, rather than a string constant in the application code.

Failover

Routing a request to a second provider or model when the first fails or times out. The capability that makes a single vendor outage survivable.

Tenant dimension

The tenant identifier carried on every metered request and every retrieved chunk, so cost and permissions can be attributed per customer.

Data residency routing

Keeping one tenant's data and requests inside a chosen region by routing on a tenant key, decided at design time rather than retrofitted.

Back to the TOC

Quiz: TEST YOUR UNDERSTANDING


1. What is the primary reason to put a model gateway in front of your application?


A) It makes the model produce better answers


B) It gives one place for routing, retries, cost metering and provider fallback


C) It removes the need for a vector store


D) It reduces the model's token prices


2. When should you reach for an agent loop rather than a fixed workflow?


A) Whenever you want better quality


B) When the number of steps genuinely cannot be known in advance


C) Whenever the task involves retrieval


D) Always, because agents are more capable


3. Why push long AI work onto a queue instead of running it inside the web request?


A) Queues are cheaper to host


B) Generations take seconds and can spike, so blocking the request causes timeouts and poor experience


C) Queues remove the need for rate limiting


D) The model API rejects requests made from a web server


4. Which statement about retrieval and fine-tuning is correct?


A) Fine-tuning adds knowledge more cheaply than retrieval


B) Retrieval updates the moment your data changes; fine-tuning changes behaviour and is more expensive


C) They are interchangeable


D) Fine-tuning is required before retrieval can work


5. What is the single most valuable evaluation practice in this lecture?


A) Reading the model's release notes each month


B) A fixed set of real inputs with expected outcomes, run in CI on every prompt or model change


C) Asking the model to rate its own answers


D) Comparing vendor leaderboards



6. A request took 6 seconds instead of 1. Which two trace fields together make it possible to attribute that time to a step?


A) model_or_tool and cost_usd


B) trace_id and step_index


C) latency_ms and outcome


D) input_tokens and model_or_tool


7. A stack is at ten million vectors and retrieval latency has grown from 12 milliseconds to hundreds. What is the correct next move?


A) Increase the model size


B) Measure whether the per-tenant filter or the similarity search dominates, then consider a dedicated vector store


C) Cache every answer


D) Reduce the context window


8. Which capability makes a model gateway a gateway rather than a configuration file?


A) Prompt caching


B) A dashboard


C) Failover across at least two providers or models


D) A vendor SDK


9. In the worked cost model, which single move accounts for about two thirds of the saving?


A) Caching the stable prompt prefix


B) Retrieving three chunks instead of five


C) Routing the easy half of the traffic to a small model


D) Switching providers


10. Why must a prompt be versioned separately from the application code?


A) Prompts are longer than code


B) A prompt is a configuration artefact with its own release cadence, so keeping it in code couples a copy change to a deploy


C) Providers require it


D) The gateway cannot read it otherwise


Answers: 1-B, 2-B, 3-B, 4-B, 5-B, 6-B, 7-B, 8-C, 9-C, 10-B.

Back to the TOC

Related Resources


U365 INSIDE Publications



External Resources



Related U365 Lectures (Coming Soon)


  • Multimodal AI: When Models See and Hear, in the AI Frontiers series

  • Model Quantization: Running LLMs on Your Laptop, in the AI Engineering series

Back to the TOC

U.Copilot for This Lecture


Use this prompt with your own AI assistant to review the stack behind a feature you own, and to get a build order out of the result. It follows the UP-Context-Engineering method: context first, then the task, then the constraints, then the output shape.


CONTEXT I am reviewing the architecture of one AI feature I own. The feature is: [one sentence] The single job it does: [one sentence] What a wrong answer costs: [state the consequence] Expected volume: [requests per day] Current stack: [list what exists today, or write "nothing yet"] Current monthly AI spend: [number, or "unknown"] TASK 1. Trace one request through these seven layers and say for each whether it exists in my feature: app and API, model gateway, orchestration, queues and workers, data, observability and evaluation, guardrails. Mark each as present, partial, or absent. 2. For every layer marked absent or partial, state the failure symptom I should expect, in one sentence. 3. Give me a build order: the three changes with the highest ratio of risk removed to effort spent. 4. Name the four cost levers for my feature with a concrete setting for each: model routing rule, retrieval size, what can be batched, and a cap on steps per request. CONSTRAINTS Use only the information I gave you plus what is in this conversation. Do not invent benchmark numbers or prices. Where a number is needed and missing, say which measurement would produce it. OUTPUT A table with one row per layer: layer, status, evidence from my description, and the symptom if it stays absent. Then the build order as a numbered list of three items, one line each. Then the four cost lever settings as a short list. End with the first three things I should instrument.

Back to the TOC

Next Steps


  • Run the mapping exercise on one feature you own, and write the seam owner for every boundary between layers

  • Add the gateway before you add the second model. Routing without a gateway turns into a rewrite

  • Build the evaluation set now. Fifty real cases with expected outcomes, in CI, before you tune another prompt

  • Set a per-user budget today. It is a few hours of work and it removes your largest uncontrolled risk

  • Read the previous lecture in this series, "RAG vs Fine-Tuning: When to Use Each", if you have not settled your retrieval layer yet


The model is a commodity you rent. Your advantage lives in the layers around it: how well you route, how well you cache, how well you check, and how well you feed it. Build the kitchen, not just the recipe.

Back to the TOC

IMPORTANT NOTICE


This lecture is published by University 365 as part of its INSIDE Publications Hub. The content is free to read for all visitors. Lectures in this series may be part of a structured academic program leading to a Micro-Credential for your Career (MCC). To enroll in an academic program, visit university-365.com/tuition.


This content is for educational purposes. While we strive for accuracy, AI is a fast-moving field. Verify current technical details against primary sources for professional applications.


Copyright University 365, Inc. All rights reserved. This content is protected under University 365's copyright policies. For permissions or inquiries, contact uda@university-365.com.



Published by the Department of Academics, University 365.

Lecture delivered by the University 365 Institute of Technology (UIT).

Sam Utteker, Dean of Technology, UIT

Signed for the academic year 2026.

Comments

Rated 0 out of 5 stars.
No ratings yet

Add a rating
Image by Erik  Lucatero

Become Superhuman

Master AI to stay irreplaceable in every field.

 

 

 

​

​

Apply for Admission Today.
Select Your Initial Access Level.


Become a DISCOVERY, INSIDER, or SUPERHUMAN Fellow.

Image by Milad Fakurian

Master Your Life with a Digital Second Brain

Turn overwhelm into clarity with LIPS + CARE
U365’s unique framework to organize your goals, projects, and knowledge into a superhuman system for success

bottom of page