The Six-Layer AI Stack Every Engineer Needs to Understand

The model is just one layer. Everything around it decides whether your AI system works in production.

That sentence is the whole article. The rest is a map you can use to find where a system is actually weak, because it is rarely the model.

---

Why the AI stack needs a map

In the 1980s, networking was a mess. Every vendor had its own protocol and its own hardware, and none of it talked to anything else. The OSI model did not build the internet. It gave engineers a shared structure: seven layers, each with one clear job.

AI has no official equivalent. But if you look at how production systems are built in 2026, the same layers keep showing up. Different people draw the lines differently, and that is fine. What matters is the job each layer does.

Here is the map I use: six layers, bottom to top, plus two concerns that run through all of them.

---

One example: an AI coding assistant for your team

Say you are building a coding assistant for thousands of engineers. The first instinct is simple: call a frontier model and let it write code.

That breaks quickly, and each failure points to a different layer:

• Every model has a knowledge cutoff. It needs to see your real codebase and internal docs. That is data and retrieval.
• The work has to be split into steps: understand the code, write the change, run the tests, review the result. That is orchestration and agents.
• The model has to run somewhere at a cost you can afford. That is infrastructure and serving.
• Engineers want it in their editor, not a separate chat window. That is the application layer.

And then the problem that hurts platform teams most: the same assistant writes different code on different machines, then behaves differently again in CI. Hold onto that. It comes back in the reproducibility section.

---

Layer 1: Compute and infrastructure

AI models need specialized hardware, mainly GPUs, because they run a huge number of calculations in parallel.

Nvidia dominates, and not only because of the chips. CUDA, the software ecosystem around them, is what much of modern AI already depends on. The layer is diversifying, though. Google has TPUs, Amazon has Trainium, and more cloud providers are building their own accelerators to cut cost and reduce reliance on one vendor.

The economics matter more than the hardware:

You train a model occasionally. You serve it on every single request.

Once a product has enough users, serving can become the bigger long-term cost. You have three places to run it:

One detail to remember: every setup depends on a specific mix of drivers, runtimes, and accelerator libraries. Two engineers on one team can be on different CUDA versions without knowing it.

---

Layer 2: Model training and development

Compute gives you capacity. The next question is which model.

Foundation models split into two camps:

• Proprietary frontier models from OpenAI, Anthropic, and Google (GPT, Claude, Gemini). Most developers use them through APIs.
• Open-weight models, which have become genuinely competitive, especially for coding and reasoning. Which lab leads changes every few months, so do not hard-code a favorite.

The trade-off is no longer "free versus paid." With open weights you avoid the per-token API charge, but you still pay for GPUs, hosting, maintenance, and the engineers who run it. A hosted model removes most of that work and gives you strong capability on day one.

The shift I would watch most is toward smaller models. Fine-tune a small model for one narrow, well-defined task and it can beat a much larger general model on that task, while running faster and costing less.

So choosing the right model for each step of a workflow is now an engineering skill of its own, not a one-time procurement decision.

---

Layer 3: Inference and serving

A trained model is a file on a disk. To become a product, it has to answer real users reliably, at scale. This is where cost accumulates, because serving happens on every request.

Three things matter here.

1. Optimization

Quantization stores the model's weights at lower precision. The model gets smaller, faster, and cheaper to run, usually with a modest quality trade-off you should measure instead of assume.

2. The serving engine

This is the software that receives requests and runs the model on GPUs. Tools like vLLM and TensorRT-LLM help a GPU handle more requests at once and use memory more efficiently.

They also support techniques like speculative decoding: a small, fast model guesses several tokens ahead, and the large model verifies those guesses together. When the guesses are right, you get several tokens for roughly the cost of one step.

3. Routing

Easy requests go to a small, cheap model. Hard ones go to the capable, expensive one. A router that decides well can cut the inference bill significantly.

If you call a hosted API, the provider handles most of this and you pay for it inside the token price. If you host the model yourself, this is where the engineering work goes. Either way, how you serve a model matters almost as much as which model you choose.

---

Layer 4: Data, retrieval, and protocols

Fast serving does not help if the model is working with the wrong information. A model does not know your company's data, and it does not know what happened last week. This layer is about giving it the right context at the right time.

Retrieval is becoming context engineering

The basic idea of RAG is simple. Turn documents into embeddings, find the pieces most relevant to the question, and hand only those to the model.

Two things are changing:

1. You may not need a separate vector database on day one. Pinecone and Qdrant make sense at large scale, but many teams start with pgvector inside Postgres and move only when they hit a real bottleneck.
2. Plain RAG is turning into context engineering. You combine retrieval with memory, hybrid search, reranking, and better filtering, so the model gets less context, but more useful context.

Protocols: real convergence

MCP (Model Context Protocol) is becoming the common way for AI applications to connect to tools and data. Think USB-C for AI. It started at Anthropic and was donated to the Linux Foundation's Agentic AI Foundation in December 2025, and it is supported across most major AI products.

A2A (Agent2Agent) started at Google and targets a different job: agents talking to other agents.

Standardization has a cost. If a widely used MCP server or SDK has a vulnerability, every application built on it can be exposed. Agent systems have already had serious security issues, and OWASP now publishes a whole Top 10 for Agentic Applications. Keep that in mind. It is the reason governance exists as its own concern.

---

Layer 5: Orchestration and agents

Once the model can reach data and tools, something has to decide what happens first, what happens next, and when the job is done.

A simple chatbot is one model call. Real tasks are workflows. Back to the coding assistant: one request may need to be broken into tasks, then the assistant has to:

1. Understand the relevant part of the codebase
2. Write the change
3. Run the tests
4. Check the result against team conventions
5. Pick the right model or tool for each step

Then comes review. Another model can check the work, automated tests verify whatever is verifiable, and for high-impact changes most teams add a human approval step before anything merges. I wrote about that last step in code review is becoming a taste problem.

Frameworks like LangChain, CrewAI, AutoGen, the OpenAI Agents SDK, and Google's Agent Development Kit live here.

One more thing to remember: every framework, MCP server, and CLI tool has a version number. Your laptop may have the latest. A teammate may be a few releases behind. CI may have something else entirely. That drift looks minor, but it can change how an entire agent behaves.

---

Layer 6: Applications

This is the layer users see. All the complexity underneath disappears behind the product, and the user only cares whether it helped them finish something.

General-purpose assistants (ChatGPT, Claude, Gemini) are a crowded space, and for everyday tasks users can switch between them fairly easily. The stronger position often belongs to vertical products built for one industry, like legal, medical, or financial AI. They combine domain knowledge, proprietary data, and workflows that are much harder to copy.

The clearest example is Cursor. It did not start by owning the models, the cloud, or the chips. It owned the developer workflow: it brought several models together inside the place developers already work. In June 2026 SpaceX announced a roughly $60 billion all-stock deal to acquire it, and the deal has since closed.

Two lessons:

• You can create enormous value at the application layer without owning the stack underneath.
• If you own the workflow and the user relationship, that position stays valuable even to companies that own the infrastructure.

The competition is only getting stronger. Anthropic is pushing Claude Code, Microsoft has GitHub Copilot, and everyone wants to own the interface where developers do their work. I covered the engineering side of that shift in what is agentic coding.

---

Cross-cutting concern 1: Governance, observability, and evals

Once an AI system touches real company data, calls tools, and takes actions, you need controls around every layer.

Teams moving to production start asking basic questions:

• Who accessed the system?
• What did the agent do?
• Can we prove it later?

That is where audit logs, cost tracking, guardrails, monitoring, and compliance come in, along with model drift, cost spikes, and the agent-specific risks from layer 4. Cloud security went from afterthought to mandatory. AI governance is on the same path.

Evaluation belongs here too. Logs tell you what happened. Evals tell you whether a change made the system better or worse. You might change the model, the prompt, the retrieval logic, or a single tool. Without a fixed set of test cases, you can improve one behavior and quietly break three others.

---

Cross-cutting concern 2: Reproducibility

Remember the loose threads? Driver versions in layer 1, the serving stack in layer 3, tool versions in layer 5. This is where they meet.

Think about the journey an AI system takes: laptop → CI → build → production. At every stage the code runs inside an environment: a Python version, a CUDA version, a serving stack, agent tools, MCP servers. Is that environment the same at all four stages?

For most teams, no.

The laptop has one setup. CI has another. Production was built from a Dockerfile someone wrote last year. The environment that trained the model often was never recorded at all. So the same prompt and the same agent code behave one way locally, another way in CI, and a third way in production.

Nobody made a mistake. There were four environments and never one definition of the environment.

The fix is a principle you already trust

Your code is version controlled. Treat your environment the same way.

• Write down everything the system needs in one definition: runtimes, GPU and CUDA setup, AI tools, MCP servers, and the names of required environment variables.
• Let every stage build from that same definition.
• Keep secrets in your secret manager. The definition only says which secrets are required.
• Version the definition, so changes are reviewable and a vulnerable package is patched once, not machine by machine.

The tooling is your choice. Nix-based environments, dev containers, pinned lockfiles, and container images built from a single source can all do this. What matters is the property: the thing that travels from dev to CI to production is one small definition that resolves the same way everywhere.

When something fails only in CI, you can rule out a whole class of version mismatches. If you skip this, the layers above it can behave differently even when the code is identical. It is the easiest layer to overlook and the one that keeps everything else consistent.

---

The whole stack in one pass

---

Which layer do you own?

The useful question is not which layer matters most. It is which layer you own, and which ones you are renting without noticing.

If you call a hosted model, you rent layers 1 to 3. Your leverage is in 4, 5, and 6, and in the two cross-cutting concerns that nobody else will do for you.

The model is just one layer. Everything around it decides whether your AI system reaches production.

---

Where to go next

• How to Transition to AI Engineering: the skills behind these layers, with a learning roadmap
• What Is Agentic Coding?: the agent loop at layer 5, in practice
• Vibe Coding vs Agentic Engineering: staying in control when agents write the code
• Code Review Is Becoming a Taste Problem: the human approval step, and where judgment still wins