The six layers I keep building into every AI system

This year I built AI for very different problems. One reads medical reports. One remembers everything a company has said to its clients. One turns a long video into short clips. One records how a song was made.

When I put them side by side, I noticed something. The products look nothing alike, but the inside is almost the same. Every one of them has the same six layers.

This post is the map. Each layer has its own deep-dive post, linked below.

#The skeleton

A model is only one box in this picture. Most of the work, and most of the bugs, live in the layers around it.

#1. Ingestion: the real world is messy

Models are great with clean text. Real input is never clean. It is a scanned PDF at an angle, a WhatsApp voice note, a 40-minute video, a project file from a music app.

So the first layer is always the same job: turn messy input into clean, structured data. OCR for scans, speech-to-text for audio, connectors for inboxes and chat apps.

Deep dive: From a scanned PDF to clean JSON.

#2. Memory: an AI that forgets is a demo

A chatbot that forgets you after every message is fine for a demo. It is useless for real work. If a client mentioned a deadline three weeks ago, the system should know it today.

Memory is not one thing. It can be a vector store for search, a summary of past conversations, or a history of what a user did.

Deep dive: Giving an AI long-term memory.

#3. Reasoning: decisions, not just text

This is where the model actually helps someone decide something. Is this lab result a risk? Which 30 seconds of this video are the best? Which song fits this mood?

The trick I learned: do not ask the model to do everything. Let normal code do the maths and the rules. Let the model explain and handle the fuzzy parts.

Deep dive: Spotting risk over time.

#4. Agents: let the AI use your software

An agent is a model that can take actions: open a page, fill a form, create a task. The hard part is not making it act. It is making it act reliably and cheaply.

Deep dives: Teaching agents to fill a form in one call and Testing AI agents before you ship them.

#5. Providers: never marry one model

Prices change. Models get better or worse. Free tiers disappear. So every system I build talks to models through one small layer that can switch providers with a config change, and can run a local model when the data should not leave the machine.

Deep dive: Never depend on one AI provider.

#6. Trust: the layer people skip

If your system touches health records or client email, trust is not optional. Personal data should be removed before it reaches a model. Everything should be encrypted. Every access should be logged.

Deep dive: Privacy before inference.

#The same skeleton, in a real pipeline

Here is one pipeline, a video tool, laid on top of these layers.

And two more deep dives that are not about a single layer, but about a full pipeline: Turning a long video into shorts on your own machine and Making your website readable by AI agents.

#What this means if you are building AI

• Start with the layers, not the model. The model is the easiest part to swap.
• Ingestion and trust take more time than you think. Plan for it.
• Keep the model's job small and clear. Code does maths, the model does language.
• Build the provider switch on day one. It costs an hour now and saves weeks later.

Once you see the six layers, a new AI product stops being a blank page. You already know five of the six boxes. You only have to design the one that makes the product special.