Never depend on one AI provider: a small layer that switches models

The first AI feature I shipped called one provider directly, all over the code. Then that provider changed its prices. Then a better model came out somewhere else. Then a client asked for everything to run on their own machine.

Each change meant touching dozens of files. So now every AI product I build has one rule: the app never talks to a model provider directly. It talks to one small layer, and that layer talks to the providers.

#The idea

#Why this works now

Most providers today speak the same language. They offer an "OpenAI-compatible" API: the same request shape, the same response shape. OpenRouter, NVIDIA NIM, and Ollama on your own computer all support it.

That means switching provider is often just three values: a base URL, an API key and a model name.

#The layer itself

It is small. This is the core of it:

type Provider = { baseUrl: string; apiKey?: string; model: string };

const PROVIDERS: Record<string, Provider> = {
  cloud:  { baseUrl: process.env.AI_URL!, apiKey: process.env.AI_KEY, model: process.env.AI_MODEL! },
  router: { baseUrl: "https://openrouter.ai/api/v1", apiKey: process.env.OPENROUTER_KEY, model: process.env.ROUTER_MODEL! },
  local:  { baseUrl: "http://localhost:11434/v1", model: "llama3.1" }, // Ollama
};

export async function complete(task: string, prompt: string) {
  const name = process.env[`AI_FOR_${task.toUpperCase()}`] ?? "cloud";
  const p = PROVIDERS[name];
  if (!p) throw new Error(`no provider called "${name}"`);
  const res = await fetch(`${p.baseUrl}/chat/completions`, {
    method: "POST",
    headers: { "Content-Type": "application/json", ...(p.apiKey ? { Authorization: `Bearer ${p.apiKey}` } : {}) },
    body: JSON.stringify({ model: p.model, messages: [{ role: "user", content: prompt }] }),
  });
  if (!res.ok) throw new Error(`${p.baseUrl} answered ${res.status}`);
  const data = await res.json();
  return data.choices[0].message.content as string;
}

The rest of the app only ever calls complete("titles", prompt). It has no idea which company is behind it.

#A different model for each task

Notice the AI_FOR_<TASK> setting. Not every job needs the biggest model.

• Writing a title for a video is easy. A small, cheap model is perfect.
• Reading a medical report needs a strong model, or a local one for privacy.
• Generating a commit message from a code diff can run on a free tier.

With one setting per task, I can pick the right model for each job and change it without a deploy.

#Fallbacks: when a provider is down

Providers go down. Rate limits hit at the worst time. So the layer can try a list in order:

In code it is a short loop around the same request:

async function completeWithFallback(prompt: string, order = ["cloud", "router", "local"]) {
  let lastError: unknown;
  for (const name of order) {
    try {
      return await ask(PROVIDERS[name], prompt); // the fetch from above, for one provider
    } catch (err) {
      lastError = err; // down, rate-limited or timed out: try the next one
    }
  }
  throw lastError;
}

The user never sees which one answered. They just get an answer. One rule: only fall back on errors like timeouts, 429 (rate limit) and 5xx. A 400 means the request itself is wrong, and sending the same bad request to the next provider will not fix it.

#Local models with Ollama

Ollama runs open models on your own computer or server, and it serves the same OpenAI-style API on port 11434. That makes "run it locally" just another entry in the list.

I use local models when:

• The data is private and should never leave the machine.
• The task is simple and runs fine on a small model.
• I want to build and test without paying per request.

#Things to watch

• Prompts are not 100% portable. A prompt tuned for one model may behave a bit differently on another. Keep a small set of test inputs and run them after every switch. I wrote about this in Testing AI agents before you ship them.
• Not every feature is everywhere. Tool calling and JSON mode differ between providers. Keep the layer's interface small, so every provider can support it.
• Log which provider answered. When something looks wrong, you want to know who said it.

#The payoff

This layer takes about an hour to write. In return, a price change, a new model, a provider outage or a "must run offline" requirement becomes a config change instead of a rewrite. It is the cheapest insurance in the whole architecture.