01Two different jobs: the gateway and the router

These two words get used interchangeably in marketing, but they do different jobs. An LLM gateway (also called an AI gateway or LLM proxy) is infrastructure that sits between your application and one or more model providers. It exposes a single API — usually OpenAI-compatible — and adds the cross-cutting plumbing every app needs: key management, load balancing, retries and fallbacks, caching, rate limiting, logging, and guardrails. Tools like LiteLLM, Portkey, Cloudflare AI Gateway, and Kong are gateways. A router is narrower: it is the decision component that chooses which model should handle a given request, weighing cost, quality, and latency. Research routers like RouteLLM and managed ones like Amazon Bedrock Intelligent Prompt Routing are about that decision.

The cleanest way to hold the distinction: the gateway is the execution and observability layer; the router is the model-selection policy. A gateway often includes simple routing (ordering, weighted load balancing, fallbacks), and many products do both — but the responsibilities are separate, and they compose. A common pattern is a smart router choosing the model and a gateway running it, logging it, and falling back if it fails.

• A gateway standardises access and reliability across providers behind one API.
• A router picks the model per request to optimise cost, quality, or latency.
• They are complementary — a router is the policy, a gateway is the layer that executes and observes it.

02Why route at all? The cost–quality trade-off

Routing exists because no single model is best for everything. Stronger models are more capable but cost more and are often slower; smaller models are cheaper and faster but weaker on hard problems. A lot of real traffic is easy — and sending an easy question to a top-tier model wastes money. The core idea behind routing is to predict how hard each query is (or how big the quality gap between models would be) and send easy queries to a cheap model and hard queries to a strong one. That moves you along a tunable cost–quality frontier: you choose how much quality you're willing to trade for savings.

How much can this save? Studies and vendors report meaningful numbers on specific datasets — RouteLLM reports more than a 2× cost reduction in some cases without measurable quality loss; the Hybrid LLM paper reports up to roughly 40% fewer large-model calls with no quality drop; FrugalGPT reports up to around 98% cost reduction while matching the best single model on certain tasks; and Amazon claims up to about 30% cost reduction for Bedrock prompt routing. Treat all of these as best-case, reported results on particular workloads — not guarantees for your traffic.

• Capability, cost, and latency pull in different directions — routing balances them per request.
• Most savings come from sending the easy majority of queries to a cheaper model.
• Reported savings (2×, ~40%, ~98%, ~30%) are dataset-specific best cases — verify on your own traffic.

03See it work: route a stream of requests

Here is the idea made concrete. Requests of varying difficulty arrive at the gateway. Pick a routing policy and send traffic. Each policy makes a different trade: cost-optimized sends easy and medium queries to a cheap small model and only escalates the hard ones; quality-optimized escalates anything non-trivial to the strong model; latency-optimized favours the fastest path — the small model and cache hits — accepting lower quality. Send the same stream under each and watch cost, average quality, and average latency land in different places, see cache hits skip the model entirely, and watch the gateway fail over to a backup provider when a call errors. The numbers here are illustrative — chosen to show the mechanism, not to quote any provider's prices.

• Cost, quality, and latency trade off against each other — the same request stream lands in a different place under each policy, and the policy decides which axis you optimise.
• Caching answers repeat questions without calling any model, cutting both cost and latency (a gateway feature).
• Failover to a backup provider on an error is a reliability feature, separate from cost/quality routing.

04Five ways a router decides

"Routing" isn't one technique. Real systems use several, and they're often combined. Switch between them to see how each one chooses a model and where you'll meet it in practice.

05Failover, evaluation, and managed options

Two more pieces complete the picture. First, fallbacks: when a request fails — a connection error, a 404, a 429 rate-limit, or a timeout — the gateway routes to an alternate model or provider. Implementations use ordered lists with per-order retries before escalating to the next option (LiteLLM order-based fallbacks, OpenRouter model fallbacks, Portkey fallbacks). This is about reliability, and it's worth keeping mentally separate from cost/quality routing: one keeps you up when a provider has a bad day, the other saves money on normal days.

Second, how do you know a router is any good? Because there was no standard way to compare routers, RouterBench introduced a benchmark, a theoretical framework, and a dataset of over 405,000 inference outcomes so routers can be compared on the cost–quality Pareto frontier. If you'd rather not build routing yourself, there are managed and commercial options: Amazon Bedrock Intelligent Prompt Routing (serverless; predicts per-request quality and routes within a single model family; optimised for English; GA in 2025), and commercial routers like Not Diamond (pre-trained and custom routers) and Martian (an OpenAI-compatible router/gateway with max-cost controls and failover).

• Fallbacks = ordered alternates + retries on errors; a reliability feature distinct from cost/quality routing.
• RouterBench gives a standard way to compare routers on the cost–quality frontier (405k+ outcomes).
• Managed routers (Bedrock, Not Diamond, Martian) trade flexibility for less to build — mind their constraints.

06Check your understanding

07Take it with you & go deeper