# Model Routing Is an Infrastructure Problem, Not a Prompt Problem

Published: 2026-08-17 · Tag: AI
Canonical: https://vondraysanford.com/writing/2026-08-17-model-routing-is-an-infrastructure-problem-not-a-prompt-prob.html
Author: Vondray Sanford (https://vondraysanford.com)

> Google Cloud API Gateway's new model routing feature is getting framed as a developer convenience, but what it's really solving is a production infrastructure problem that anyone running multi-model AI workloads has already hit the hard way.

Google Cloud API Gateway's new model routing feature is being framed as a developer convenience: swap out models without changing code, configure rules in one place, done. That pitch isn't wrong, but it undersells the actual problem it's solving. Anyone who's run multi-model AI workloads in a real enterprise environment knows the pain isn't the *endpoint URL*. It's everything that accumulates around it: latency differences between providers, cost variance across model tiers, fallback logic when a model is degraded, and the fact that hardcoding any of this into application code is a disaster waiting to happen.

I've been building AI-powered services for a while now. The Injury Evaluation service I architected uses ML integrations across a pipeline that feeds data from systems processing over a trillion dollars in historical insurance claims. Model selection in that context isn't academic. The wrong call on routing is a support ticket, a bad recommendation, or a downstream system failure. So when I saw Google's model routing announcement, I didn't read it as a product feature. I read it as infrastructure catching up to a problem engineers have been duct-taping around for two years.

## What Hardcoded Model Endpoints Actually Cost You

Here's the failure mode nobody talks about in blog posts: you pick a model at integration time, you ship it, and six months later the model is deprecated, rate-limited, or just slower under your new load profile. Now you've got endpoint strings scattered across config files, environment variables, and maybe a few places where a junior dev inlined the URL directly into a service call. Good luck finding all of them.

The more serious version of this is multi-environment drift. You're running Gemini in prod because it passed your eval suite, but your dev and staging environments are pointing at a cheaper model to save money, and now your integration tests aren't actually testing what's running in production. I've seen this exact pattern cause subtle regressions that only surface under production load, not because the code changed, but because the model behavior was different and nobody noticed until a real user hit it.

Google's API Gateway approach centralizes routing rules so you're not making model selection decisions inside application code. You define the rules, route 80% of traffic to Gemini, fall back to Claude on timeout, cap OpenAI usage by token budget, and the gateway enforces them. That's the same principle as a load balancer or a feature flag system. The model is just another upstream. Treating it that way is the right abstraction.

## The Part That Matters: Fallback and Cost Control

Dynamic routing without fallback logic is just a fancier hardcoded endpoint. The real value shows up when a model provider has a degraded availability window, which happens more than the status pages admit, and your gateway can automatically shed traffic to a secondary model without your application knowing or caring.

I dug into how the routing rules are configured in API Gateway and the pattern is essentially what you'd design yourself if you were building this from scratch: priority-ordered backends, condition-based routing by request attributes, and timeout thresholds that trigger the next rule in the chain. If you've ever written a circuit breaker pattern in .NET using Polly, this is the same mental model applied at the infrastructure layer instead of the application layer.

```
# Conceptual routing rule structure
routes:
  - model: gemini-2.0-flash
    weight: 80
    timeout_ms: 3000
  - model: claude-3-5-sonnet
    weight: 20
    fallback: true
    trigger: timeout | error_rate > 0.05
```

Cost control is the other dimension that gets underplayed. In enterprise AI deployments, the token budget is a line item someone is watching. Being able to route low-complexity requests to a cheaper model tier, and reserve the expensive frontier model for high-stakes or high-complexity calls, is something teams are currently implementing by hand in middleware. Centralizing that in the gateway means it's consistent, auditable, and doesn't require every service team to reinvent the same logic.

## Where This Fits in a Real Architecture

If you're running multiple AI services, and most enterprise shops with more than one AI initiative are, the model routing layer becomes part of your platform, not your product. This is the same evolution that happened with service meshes. Individual teams stopped managing their own retry logic and TLS termination and moved it to infrastructure. Model routing is heading the same direction.

For AgentReview, my multi-agent PR review system, each agent currently resolves its model via configuration at startup. That's fine at small scale, but as I think about making it more production-grade, having a routing layer that can make per-request decisions based on load, cost, or context window requirements is the obvious next step. Right now if Anthropic has a blip, the agent fails. With a routing gateway in front, it degrades gracefully.

One current limitation: model routing at the gateway level works well for request/response patterns, but streaming responses complicate things. If you're mid-stream on a response and the backend degrades, fallback gets messy. That's a hard problem with streaming anywhere, not a knock on Google's implementation specifically. Understand it before you design your fallback strategy around latency-sensitive streaming use cases.

## Start Treating Models Like Upstreams

The mental shift that makes this whole category of tooling click is simple: a model is an upstream service, not a static dependency. You wouldn't hardcode a database connection string into your application and call it a day. You'd put it behind a connection pool, add retry logic, and configure failover. Models deserve the same treatment, and infrastructure tooling is finally mature enough to provide it.

If you're in the early stages of a multi-model deployment, start with a clear routing contract. Define what attributes you'll use to make routing decisions (request type, user tier, token estimate, latency budget) before you pick a tool. API Gateway's model routing, or any similar proxy layer, will only be as smart as the rules you feed it. Garbage-in routing rules produce expensive, slow, or broken AI features just as reliably as a bad prompt does.

The actual work is designing the decision logic. The gateway is just where you enforce it.