# Microsoft's Three-Layer LLM Routing Architecture: What It Gets Right (and What You Still Have to Figure Out Yourself)

Published: 2026-07-30 · Tag: AI
Canonical: https://vondraysanford.com/writing/2026-07-30-microsoft-s-three-layer-llm-routing-architecture-what-it-get.html
Author: Vondray Sanford (https://vondraysanford.com)

> Microsoft's reference architecture for routing AI agent traffic on AKS splits the problem into three clean layers. I've built and run AI services on Azure in production, so I went through it looking for what holds up and what it leaves you to solve on your own.

Most teams building AI agents hit the same wall eventually. The model works fine in isolation, but the moment you put real traffic on it, everything gets complicated. Then you start asking the usual questions: Which model handles which request? What happens when a GPU replica is saturated? Who manages timeouts, retries, and cost guardrails across a fleet of agents? Microsoft just published a reference architecture for routing agent traffic on Azure Kubernetes Service (AKS) that breaks this problem into three distinct layers. It's worth going through carefully as a summary and as a framework that you can reason about when you're building this stuff.

I've shipped AI services into production on Azure, including an ML-backed evaluation service and an AI-driven PR automation tool used daily by over 100 engineers, so I've bumped into most of these decisions firsthand. The three-layer framing Microsoft is proposing lines up closely with the hard choices I've had to make, and I want to walk through each one with some actual context.

## Layer 1: Which Model Answers the Call

This is the most strategically loaded decision of the three. Not every agent request deserves a frontier model because some requests are simple classification tasks, long-context summarization jobs, or are just low latency above all else. Routing them all to Opus 5 (or Gemini Ultra, or whatever your preferred heavyweight is) is expensive and often overkill.

Microsoft's architecture treats model selection as a routing concern, not a hardcoded application decision which is the right call. You want a layer that can inspect the incoming request, including intent, estimated complexity, token budget, and latency SLA, and dispatch it to the appropriate model endpoint.

In practice, this means you're building or adopting a router that understands your model catalog. I've seen teams start with a simple rules engine, then graduate to a classifier model that does the routing itself. The tricky part is that the routing logic becomes a critical path dependency so if your router is wrong or slow, every downstream agent call pays for it. You have to treat it like any other latency-sensitive service and make sure it has low overhead, is well-tested, and is independently deployable.

One thing the reference architecture doesn't spell out is how you version the routing logic alongside your models. When you swap out a model or add a new one, your routing rules probably need to change too. That coupling is easy to miss early on and painful to untangle later.

## Layer 2: How the Call Is Managed

This layer covers all the middleware concerns that sit between the router and the model like rate limiting, retries, circuit breakers, token budgets, authentication, and observability. If you've built any kind of API gateway before, this layer will feel familiar, but LLM traffic has some quirks that make it harder than a standard REST proxy.

The big one is streaming, a cool visual feature that is actually important for user experience. Most LLM responses are streamed token by token, which means your traditional "did this request succeed?" metrics don't map cleanly. A request can start streaming successfully and then stall 80% of the way through. Your circuit breaker logic, timeout strategy, and cost tracking definitely has to account for that.

The other quirk is token-based billing. For example, with a normal API you might rate-limit by requests per second, but with LLMs you care about tokens per minute, prompt tokens vs. completion tokens, and the ratio between them for a given workload. I've done some of this with our internal tooling, tracking token consumption per engineer per day on the PR automation system, and getting that instrumentation right takes deliberate effort. It's not something you bolt on after the fact.

Microsoft's recommendation to handle this at the AKS infrastructure layer rather than inside each agent's application code is solid. You don't want 12 different agent services each implementing their own retry logic and rate limiting, this becomes 12 different places you need to maintain and keep track of changes. You have to centralize it, make it configurable, and surface the metrics in one place.

## Layer 3: Which GPU Replica Handles It

This layer is the most infrastructure-heavy of the three and the one where AKS specifics matter most. When you're running self-hosted models, and more teams are doing this, especially with capable smaller models and hardware like the NVIDIA DGX Spark becoming more accessible, you need intelligent load balancing across your GPU replicas.

The naive approach is the round-robin solution, but the problem is that LLM inference isn't uniformly stateless. You have to factor in KV cache warm state, current batch sizes, and memory pressure, which all vary per replica. A round-robin load balancer doesn't know that replica 3 is in the middle of a 32k-token context window and is a bad choice for the next request.

I've been running local inference on my own DGX Spark and macbook for RAG work, and even at small scale you notice that not all requests are equal in terms of what they demand from the hardware. Routing a long-context request to a replica that's already memory-pressured will tank latency for both that request and the ones already running on it. Smarter scheduling matters a lot more than people expect until they've seen it go wrong.

On AKS specifically, this means you probably want a custom scheduler or at least a load balancer that can query replica health metrics before dispatching. Microsoft's reference architecture leans on Kubernetes-native tooling here, which makes sense because it keeps the operational surface area manageable.

## What the Architecture Doesn't Cover & What You Still Need to Figure Out

Reference architectures are useful starting points, but they tend to stop where the hard product decisions begin. A few things I'd flag that you'll still need to work out on your own:

**Agent-level observability.** Knowing a request was routed to model X on replica Y is useful infrastructure data, but it doesn't tell you why an agent produced a bad output, how many tool calls it made, or where in a multi-step chain it went off the rails. You need tracing that spans the whole agent execution, not just the model hop.

**Cost attribution.** If you're running agents for multiple teams or customers, you need to tie token spend back to the right cost center. This is a data modeling problem as much as an infrastructure one. You need to figure out your tagging strategy before you're trying to reverse-engineer it from billing logs.

**Fallback behavior under degradation.** What does your agent do when the routing layer decides no model is available or the request exceeds budget? Does it fail gracefully? Queue and retry? Fall back to a cheaper model? These edge cases need explicit product decisions, not just infrastructure defaults.

**The three-layer framing is useful.** Model selection, call management, and replica dispatch are the right abstractions. But the work of wiring them together for your specific workload is still on you. Start by getting clear on which layer is actually causing your current pain, and build from there rather than trying to stand up all three at once. Pick your biggest failure mode, fix it, then move on to the next one.