# Hybrid AI Orchestration Is the Architecture Nobody's Talking About Yet

Published: 2026-10-02 · Tag: AI
Canonical: https://vondraysanford.com/writing/2026-10-02-hybrid-ai-orchestration-is-the-architecture-nobody-s-talking.html
Author: Vondray Sanford (https://vondraysanford.com)

> Google's Antigravity SDK just added local model support, and the architecture it enables — cloud planners delegating to local executors — is one of the most practically useful patterns I've seen for real enterprise AI workloads.

Most AI architecture discussions split into two camps: everything in the cloud, or everything local. The cloud people want scale and convenience. The local people want privacy, latency, and cost control. Google's Antigravity SDK just shipped local model support, and the hybrid pattern it's enabling — a cloud model acting as a lightweight planner while local models handle the sensitive or latency-critical work — is actually the architecture I've been building toward with my own DGX Spark setup. It's worth explaining why this split matters and how to think about it structurally.

## The Problem with Treating Inference as One Monolithic Step

When you're building an agentic system, not every task needs the same model. A reasoning step that plans a multi-stage workflow is fundamentally different from a document extraction step that processes a sensitive PDF. One needs breadth and instruction-following. The other needs low latency, data locality, and zero egress.

The mistake I see constantly is treating the entire agent pipeline as a single inference call. You pick one model, send everything to it, and hope it handles the full spectrum. That works fine for demos. It falls apart in production when you're dealing with PHI, PII, or regulated financial data that legally cannot leave a network boundary. Or when you need sub-second response times for a step that runs inside a user-facing workflow and a round-trip to a cloud API adds 800ms you don't have.

Hybrid orchestration is the honest answer to this. You let a capable cloud model do the high-level planning — "here are the steps, here are the tools, here's the order" — and you route individual task execution to local models that never send that data anywhere. The cloud model never sees the sensitive payload. The local model never has to reason about the full orchestration graph. Each does what it's actually good at.

## What Running Local Inference Actually Teaches You

I run two NVIDIA DGX Sparks bridged over ConnectX-7 — 128 GB of unified memory each. I've been using that setup to benchmark models, run RAG workloads for DocQuery, and power local inference for AgentReview. So when Google says "run Gemma 4 26B locally via LiteRT," I have real opinions about what that actually involves.

The first thing you learn running local inference is that memory bandwidth matters more than raw parameter count for latency. A well-quantized 26B model on hardware with high unified memory bandwidth can outperform a larger model on a GPU-memory-bottlenecked setup for single-user, interactive workloads. That's why the DGX Spark's architecture is genuinely interesting for this pattern. The unified memory means you're not constantly shuffling data between host and device.

The second thing you learn is that local models need tighter prompting. Cloud frontier models have been RLHF'd to handle ambiguous instructions gracefully. A smaller local model running quantized will drift if your prompts aren't precise. In a hybrid setup, this is actually fine. The cloud planner can emit well-structured, constrained instructions to the local executor, which then doesn't need to do heavy lifting on instruction interpretation. The architecture compensates for the capability gap by design.

The third thing — and this one surprises people — is that cold start is your enemy, not throughput. Once a model is resident in memory and warmed up, generation speed is usually acceptable. It's the first call after a long idle period that kills UX. If you're designing a hybrid system, you need a keep-alive strategy for your local model process. Don't let the runtime unload between requests if your workload is interactive.

## How to Actually Structure a Hybrid Pipeline

Here's the mental model I'd use. Think of your agentic workflow as a directed graph of tasks. Each task node gets tagged with two properties: `data_sensitivity` and `reasoning_complexity`. High sensitivity, low complexity? Route local. Low sensitivity, high complexity? Route cloud. High on both? You have a design problem — split the task.

In practice, the routing logic itself doesn't need to be fancy. A simple classifier or even a static config per task type gets you 80% of the way there. Something like:

```
// Pseudocode — task router
if (task.DataSensitivity == Sensitivity.High)
{
    return LocalModelEndpoint; // never leaves the network
}
else if (task.ReasoningComplexity == Complexity.High)
{
    return CloudModelEndpoint; // frontier model for planning
}
else
{
    return LocalModelEndpoint; // default to local for cost and latency
}

```

The router is a first-class engineering concern, not an afterthought. It should be configurable, observable, and testable. I'd wire it up to emit telemetry — which endpoint each task hit, latency, token count — so you can audit the routing decisions and tune over time. This is exactly the kind of thing I built observability for in DriftWatch: the pipeline behavior is as important to monitor as the model output.

One more thing worth calling out: MCP (Model Context Protocol) fits cleanly into this architecture. If your local model is exposed via an MCP server — which is exactly what I've been doing with AgentReview — the cloud planner can call it as a tool without knowing it's local. The planner just sees a tool endpoint. Routing transparency is handled at the infrastructure layer. That's a clean separation of concerns, and it means you can swap local model backends without touching your orchestration logic.

## This Pattern Is Worth Investing In Now

The Antigravity SDK shipping this locally is a signal, not just a feature. The major cloud providers are acknowledging that "everything goes to the cloud" isn't the right model for enterprise AI workloads. Regulatory pressure, latency requirements, and cost at scale are all pushing in the same direction.

If you're building agentic systems today, start tagging your tasks by sensitivity and complexity now, even if you're not routing yet. When you're ready to introduce a local model — whether that's via LiteRT, Ollama, llama.cpp, or a DGX Spark in your closet — the routing layer is already reasoned through. You're not retrofitting the architecture under a live system.

The pattern is simple: cloud for breadth and planning, local for sensitivity and speed. Get comfortable with both sides and design the boundary intentionally. That's where the interesting engineering actually is.