I wanted to see if you've got a decent laptop, a server-class local inference machine, and an Azure account (all pointed at the same RAG app) how do they actually stack up? I saw the real wall-clock latency, the dollars per thousand queries, and whether the cheapest option hallucinates all over your documents.

In DocQuery Phase 2, I took the interfaces extracted in Phase 1 (IEmbeddingProvider, ILlmProvider, and IVectorStore) and gave each one a second, Azure-backed implementation, and wired everything so a single config value flips the entire AI stack. I used the same React UI, the same .NET API, and the same ChromaDB-or-Azure-AI-Search vector layer. Then I ran a committed, reproducible benchmark harness against all three stacks and wrote down what actually happened.

The Swappable Architecture Proved Itself Before the Benchmark Did

The whole point of the Phase 1 interface extraction was to avoid the situation where swapping an AI provider means rewriting your application. I figured I'd validate that thesis at leisure during benchmarking and funny enough, Azure validated it for me by breaking things mid-project instead.

I forgot that GPT-4o hit its retirement date in March 2026, so the model I'd planned in my README was gone before I finished deploying. Then quota limits blocked text-embedding-ada-002 on top of that. The two providers that I wanted to were officially obsolete. Luckily, I only needed to make two config-string changes. I swapped in in gpt-5-mini, swapped in text-embedding-3-small, and redeployed without needing to touch any code.

I also hit a cryptic 400: API version not supported from the Azure OpenAI SDK. It also turns out Azure AI Foundry surfaces several endpoints per resource, and I grabbed the project-scoped URL instead of the bare base URL the SDK actually wants. I then trimmed the path segment and then everything worked. I've definitely had friction points like this kill momentum on side projects but this wasn't an issue since I was catching them this early. I've learned that it's better to find these issues while the blast radius is small aka why it's better to build incrementally.

The other thing I shipped in this phase was a thread-safe session conversation memory with a capped history buffer, and a docker-compose stack with a pinned ChromaDB image, persistent volume, and containerized API and UI. Ollama stays on the host device (my macbook) deliberately because containers on macOS can't reach the GPU, so putting it in a container just adds latency for no reason. Sometimes the best call is to just not containerize the thing.

The Benchmark Numbers

I had three tech stacks with the same corpus, the same questions, a committed harness so the results are reproducible:

The first surprise was that the Azure hosted cloud model generates tokens faster but my laptop answers questions faster end-to-end. Azure hits 85 tok/s on raw generation but lands at 4.3s per query. The MacBook does 57 tok/s and lands at 2.4s. Network round-trips and reasoning warm-up eat Azure's throughput advantage. "Tokens per second" and "time to answer" are not the same number, and optimizing for one doesn't automatically move the other.

The second surprise was the DGX Spark (RIP $4.5K lol). The machine has 128GB of unified memory, which means a 70B model actually fits on the device and has plenty of room to spare. The model was a stable 43GB DGX Spark resident without needing any any model swapping or quantization compromise. But, fitting in memory and running fast are different things entirely. The memory bandwidth is what limits generation speed on large models, and the Spark hits almost exactly the theoretical ceiling. It's rate was 5.8 tok/s measured against what the math predicts. The model loads and runs, but it's just not fast for interactive queries at that size. For batch processing or workloads where you need the quality headroom a 70B gives you, it's the right call, but, for a RAG chatbot where someone's waiting on an answer, 21 seconds is a long time to stare at a spinner.

The third surprise was the quality of the responses. On factual retrieval over a small corpus, answer quality scores ranged from 8.4 to 9.0 out of 10, with zero hallucinations from any provider. The free local 8B model held its own against GPT-5-mini on this workload. That's not a universal finding for all use cases because corpus size, query complexity, and domain all matter, but it does suggest that for focused retrieval tasks, you're not automatically leaving quality on the table by staying local.

The Azure Cost Story Is Not What You'd Expect

I want to talk about the $245/month line item I almost didn't catch. The Azure AI Foundry setup wizard silently provisioned a Standard S1 Azure AI Search service as a dependency. List price: $245/month. I caught it because a $1.01 charge showed up for its three-hour existence before I deleted it. I created the resource manually after that, picked the Free tier, and moved on.

The wizard defaults to billing hourly. This issue isn't specific to Azure because it's how managed cloud services work. The fix? Don't let wizards create cost-bearing resources. Set it up manually, pick your tier, know what you're agreeing to. It's also worth knowing for anyone building on Azure Cosmos DB, the capacity mode (provisioned vs. serverless) is chosen at creation and CANNOT be changed later. That's the kind of decision you want to make on purpose and not find out that you needed the other option later on by accepting a default.

My total Azure spend for the entire phase, including all the benchmarks, all the API calls, all the embedding runs, was $1.06 out of a $50 budget. Cost anxiety about cloud AI development is mostly a tooling and service set up problem. Setting up budget alerts on your resource group, scoping resources so you can delete them cleanly, and removing whatever wizards spin up without asking are habits that will get you a long way.

Where This Goes Next

As of now the next phase (Phase 3) is study mode so adding flashcards and quizzes features, generated from my OMSCS course materials. I'm also adding streaming responses (the 21-second Spark latency is much more tolerable if tokens are showing up while the model is still thinking), hybrid search to mix dense vector retrieval with keyword matching, and eventually deploying the demo behind Cloudflare Pages with a tunnel back to the DGX Spark so it's publicly accessible without punching holes in my network.

The repo is at github.com/vondraysanford/docquery if you want to dig into the implementation. The integration tests are opt-in and no-op if you don't have an Azure account, so you can run the local stack without touching any cloud credentials. Clone it, point Ollama at a model you have pulled, and it should just work.

The architecture lesson from this phase is that the interfaces you extract for testability also become escape hatches when providers disappear, quota limits kick in, or a better model ships. I didn't design for that explicitly, but it was a side effect of doing things the right way. That's usually how the good side effects work :).