A working RAG pipeline and a usable RAG app are two different things. After Phases 1 and 2 of DocQuery I had solid local + Azure infrastructure with ChromaDB, .NET API, and Ollama, a provider abstraction layer. But the experience of actually using it was rough, you'd submit a question, wait, and eventually get an answer. Fine for a demo, but not fine for something you'd actually reach for.
Phase 3 closed that gap with these three things. Streaming answers over Server-Sent Events (SSE), a live provider selector with health-check feedback, and a side-by-side comparison mode where two AI stacks race each other on screen. The infrastructure barely changed, but the way it feels to use changed completely.
Getting the SSE Event Order Right
Streaming itself isn't the interesting part anymore because every chat UI does it now (it's still cool to me). What matters is the order of events, this is the sequence I landed on. First the citations fire the moment retrieval finishes, before the model writes a single token. Then, token deltas stream as they generate and then a done event closes the stream.
That ordering is deliberate so users see what grounded the answer while the answer is still being written. The source documents appear, and then the model starts filling in the response against them. It reframes the wait so instead of watching a cursor blink, you're reading the sources and forming your own expectation before the model even starts.
This matters a lot when the model is slow (IF YOU HAVEN'T READ DocQuery Phase 2, read it here). For example, my DGX Spark running Llama 3 70B takes around 20 seconds per answer :(. Watching it type token by token for 20 seconds is tolerable, but staring at a spinner for 20 seconds is not. SSE made the latency feel shorter by giving users something real to look at immediately, which is the whole trick.
On the backend, the .NET API streams the response using a standard text/event-stream content type. The retrieval step runs first, the citation payload goes out as its own event, and then the generation step begins and emits deltas as they arrive from the model. The frontend accumulates deltas and renders them incrementally. Nothing crazy or out of the ordinary, just getting the sequence right.
Provider Switching and the Health-Check Tooltip
The backend registers every configured provider stack as a keyed service in .NET's DI container. A local 8B Llama3 model running on Ollama, my DGX Spark 70B connected over an SSH tunnel, and Azure gpt-5-mini. Each one gets a key, and each HTTP request picks its stack via a header so we don't need to restart, rebuild, or config changes to swap providers mid-session.
The selector health-checks each provider on load and disables unreachable ones, but shows the reason as a tooltip. So instead of a grayed-out button with no explanation, you get "Ollama unreachable at localhost:11435 — is the tunnel up?" The UI teaches you the fix rather than hiding the option (this feature is for me). It's a small thing, but it saved me a lot of mental overhead when I was actively toggling between my laptop, the DGX Spark, and Azure during development.
Every answer also gets an attribution badge showing which provider produced it and how long it took, measured client-side. I didn't originally plan this as a benchmark feature, but it became one. After a few sessions you naturally start building intuition about which stack is faster for which question type.
Side-by-Side Mode and the Embeddings Constraint
The comparison mode fans one question out to two providers at once. Both answers stream concurrently in two columns, each with its own source citations and timing. Watching the local 8B finish in 4 seconds while the 70B is still on its first sentence makes what used to be a dry benchmark table visceral, you can literally see the tradeoff.
Building this exposed the most important design constraint in the whole project, chat models are freely swappable, but embeddings are sticky. If you embed documents with one model, you can only search them with that same model's embeddings. Switching the chat model is fine. Switching the embedding model means your documents effectively disappear from that store.
The fix was to scope stores by embedding model, not by provider profile. My Local and Spark profiles share one ChromaDB store because they use the same embedding model, only the chat model differs. Azure uses a completely separate store so I also scoped the document list per store so the UI always shows the truth about which documents exist for the provider you're currently looking at. Getting this wrong is subtle but painful because documents don't throw errors, they just return zero results, and you spend a while wondering if your retrieval pipeline broke.
The moment that proved the whole Phase 2 provider abstraction was worth building was when I sent the same question twice in a single running API process, once with a local header and once with an Azure header, and got back one answer from Llama 3 on my laptop and one from gpt-4o-mini in East US. Phase 3 is where it paid off.
Phase 4: Going Public
The next phase, phase 4, is the live deployment. DocQuery is going public at docquery.vondraysanford.com as a "Ask my portfolio" feature. Recruiters and engineers can ask questions about my work and get cited answers from an Azure-hosted deployment. The corpus will be my resume, project write-ups, and these posts. It's the most honest portfolio demo I can think of. Instead of reading a bullet list, you ask a question and see what comes back.
The code and a demo GIF of the two-column race mode are at github.com/vondraysanford/docquery. The phase 3 GIF shows the comparison mode better than I can describe it in text.
The takeaway from Phase 3 is that streaming, attribution, and comparison are what makes a pipeline useful instead of just functional. Build the infrastructure first, but don't stop there because the UX layer is where the work becomes real.