I've been running local AI inference on a single DGX Spark for a while now. The single-node setup handles a lot, but there's a class of open-weights models that just won't fit: 300B-plus parameter counts where even 128 GB of unified memory isn't enough. Today I bridged two Sparks and loaded GLM-5.3-Flash, a 320B mixture-of-experts model from Z.ai, across both. This is a record of what that looked like: the recipe choice, the setup, the failures in the order I hit them, and the first correct answer out of VS Code. What I built with the model once it was up is a separate post.

Why Two Sparks for One Model

GLM-5.3-Flash is a 320B MoE with 18B active parameters per forward pass. That active-parameter count is why it's fast at inference: you're only computing a fraction of the weights on any given token. But the full checkpoint still has to live in memory, and a 4-bit quantized version weighs in at 164 GiB across 120 shards. One Spark has 128 GB of unified memory. The math doesn't work.

Both nodes are GB10 DGX Sparks, 128 GB unified memory each. I'd already bridged them using NVIDIA Sync's Cluster Assistant. The interconnect is ConnectX-7 over RoCE v2 at 200 Gbps per link, with cluster addresses on the 10.100.240.x and 10.100.241.x subnets. The same admin user runs on both nodes, which matters more than it sounds like it should. More on that in the failures section.

The serving strategy is tensor-parallel 2: the model splits across both nodes along the tensor dimension. NCCL handles all-reduce operations between them over the RoCE v2 fabric. Every forward pass touches both machines, so interconnect latency matters. That's the whole point of the direct 200 Gbps cable between the two ConnectX-7 ports.

Why This Recipe

Two community approaches exist for running a model this size on Sparks. The alternative, from 0xSero, uses a 2.05 bits-per-weight quantization that fits on a single node. I looked at it seriously. The reason I went with the MiaAI-Lab two-node recipe comes down to one number: the 2-bit version agrees with BF16 on top-1 token selection about 79 percent of the time. That's not good enough for a coding assistant. I want the model to actually know what it knows.

The MiaAI-Lab recipe uses a 4-bit EXL3 quantization built by brandonmusic using turboderp's ExLlamaV3 kernels. It runs on vLLM with custom EXL3 kernel support, tensor-parallel 2 over NCCL and RoCE v2, and DFlash2 speculative decoding from incoai. The 4-bit checkpoint scores within rounding of the official FP8 on an independent KL divergence panel. That's the quality bar I care about. The recipe is also more mature: better documented, more people have hit the same errors, and the error messages are more useful when something goes wrong. The MIT license on the base model from Z.ai means I can use it without thinking too hard about it.

Setup: A Narrative, Not a Tutorial

First thing I did was generate a read-only Hugging Face token. The GLM-5.3-Flash checkpoint isn't gated, so no special access is required. But an authenticated token lifts the anonymous rate limits, which matters when you're pulling 120 shards. I didn't want a throttled download halfway through.

I ran the download on the head node only, then used rsync to push the checkpoint to the worker over the ConnectX-7 cable. Transfer rate came in around 391 MB/s. That's not the link ceiling. 200 Gbps would be roughly 25,000 MB/s. We were rsync- and NVMe-bound. The cable wasn't the bottleneck.

Before launch I needed to map the ConnectX-7 ports on both nodes using ibdev2netdev, then read the RoCE v2 GID table to confirm the right entry. The relevant entry is index 3, the IPv4-mapped address, and there are four RoCE devices per node, two of which are cabled. Getting that mapping right is what lets vLLM and NCCL find each other over the fabric.

One step that's easy to skip: flush the page cache before launching the server. Because memory is unified on the GB10, the CPU and GPU share the same pool, and the OS will happily cache the checkpoint files in page cache while vLLM is also trying to load them into working memory. Flush it first.

What Broke, In Order

I'm documenting these in sequence because if you're doing the same setup, you'll hit them in roughly the same order.

1. Docker group membership. Docker was preinstalled on the worker node, but the admin user wasn't in the docker group. The vLLM container wouldn't start. Fix: add the user, log out, log back in.

2. Missing python3-venv. The Hugging Face CLI install requires a virtual environment. python3-venv wasn't on the worker. The error is obvious once you see it, but it blocked the CLI install before I could even start the download there.

3. Worker network lines left at recipe defaults. This one bit me twice. The MiaAI-Lab recipe has placeholder network interface names for the worker node. I left them as-is the first time. vLLM's preflight check caught it. The GID table came back empty. I updated the interface names, restarted, and got past preflight. Then Gloo threw a runtime error: Unable to find address for: enp1s0f0np0. That's the head node's actual network interface appearing in the worker's routing table where it shouldn't be. Second fix: verify the interface names on both nodes independently, not just the one you're sitting in front of.

4. Boot warmup overlap. The first query I sent after the server came up returned 323 tokens of unrelated HTML. It looked like the model was hallucinating badly. It wasn't. The server was still completing its warmup pass when the first request hit. Every response after that was clean. Asking "17 times 23" at temperature 0 returned 391. Correct, and fast.

5. VS Code ERR_ADDRESS_UNREACHABLE. I connected GitHub Copilot Chat to the server using a custom OpenAI-compatible endpoint, chat completions API type, 32768 max output tokens. The connection string used the .local mDNS name. VS Code threw ERR_ADDRESS_UNREACHABLE. curl worked fine from the same machine. Two things fixed it: switch the endpoint to the LAN IP address instead of the .local hostname, and grant VS Code the macOS Local Network permission in System Settings. Both were required.

One configuration note on the Copilot Chat integration: I set the server default reasoning effort to high, not max. At max, GLM-5.3-Flash burns its entire thinking budget on reasoning traces before producing any output. High gives you the reasoning quality without consuming the whole budget every time.

Where This Post Stops

Once the endpoint was stable, the first thing I did was hand the model a written spec and point it at the vLLM server that was serving its own responses. That session became SparkDash, and it has its own write-up: the spec, the scorecard I wrote before starting, what the model got right, and the bugs it fixed on its own. None of that is repeated here. This post ends at the point where the cluster answers correctly and the editor can reach it.

I saw throughput numbers during the bring-up and I'm not reporting them as benchmarks. The conditions weren't controlled, I wasn't running SparkBench methodology, and a figure from a single session isn't a reproducible claim. What I can say is that latency felt usable from the editor: not instant, but not the kind of pause that breaks your flow.

Numbers Table

Item Value
Model GLM-5.3-Flash (Z.ai), 320B MoE, 18B active
Quantization 4-bit EXL3, 164 GiB, 120 shards
Hardware 2× DGX Spark GB10, 128 GB unified memory each
Interconnect ConnectX-7, RoCE v2, 200 Gbps per link
RoCE devices per node 4, of which 2 are cabled
RoCE v2 GID table entry index 3 (IPv4-mapped)
Parallelism Tensor-parallel 2 over NCCL
rsync transfer rate ~391 MB/s (rsync/NVMe bound, not link bound)
2-bit alternative top-1 agreement with BF16 ~79%
Max output tokens (Copilot Chat) 32,768
Server default reasoning effort high (max spends the whole budget on reasoning traces)
Failures documented, in order 5
Boot warmup artifact 323 tokens of unrelated HTML, first request only
First clean arithmetic answer (17 × 23, temp 0) 391 ✓

Scope Note: SparkBench vs. This

This session was a detour from my main project, SparkBench, a systematic benchmark suite for the DGX Spark. I'm not adding this run as a condition in SparkBench. The recipe uses a community-built container image rather than the pinned NGC container, and the EXL3 checkpoint is outside NVIDIA's support matrix. Those two things together mean I can't make apples-to-apples comparisons against the controlled conditions SparkBench is designed to produce.

The port mapping steps and the specific gotchas, the GID table verification and the page-cache flush in particular, went into SparkBench's cluster documentation. That doc is where I keep the setup knowledge that applies across all the work I do on this hardware, regardless of which recipe I'm running.

What I'd Do Differently

Verify the worker network config before the first launch attempt, not after. The double failure on the interface names cost time. The fix is simple: before starting vLLM, run ibdev2netdev on the worker and confirm the output matches what's in the config file. Ten seconds of checking saves twenty minutes of restarting.

Send a throwaway warmup request before the first real query. The 323-token HTML artifact from the boot warmup would have been invisible if I'd sent a dummy request first and waited for a clean response. I'll build that into my startup script going forward.

Use the LAN IP from the start for any VS Code connection. The .local hostname resolution through mDNS is flaky enough in VS Code's network stack that it's not worth debugging. Just use the IP. Same goes for the macOS Local Network permission: grant it during initial setup, not while you're already in the middle of debugging something else.

Credits and Links

This wouldn't have been possible without a few people's work: MiaAI-Lab for the two-node EXL3 recipe; brandonmusic for the EXL3/TR3 quantization; turboderp for ExLlamaV3; incoai for DFlash2; Z.ai for GLM-5.3-Flash and the MIT license; and the NVIDIA DGX Spark playbooks that got the cluster standing in the first place.