I ended the last post by promising a fine-tune. This is not that post. Again.
What happened is that I pulled one more model onto Andromeda — the DGX Spark I bought to do training — and it broke the post I'd just published. Not the arithmetic. The arithmetic is fine. It broke the sentence I built on top of the arithmetic, which is worse, because the arithmetic is the part I checked.
Here's the line I'm here to retract:
The Spark's 120 GB buys you capacity, not speed. It can hold models nothing else your size can — but it reads them at unified-memory bandwidth, so the bigger the model, the slower it runs. Fast or big. Still not both.
Every clause of that is defensible except the last two words.
## The Claim I Made
The centrepiece of the last post was this: decode throughput on the GB10 scales inversely with model size, and the constant of proportionality is memory bandwidth.
| Model | Q4 weights | Tok/s | Weights × tok/s |
|---|---|---|---|
| qwen2.5:7b | 4.7 GB | 48 | ~226 GB/s |
| qwen2.5:32b | 19 GB | 10.5 | ~200 GB/s |
| llama3.1:70b | 42 GB | 4.8 | ~202 GB/s |
Multiply the weights by the speed, get the same number every time. Every token, the GPU streams the model out of memory. Double the model, halve the speed.
I called it physics. It is physics. The GB10's datasheet says 273 GB/s of LPDDR5X, and ~200 GB/s actually achieved is a thoroughly ordinary 73% of it. Nothing mysterious.
The mistake wasn't the law. It's that I applied it to models, when it only ever applied to bytes read per token — and I'd quietly assumed those were the same thing.
## The Model That Breaks It
Qwen3.5-122B-A10B. The answer is in the name, and I read straight past it.
122B total parameters. A10B — about ten billion active per token. It's a Mixture-of-Experts model: every layer holds 256 experts, and the router wakes 8 of them for any given token. The other 248 sit there, resident, doing nothing at all.
Note
Straight from the checkpoint's config.json: 48 layers, 256 experts, num_experts_per_tok: 8, plus one shared expert that fires every time. It's also hybrid-attention — full_attention_interval: 4, so only 12 of the 48 layers run full attention and the other 36 use linear (DeltaNet) attention. That's why a 262K context fits on a desk at all.
So: when this thing decodes one token, how much does it actually read?
Not 122B worth. Roughly a tenth of it. The weights are resident — all of them, in the Spark's unified memory, which is precisely what the 120 GB is for — but they are not streamed. Each token pays for the experts it woke up, not the ones it could have.
My law multiplies bandwidth by bytes-per-token. I had been feeding it bytes-in-memory. For every dense model I'd ever run, those two numbers are identical, so the bug had no way to show itself. It took a sparse model to reveal that I'd been measuring the wrong weights the entire time.
## The Numbers
Same box, same night, same harness, same three prompt classes as both previous posts — a ~50-token chat question, a ~460-token code review, a ~1,360-token agent prompt with tool definitions. Five runs per cell, median, capped at 512 output tokens so every row measures sustained decode rather than how chatty a model feels.
I re-ran the dense models instead of quoting July's table, so every number below was measured on the same hardware on the same evening.
| Model | Type | Resident | Prompt | Wall (s) | Out tok | Tok/s |
|---|---|---|---|---|---|---|
| qwen2.5:7b | dense | 4.7 GB | medium | 11 | 512 | 45.3 |
| qwen2.5:7b | dense | 4.7 GB | long | 11 | 512 | 45.2 |
| qwen2.5:32b | dense | 19 GB | medium | 48 | 512 | 10.6 |
| qwen2.5:32b | dense | 19 GB | long | 49 | 512 | 10.5 |
| llama3.1:70b | dense | 42 GB | medium | 106 | 512 | 4.8 |
| llama3.1:70b | dense | 42 GB | long | 107 | 512 | 4.8 |
| qwen3.5:122b | MoE | 81 GB | medium | 18 | 512 | 29.2 |
| qwen3.5:122b | MoE | 81 GB | long | 19 | 512 | 27.3 |
| nemotron-3-super | MoE | 86 GB | medium | 24 | 512 | 21.4 |
| nemotron-3-super | MoE | 86 GB | long | 25 | 512 | 20.2 |
The dense rows reproduce July to the decimal — 10.5 and 4.8. The law is intact, the box hasn't changed, the harness agrees with itself two weeks apart.
Then there's the 122B, running six times faster than the 70B while being nearly twice its size on disk.
Put it through the law:
| Model | Resident | Tok/s | Resident × tok/s |
|---|---|---|---|
| qwen2.5:7b | 4.7 GB | 45.3 | 213 GB/s |
| qwen2.5:32b | 19 GB | 10.5 | 200 GB/s |
| llama3.1:70b | 42 GB | 4.8 | 202 GB/s |
| qwen3.5:122b | 81 GB | 27.3 | 2,211 GB/s |
Three rows agree on ~200 GB/s. The fourth says my desktop moved 2.2 terabytes per second across a bus that tops out at 273 GB/s.
Important
When your measurement says the hardware is running eight times faster than physics allows, the hardware isn't what's wrong. The model of the hardware is. My law didn't break here — it started screaming that one of its inputs was garbage.
## What It Actually Streams
The good thing about being wrong this specifically is that it's checkable. So I checked: read the tensor headers straight out of the Intel AutoRound INT4 checkpoint and add up what a single decode step actually touches — every always-on tensor, plus 8/256ths of the routed experts.
| Component | Size | Streamed per token? |
|---|---|---|
| Routed experts (256 of them) | 60.25 GB | only 8/256 → 1.88 GB |
| MTP head | 5.05 GB | no — idle unless speculating |
| Linear attention (36 layers) | 1.66 GB | yes |
embed_tokens |
1.53 GB | no — it's a lookup, one row |
lm_head |
1.53 GB | yes — full matmul, every token |
| Shared expert | 0.91 GB | yes — fires every token |
| Full attention (12 layers) | 0.78 GB | yes |
| Vision tower | 0.61 GB | no — no image in the prompt |
| Router | 0.08 GB | yes |
| Total resident | 72.38 GB | |
| Streamed per token | 6.83 GB — 9.4% |
Nine point four percent. That's the whole post.
Now re-run the law with the right input. Serving that exact checkpoint on vLLM, I measured 27.2 tok/s:
6.83 GB × 27.2 tok/s = 186 GB/s
Which is the same ~200 GB/s wall the dense models hit, on the same bus, give or take the efficiency you'd expect from gathering eight scattered experts instead of streaming one contiguous blob.
Important
The wall never moved. The 122B isn't beating memory bandwidth — it's paying a smaller toll, because 90% of the model isn't on the road for any given token. "Fast or big, pick one" was never a law about the Spark. It was a law about dense models on the Spark.
## The Second Way Through
There's another trick, and it's a genuinely different one.
MoE reduces the bytes you read per token. Speculative decoding doesn't reduce the bytes at all — it gets more tokens out of each read. The model drafts a couple of tokens ahead, then verifies them all in one forward pass. Verifying three candidates streams the weights once, the same as generating one token would, because the memory traffic is the same and only the arithmetic goes up. Bandwidth-bound work has compute sitting idle by definition. So you spend it.
Qwen3.5 ships the machinery for this in the box: an MTP (multi-token prediction) head, right there in the checkpoint — the 5.05 GB I excluded from the table above.
Here's the whole ladder, on stock NVIDIA vLLM, no patches:
| Config | short | medium | long |
|---|---|---|---|
| vLLM 0.21, stock | 28.3 | 27.9 | 27.2 |
| + FlashInfer attention | 28.1 | 27.7 | 27.1 |
| + MTP-2 speculative decoding | 37.8 | 37.4 | 34.9 |
One flag. +31%.
And then the punchline, which I did not see coming. Apply my law to the MTP row:
6.83 GB × 36.2 tok/s = 247 GB/s — 90% of the datasheet, above what the box demonstrably achieves in practice.
Broken again. Same law, same checkpoint, same bus, and it's lying again — because MTP violates the other assumption buried in it: one weight-read per token. With speculative decoding you get roughly 1.3 tokens per read, so "bytes per token" stops being a meaningful denominator.
Twice now, the law has caught its own bad assumptions by producing an impossible number. That's the most useful property a wrong model can have.
## The Free Tier, Revisited
One more row from that table deserves its own paragraph.
nemotron-3-super is the model I used in the very first post as the free cloud leg — 120B, via NVIDIA's free-tier API. It's also, I now notice, a 123.6B MoE with 512 experts and 22 active. The model I used to prove that free inference is slow was itself an MoE the whole time. It was just running on someone else's hardware.
Here's what it cost me then, and what it costs me now:
| Long prompt | Tok/s | |
|---|---|---|
| May — NVIDIA's free tier | 120s | 21 |
| July — Andromeda, on my desk | 25s | 20.2 |
The same number. I wrote this in May:
Public benchmarks for hosted Nemotron 3 Super clock it at ~178 tok/sec. I'm getting 21. That's the queueing tax — not a bug, the business model.
I was right about the tax. What I missed is that 21 tok/s is roughly what that model costs to run on a desk. The free tier wasn't throttling me down to 21 out of spite — they were metering me to about what the physics costs, and pocketing the difference in scheduling. Andromeda now serves the same model at the same speed, except it's a floor rather than a ceiling, there's no queue, and nothing leaves the house.
## The Quadrant, Re-Revisited
Three posts, three verdicts. I'd like to think they're converging rather than just oscillating.
- May: fast + free is empty. Pick one of dollars, latency, or hardware.
- July: I picked hardware. Got fast + local + small. Fast or big, not both.
- Now: fast + big + local, on a 122B, on a box that idles at 4 W.
The first post's conclusion still stands, and I'm not walking it back: for a chat assistant, a few dollars a year of API credit beats a $3–4k box on economics, and it isn't close. Nothing here changes the money.
What changes is the shape of the trade. MoE wants enormous capacity and modest bandwidth. That is the GB10's exact personality — 128 GB of it, at a bandwidth a discrete GPU would be embarrassed by. I spent two posts describing that as the Spark's flaw.
It's the Spark's thesis.
Important
The uncomfortable version: I bought this box for training, then wrote two posts explaining why it was mediocre at inference. It isn't mediocre at inference. It's mediocre at dense inference — and dense models are what I happened to reach for, not what the hardware was designed around.
I'm not even first to this. vLLM's own DGX Spark write-up says the box is best suited to MoE serving, and calls 100–130B models with roughly 10–15B active parameters a strong fit. They published that in June. I benchmarked a dense 70B in July and concluded the hardware had a bandwidth problem.
## What It's For, Updated Again
- Interactive chat, big MoE — the new row. A 122B at 29 tok/s on Ollama, 37 on vLLM with MTP-2. Local, private, offline. This did not exist for me two weeks ago.
- Interactive chat, small dense — still fine, still ~45 tok/s.
- Big dense models — still batch-only.
llama3.1:70bis still 4.8 tok/s and no amount of MoE enthusiasm fixes it. That wall is exactly where I left it. - Privacy-required interactive work — now at frontier-ish model sizes instead of 7B ones.
The honest one-liner: the Spark doesn't run big models fast. It runs sparse models fast, and sparse models happen to be big.
Things I learned the hard way — a debugging aside
Two Ollamas. Midway through, docker exec ollama ollama list told me the box had llama3.3:70b and qwen3:32b. The API on :11434 disagreed — model 'llama3.3:70b' not found — and offered a completely different catalogue, including the exact qwen2.5:7b / qwen2.5:32b / llama3.1:70b trio from May.
Two Ollamas. My docker-compose.yml starts one in a container; a native systemd ollama.service was already squatting on port 11434. docker port ollama returns nothing at all — the container's port mapping never bound, so it's been running blind next to the real one for weeks. Every number in this post came from the systemd instance, which is the one holding the GPU and the models.
Lucky, in the end: that's the instance that still had May's exact models on disk, which is the only reason the same-night reproduction was possible. My own SETUP.md insists "Docker, NOT the snap." It's neither. It's the thing I installed first and forgot about.
The recipe I didn't need. There's an excellent community recipe that takes this exact model from 28.3 to 52 tok/s on this exact box. I was all set to follow it: build vLLM from source for SM121, patch a hybrid INT4+FP8 checkpoint, hand-register the MTP tensors. Thirty to sixty minutes of NVCC, plus another 122 GB of downloads.
I got about an hour in before checking whether any of it was still necessary. It mostly isn't. The recipe was written in April against vLLM 0.19, when there were no SM121 wheels and no native MTP. The NGC image I already had on disk — nvcr.io/nvidia/vllm:26.05.post1-py3 — ships vLLM 0.21 with SM121 kernels, Qwen3_5MoeForConditionalGeneration, and Qwen3_5MoeMTP built in. Intel has since fixed the MTP index registration upstream too, so that patch is a no-op: all 785 tensors were already mapped.
Three months has eaten most of it. The SM121 build is stock. The MTP wiring is stock. FlashInfer, worth +16% on 0.19, is worth nothing measurable on 0.21 — the defaults caught up and I measured it twice to be sure.
The fair comparison is rung for rung: the recipe's hybrid-checkpoint + MTP-2 step lands at 38.4 tok/s. Stock 0.21 with one flag and no build at all gets 36.2 — 94% of it. The recipe's headline 52 is real, but the last stretch rides on an INT8 LM-head patch pinned to vLLM 0.19.1, and I'm not backporting a version-locked patch to chase 40% on a box I bought for training.
That's no knock on the recipe. It's what a good recipe does: show the ceiling, and then wait for upstream to come and take it.
## Up Next: The Fine-Tune, For Real
Two posts ago the Spark was going to be a training box, and I owed you a fine-tune. One post ago, I owed you a fine-tune. I still owe you a fine-tune.
In my defence, both detours were the same detour. I keep discovering that the thing I measured wasn't the thing I thought I was measuring. The NUC taught me that "free" was the expensive word. The Spark taught me that "big" was.
Next one's the training post. A base model, a dataset, LoRA versus a full run, and what the throughput, memory footprint, and loss curves actually look like on hardware that costs less than a month of the cloud instance you'd otherwise rent.
And if that post ends with me discovering I've measured the wrong thing a third time — well, at least the trend is honest.