A 51 GB Table, Two Sparks, and One Line of Code

Update, 8 September 2026. The multi-token prediction result below did not survive later testing. On my patched image that speedup came with silent output corruption, and the cause turned out to be the one-line kernel patch this very post describes. The full correction, with measurements, is in The Wall of Exclamation Marks. Everything else here still holds, and the model now runs faster than these numbers on the upstream image.

Qwen released Qwen3.8-Flash-Next as an open-weight preview of its next architecture: 125 billion total parameters, but only about 6 billion active per token. On paper that is a friendly shape for a small machine. In practice it took me most of a day and five distinct failures to get it serving on my hardware.

The final number is 49.4 tok/s single-stream across two NVIDIA DGX Spark GB10 boxes. Getting there was not about tuning. It was about finding out, four times over, that the thing I believed was the problem was not the problem.

The unusual part of this model

Alongside the usual attention and expert weights, this architecture carries a PLE n-gram embedding table of roughly 51 billion parameters. It is a lookup structure, not compute — but it still has to live somewhere.

My two Sparks are named Jean-Luc and Kathryn. Each has 128 GB of unified memory, of which about 121 GB is usable. Two nodes, 242 GB combined. A model whose weights total 122 GB should fit comfortably.

It did not.

Failure one: vLLM cannot split the table

vLLM loaded the model onto the head node and died at roughly 116 GB of 121 GB. Kathryn, meanwhile, sat at 51 GB — barely working.

The reason is structural. vLLM expands that n-gram table to BF16 (about 95 GB) and keeps one copy on rank 0. The second machine cannot help carry it. So the head node holds its share of the model plus the entire table, and there is no configuration that makes 95 GB and 40 GB fit into 121 GB alongside a KV cache.

I proved to myself that the KV budget was not the issue by lowering the memory utilization setting from 0.45 to 0.38. The crash point moved by 0.1 GB. When a large change to a parameter produces no change in the outcome, that parameter is not the cause.

Failure two: the quantization mirage

If the table is too big, quantize it. I screened more than eighteen community quantizations of this model, and found a trap worth sharing.

Several of them ship the table as FP8 on disk — the file sizes genuinely shrink, 170 GB down to 126 GB. But their config.json lists the table in the quantization ignore block. The runtime therefore treats it as unquantized and expands it back to BF16 on load. The disk saving is real; the memory saving is imaginary.

That gave me a screening rule that saved hours: read the ignore list before downloading 170 GB. If the n-gram table is excluded from quantization, the checkpoint will fail the same way, and you can reject it in ten seconds instead of an hour.

Failure three: solving memory, and hitting a wall anyway

I eventually found a quantization where the table itself is genuinely 4-bit — 98.7 GB total, with the table declared as a quantized target rather than an ignored one.

It worked. No out-of-memory at any point, on either machine. The capacity problem was completely solved.

It then failed on kernel shape constraints instead. Split across two GPUs, some dimension was not a multiple of 16, which the 4-bit kernels require. On a single GPU, a different kernel refused a layer with 96 output features when it needs at least 128. The model fit and still would not run.

The switch that worked

I moved from vLLM to SGLang, and read its source rather than its documentation. Two things there decide the outcome:

Those two together turn one 95 GB block on a single node into roughly 24 GB per node. The load finally balanced: 102 GB on Jean-Luc, 97 GB on Kathryn. Both machines doing real work, which is the whole point of owning two.

The last blocker was one line

The model still refused to start, always with the same compiler error deep inside an attention kernel.

The cause, found by reading the serving code, is a hardware gate:

if not is_sm100_supported():
    return None

GB10 reports compute capability sm_121. It is Blackwell, but it is not sm_100. So the fast attention path was refused, and the fallback kernel was one that cannot compile for these shapes on this chip. The machine fell into a gap between two hardware generations.

No flag fixes this. I verified that: the failure is identical with every attention backend on offer, with CUDA graphs disabled, and with autotuning disabled — because the model's own layers select that kernel regardless of what you pass on the command line.

Relaxing that one condition to accept sm_12x devices made the model serve. One line, after everything else had already been correct for hours.

What the numbers say

With it running, I measured every optimization I had been assuming would help.

Three things in that table are worth more than the headline.

CUDA graphs are worth 2.4x. Not a few percent — the difference between 36.5 and 14.9 tok/s. And they only work at all because of the one-line patch, since the same broken kernel is used during graph capture.

Two good optimizations cancelled each other. Expert parallelism helps on its own, +4%. Multi-token prediction helps a lot on its own, +35%. Together they are *worse than either*, at 35.8 tok/s. Had I stacked them because both looked good individually, I would have shipped a slower configuration and never known.

Speculation trades throughput for latency. Multi-token prediction wins 35% single-stream but costs 12% of aggregate throughput under load. For interactive work that is the right trade. For a batch workload it is not, and K=1 is the interesting middle: +25% latency at *full* throughput.

I also found the ceiling honestly. K=4 fails, because the architecture's compression ratio caps how many tokens may be drafted. K=3 is not a tuning choice, it is the maximum the model permits.

What the benchmark does not tell you

That table is a 16,000-token context with synthetic prompts. Days later I pulled 248 decode samples out of a real working session — roughly 170,000 tokens resident in a 254,000-token context — and the picture is more interesting.

Two things fall out of that, and they point in opposite directions.

The 49.4 tok/s headline is the short-context best case. Under real load the median is 37.9, and the p75 of 50.2 is where the benchmark condition reproduces. If I quoted 49.4 as "the speed of this model" I would be quoting the best row of a favourable test.

But speculation works better on real text than on my benchmark. Acceptance in production has a median of 2.38 against the benchmark's 1.75, and it hits the theoretical maximum of 4.00 outright at times. Real prose is more predictable than three synthetic prompts about cache lines and B-trees.

So the throughput I lose at long context is the KV cache, not worse drafting. That is a useful distinction: it means the fix for slow long-context work is memory layout, not more aggressive speculation.

One more failure, four days later

The model had been serving for days when a subtler problem appeared. Unloading it sometimes left both containers alive. They kept answering health checks perfectly, so the next request simply re-attached to the old server — and a context-length change silently did nothing at all.

The cause was a race in my own teardown. The router kills the wrapper if it does not exit promptly, and my teardown fetched a log over SSH before stopping two containers sequentially, up to forty seconds. It lost that race and orphaned the pair.

What makes this one worth mentioning is that my own safety net could not catch it. The memory guard reaps containers that are not serving. This one was serving — perfectly healthy, just no longer owned by anything. Killing both nodes in parallel first and collecting the logs afterwards fixed it.

A process that looks healthy and answers correctly can still be wrong, if nothing owns it any more.

What I take from this

On-disk size is not memory size. A quantized checkpoint tells you what the file weighs, not what the runtime will do with it. Verify the second thing.

Measure the combination, not just the parts. Two independently-good changes are not automatically good together, especially when one adds communication and the other depends on tight timing.

Blackwell is not one thing. sm_100, sm_120 and sm_121 are different targets, and a kernel gate written for a data-centre GPU will quietly exclude a desktop one. If you run AI on hardware that is not the reference platform, expect to meet this.

Read the source. Every genuinely useful fact in this project — how the table is stored, why the fast path was skipped, where the ceiling is — came from reading the serving framework's code. Documentation and search did not produce them, and in one case actively misled me.

None of this is exotic infrastructure work. It is the ordinary reality of running current models on hardware you actually own, and it is why I keep doing it locally: you cannot learn where a system breaks if you only ever see the part a vendor chose to expose.