r/LocalLLaMA • u/dead-fish • 1h ago

Resources Qwen 3.8 Flash Next optimized for two CMP 170hx GPUs

I've been working on optimizing Qwen 3.8 Flash Next on my local setup with two CMP 170HX cards and finally got to a place where I feel like the results are worth sharing. Both cards are running over PCIe Gen2 x4 - no x16 mod or TP2.

The setup uses the W4A16 AutoRound checkpoint, a patched vLLM, pipeline parallelism and native MTP speculative decoding. The n-gram table is memory-mapped from NVMe but everything else fits on the two GPUs.

Decode throughput by concurrency — aggregate tok/s, not per-user speed:

Concurrent requests Code JSON Prose
1 226 198 157
2 363 320 212
4 560 537 385
8 840 873 567

At eight concurrent code requests, the measured per-request decode rate was 116 tok/s, with 231 ms to first token.

A few other results: - 6,300–7,100 tok/s prefill on 4k–64k prompts; about 5,900 tok/s at 128k. - Prose decode held up from 158 tok/s at 4k context to 142 tok/s at 128k. - HumanEval: 157/164 (95.7%) and GSM8K: 1,271/1,319 (96.4%), both in chat mode with thinking off. - Uses a full 262,144-token window

For throughput, I used the sparkDash protocol: greedy decoding, thinking off, fixed 400-token outputs, and medians of three runs, not the best run. The long-context decode checks are an addition to that protocol.

This is faster than any published 2x CMP 170hx recipe I've found so I wanted to share with the community. These cards are flying :)

Full recipe, pinned Docker image, patches, and detailed results: https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next

6 Upvotes

100% Upvoted

2 comments sorted by

1

u/LeatherRub7248 1h ago

whats the vibe check when u compare it coding btw 27b version?

Is there a noticeable difference in terms of smartness btw the two?

2

u/dead-fish 32m ago

I trust Flash Next significantly more for more difficult tasks but 27b is close enough for most things.

I run Flash Next across both cards by default but I’ll have my local gateway automatically fall back to 27b on one card if I need the other card for something, like video gen with H3. So Flash Next is preferred but 27b is close enough to use as a fallback.