EngramHalo.cpp: Qwen 3.8 Flash Next at 38 t/s on Strix Halo
Following up on the hipCUB fix post, I built and benchmarked EngramHalo.cpp, a llama.cpp fork with RDNA 3.5 kernel patches and MTP speculative decoding. The result: 28-38 tok/s decode on the same hardware and quant, nearly doubling the previous 15-20 tok/s baseline.
In my last post, I got Qwen 3.8 Flash Next running at 15-20 tok/s on my 128 GB Strix Halo machine. The hipCUB fix was the breakthrough that removed a 3-4x decode cliff past 1K context, and in the “What’s Next” section I mentioned EngramHalo.cpp as an unvalidated community fork claiming 39.3 tok/s. I said I hadn’t tested it yet.
Now I have. The claims are real. With the same hardware and the same AtomicChat AD-4.27bpw quant, EngramHalo.cpp with MTP pushes decode to 28-38 tok/s at working context depths. The peak hit 38.5 tok/s on a code prompt at 3.4K context, nearly matching their published 39.3 number (which used a smaller IQ3 quant in RAM mode). This isn’t a different model or a different machine. It’s the same setup from last week, just a better fork.
What EngramHalo.cpp Actually Does
EngramHalo.cpp is a llama.cpp fork on the strix-halo-qwen4exp branch by Aristo94. It carries a patch series on top of the upstream qwen4exp merge (PR #27742) that targets the Radeon 8060S (gfx1151) specifically. The patches fall into four categories:
1. The hipCUB fix, but better. EngramHalo includes a custom wide top-k selection kernel that fixes the same decode cliff I worked around last time, but goes further. Instead of just enabling hipCUB to unblock the top_k GPU path, their kernel is tuned for the specific shapes that QSA generates.
2. QSA sparse gather. This is the big one for decode throughput. Stock llama.cpp runs QSA “sparse” attention by multiplying against the full KV cache with a mask, meaning you pay the bandwidth cost of the entire cache even though the model only looks at ~2,048 selected positions. EngramHalo rewrites this to actually gather the selected rows first, so decode at depth reads ~2.3K rows instead of the full cache. This kicks in automatically past 16K context. Quality impact is negligible: 0.03% PPL delta on wikitext-2.
3. MTP draft head. Multi-token prediction using draft weights baked into the official Qwen 3.8 Flash Next checkpoint. A small Q8_0 sidecar GGUF (~4 GB) contains the 4B MTP head. During decode, the draft head proposes multiple tokens that the main model verifies in a single forward pass. When acceptance rates are high (code, structured text), you get 1.5-2x effective throughput. When acceptance is low (creative prose, highly variable output), the overhead mostly cancels out.
4. Kernel-level tuning. Skipping fully-masked warp slices in the flash attention vector kernel, picking the right kernel variant for qwen4exp’s hd-256 GQA-2 shape on RDNA, and batched mmap readahead for the SSD-backed n-gram table instead of one page fault per row.
Building It
EngramHalo ships a Dockerfile.rocm-7.14 that builds everything from source. One thing to note: this uses ROCm 7.14 (TheRock), not the ROCm 10.0 I was running before. That’s what they tested on and I stayed with it.
cd ~/Code
git clone --branch strix-halo-qwen4exp https://github.com/Aristo94/EngramHalo.cpp.git
cd EngramHalo.cpp/docs/strix-halo
podman build -f Dockerfile.rocm-7.14 -t engramhalo .
toolbox create engramhalo --image localhost/engramhalo -- \
--device /dev/dri --device /dev/kfd \
--group-add video --group-add render \
--security-opt seccomp=unconfinedThe container build takes about 5 minutes. Most of that is compiling HIP kernels for gfx1151.
You also need the MTP sidecar GGUF. A prebuilt Q8_0 version is on HuggingFace:
hf download EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF \
--local-dir ~/Secondary/Models/Qwen3.8-Flash-Next/Per the EngramHalo docs, the Q8_0 sidecar actually outperforms BF16 because it halves the draft read bandwidth while keeping quant-matched errors, which improves acceptance rates.
Benchmark Results
I ran three backends across six context depths on the same hardware (Ryzen AI MAX+ 395, 128 GB LPDDR5X, Fedora 44) using the same AtomicChat AD-4.27bpw quant. Protocol: stream: false, temperature 0, 128 generation tokens, full llama-swap restart plus process kill between backend switches to ensure clean memory.
Decode Throughput (tok/s)
| Context Depth | hipCUB Baseline | EngramHalo | EngramHalo + MTP |
|---|---|---|---|
| ~185 tokens | 21.9 | 20.9 | 29.3 |
| ~1K tokens | 20.4 | 21.2 | 28.2 |
| ~3.4K tokens | 19.8 | 20.5 | 38.5 |
| ~6.6K tokens | 19.0 | 19.4 | 23.5 |
| ~13K tokens | 17.6 | 18.4 | 31.7 |
| ~26K tokens | 15.5 | 16.8 | 15.0 |
Prompt Processing (tok/s)
| Context Depth | hipCUB Baseline | EngramHalo | EngramHalo + MTP |
|---|---|---|---|
| ~185 tokens | 89.6 | 91.8 | 85.3 |
| ~1K tokens | 53.8 | 390.4 | 196.3 |
| ~3.4K tokens | 431.9 | 450.7 | 399.0 |
| ~6.6K tokens | 411.1 | 452.6 | 406.1 |
| ~13K tokens | 362.2 | 432.9 | 387.2 |
| ~26K tokens | 291.6 | 395.1 | 351.1 |
A few things jump out.
MTP is transformative at working depths. The 38.5 tok/s peak at 3.4K context is a +94% improvement over the hipCUB baseline on the exact same hardware and quant. At most depths between 1K and 16K, you’re getting 28-32 tok/s, comfortably above the ~20 tok/s ceiling from before.
The 32K regression is real. At 26K context, MTP drops to 15.0 tok/s, actually slightly worse than the baseline. MTP acceptance rates fall at high context depths, and the overhead of draft verification starts to outweigh the gains. This makes EngramHalo+MTP best suited for interactive use and shorter turns rather than deep-context agent sessions.
Kernel patches alone are worth it for prompt processing. Even without MTP, EngramHalo improves PP by 8-35% across all depths (most visible at 26K: 395 vs 292 tok/s). The decode gains without MTP are modest, +1-8%, but they’re consistent and grow with depth as the QSA gather kicks in past 16K.
MTP acceptance is prompt-dependent. The dip to 23.5 tok/s at 8K vs 38.5 at 4K and 31.7 at 16K tells you that speculative decoding is variable. Code prompts and structured text get high acceptance. Prose and diverse completions get lower acceptance. Your actual throughput will depend on what you’re generating.
How It Compares to Their Published Numbers
EngramHalo’s README claims 39.3 tok/s using the Unsloth UD-IQ3_XXS quant (a more aggressive ~3 bpw quantization) in RAM mode, where the entire 26.8 GB n-gram table is pinned in memory. That’s the best-case configuration: smaller quant means less work per token, RAM mode avoids any SSD latency.
My setup uses AtomicChat AD-4.27bpw (~4.27 bpw, larger quant) in SSD/mmap mode. The closest comparison in their table is the IQ4_XS SSD column, which shows 31.1 tok/s. My measured average across 1K-16K is about 30.3 tok/s, right in line with their numbers. The 38.5 peak at 3.4K is a code prompt with high MTP acceptance, which is the same condition under which they measured 39.3.
The published claims check out.
The Updated Configuration
Here’s what I’m running now. The key differences from the previous post: the container changed from llama-rocm-10.0-qwen38fn-hipcub to engramhalo, there’s a new -md flag for the MTP sidecar, --spec-type enables MTP and n-gram speculative decoding, -t 4 is per EngramHalo’s recommendation for mmap mode, and ROCBLAS_USE_HIPBLASLT=1 is required by their kernels.
macros:
"llama_server_engramhalo_base": >
/usr/bin/toolbox run --container engramhalo
env ROCBLAS_USE_HIPBLASLT=1
/usr/local/bin/llama-server --host 0.0.0.0 --port ${PORT}
"qwen38fn_model": >
~/Secondary/Models/Qwen3.8-Flash-Next/AD-4.27bpw/
Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/
Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf
"qwen38fn_mtp_sidecar": >
~/Secondary/Models/Qwen3.8-Flash-Next/mtp-Qwen3.8-Flash-Next-Q8_0.gguf
models:
"qwen3.8-flash-next":
cmd: >
${llama_server_engramhalo_base}
-m ${qwen38fn_model}
-md ${qwen38fn_mtp_sidecar}
--mmproj ${qwen38fn_mmproj}
-fa on -ngl 999 -b 2048 -ub 2048 -t 4
-fit off
--ctx-size 131072
--cache-type-k q8_0 --cache-type-v q8_0
--spec-type draft-mtp,ngram-mod
--spec-draft-n-max 4 --spec-draft-p-min 0.75
--jinja
--chat-template-file ${qwen38_fixed_template}
--reasoning-format deepseek
--reasoning-budget 8192
--reasoning-budget-message "Let me provide my answer now."
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--presence-penalty 0.0 --repeat-penalty 1.0
--chat-template-kwargs "{\"preserve_thinking\":true}"Everything from the previous post still applies: the froggeric fixed template, --reasoning-budget 8192, no --no-mmap, explicit --min-p 0.0, -fit off. The --spec-draft-p-min 0.75 threshold prevents MTP from regressing on prose by rejecting low-confidence draft tokens early.
Vision still works. The mmproj is loaded alongside MTP and I confirmed it processes image inputs correctly, MTP operates during decode and doesn’t interfere with the vision projector’s work during prompt processing.
Practical Takeaway
If you followed the previous post and are running the hipCUB toolbox at 15-20 tok/s, this is a straightforward upgrade. Clone, build, swap the container, add the MTP sidecar. Your model files, templates, and routing don’t change. You get roughly 1.5-2x decode speed at the context depths that matter for interactive use.
The one caveat is the 32K regression. If you’re running deep-context agent sessions that consistently fill 20K+ tokens of context, you won’t see the MTP gains and might see a slight regression. For that workload the kernel-only variant (without the -md and --spec-type flags) gives you the PP improvements without the decode tradeoff.
For everything else, chat, code generation, thinking prompts, vision queries, this is the fastest way to run Qwen 3.8 Flash Next on 128 GB Strix Halo right now.