Qwen 3.8 Flash Next is a 125B MoE with only 6B parameters active per token. It ships with a 51B n-gram embedding table, native 262K context, vision support, and hybrid thinking. The architecture is qwen4exp, a preview of Qwen 4 that pairs Gated DeltaNet layers with Qwen Sparse Attention (QSA). It’s a wildly different model to run locally compared to dense transformers, and I’ve spent the last few days getting it dialed in on my 128 GB Strix Halo machine.

This post covers the full setup, including a quant selection that actually fits in memory, a critical HIP performance fix that most people are hitting without knowing it, chat template pitfalls that silently break agentic workflows, and the exact configuration I’m running in production. If you’re on Strix Halo (or similar 128 GB unified memory hardware), this should get you to a working setup without the detours I took.

#Hardware

ComponentSpecification
CPU/GPUAMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151)
RAM128 GB LPDDR5X unified memory (~218 GB/s)
SSDSamsung 990 Pro 2TB NVMe
OSFedora 44
BackendROCm 10.0 (HIP) via containerized toolbox
Model routerllama-swap → llama-server (llama.cpp build 10687)

#Why This Model is Different

Most LLMs you can describe with two numbers: parameter count and how much VRAM you need. Qwen 3.8 Flash Next has three distinct memory pools that behave completely differently:

  1. Active transformer weights (~54.5 GB): These need fast memory (GTT / GPU-accessible RAM). This is the part that acts like a normal model.
  2. N-gram embedding table (~38.4 GB): A lookup table of 20 million bigram/trigram entries, read once per forward pass at ~2.7 KB per token. At typical generation rates that’s ~3 MB/s of random reads. It does not need fast memory and pages cleanly from SSD via mmap.
  3. KV cache: QSA uses a fixed 512-block, 2048-token attention budget, so KV cache growth is sublinear. Going from 32K to 131K context costs only ~3 GiB more GTT, not the ~16 GiB you’d expect from a standard attention model.

This three-pool layout is the key to running a 125B model on 128 GB. You only need ~55 GB of fast memory for the weights that matter, the 38 GB n-gram table trickles in from NVMe, and the KV cache barely grows with context.

#Quant Selection: AtomicChat AD-4.27bpw

I tested two quants:

QuantbpwTotal SizeIn Fast MemoryFits 128 GB?
Unsloth UD-Q4_K_XL~4.0111 GB (4 shards)~79 GBBarely, crashes at 32K ctx
AtomicChat AD-4.27bpw4.2792.9 GB (33 shards)54.5 GBYes, 17-22 GB headroom

The Unsloth build pins embed/output tensors at Q8_0, which pushes the active weight footprint to ~79 GB. Add n-gram table pages and system overhead, and you’re left with 1-2 GB free at 8K context. At 32K it crashes outright.

The AtomicChat build isolates the n-gram table into its own shard that pages from SSD, needing only 54.5 GB in fast memory. Quality is excellent: KLD 0.0842 vs BF16, 89.49% top-1 token match, PPL ratio 1.026. It matches a 5.00 bpw build within measurement error while being 17.6 GB smaller.

If you have 128 GB, use the AtomicChat build. The Unsloth build is for systems with more memory or that can tolerate tiny context windows.

#The hipCUB Decode Cliff (Critical Fix)

This is the single most important thing in this post. Without this fix, your decode speed will collapse from ~20 t/s to ~5 t/s the moment context exceeds ~1,000 tokens, and you probably won’t realize why.

#The Problem

The qwen4exp architecture uses QSA (Qwen Sparse Attention) layers that call ggml_top_k with ne[0] == n_kv. Without the hipCUB library, the HIP backend’s supports_op function caps TOP_K and ARGSORT operations at ne[0] <= 1024. Once your context grows past ~1K tokens, every QSA layer (12 of them, every decode token) falls back from GPU to CPU, with a full GPU→CPU→GPU sync per layer per token. The result is a 3-4x throughput cliff that makes the model unusable for real work.

The fix is llama.cpp commit 5182cef and issue #27856. You need hipcub-devel and rocprim-devel installed at build time, and -DGGML_CUDA_USE_CUB=ON in your cmake flags.

#My Results With and Without the Fix

I built a custom toolbox from the kyuz0/amd-strix-halo-toolboxes Dockerfile.rocm-10.0 with hipCUB enabled. Here’s what the decode throughput looks like across context depths:

Context DepthWith hipCUB (tok/s)Without hipCUB (tok/s)Improvement
~185 tokens20.6~20baseline
~1K tokens20.16.13.3x
~3.4K tokens19.0~5.53.5x
~6.6K tokens18.8~5.53.4x
~13K tokens17.5~5.53.2x
~26K tokens15.3~5.03.1x

Without the fix, anything past a short prompt drops to ~5-6 tok/s. With it, you sustain 15-20 tok/s across the full working range. Prompt processing peaks at ~426 tok/s around 4K depth and gracefully declines to ~284 tok/s at 26K.

#How to Build the Toolbox

If you’re using the kyuz0 toolbox ecosystem, the quickest path is to modify the Dockerfile.rocm-10.0:

  1. Add hipcub-devel rocprim-devel to the dnf install in the builder stage
  2. Add -DGGML_CUDA_USE_CUB=ON to the cmake flags
  3. Build:
cd ~/Code/amd-strix-halo-toolboxes
podman build -f toolboxes/Dockerfile.rocm-10.0-qwen38fn-hipcub \
  -t localhost/llama-rocm-10.0-qwen38fn-hipcub:latest toolboxes/

Then create the toolbox container:

toolbox create llama-rocm-10.0-qwen38fn-hipcub \
  --image localhost/llama-rocm-10.0-qwen38fn-hipcub:latest \
  -- --device /dev/dri --device /dev/kfd \
  --group-add video --group-add render --group-add sudo \
  --security-opt seccomp=unconfined

If you’re building llama.cpp from source instead of using toolboxes, just make sure the two packages are installed and the cmake flag is set. The fix is in upstream master, no forks needed.

#Chat Template: Use froggeric’s Fixed Template

The GGUF-embedded chat template is byte-identical to Qwen 3.8 27B’s, and it has critical bugs for agentic use:

  • Defaults reasoning_effort to xhigh, which burns the entire token budget on reasoning and leaves zero tokens for actual content
  • Calls raise_exception on standard OpenAI values like "minimal" or "high", causing hard errors instead of fallbacks
  • Produces empty responses and infinite retry loops in coding agents (Pi, OpenCode, Claude Code)

Fix: use froggeric/Qwen-Fixed-Chat-Templates (v22.1+) with --reasoning-format deepseek. The fixed template defaults to medium effort and normalizes all client aliases without raising.

Download the template and point llama-server at it:

--jinja --chat-template-file ~/llama-swap/templates/chat_template.jinja
--reasoning-format deepseek

#Runtime Flags: What’s Different From Standard Strix Halo

If you’ve been running dense models on Strix Halo, your muscle memory for flags will trip you up here. Three things are different:

FlagStandard Strix HaloQwen 3.8 Flash NextWhy
--no-mmapRequiredDO NOT USEN-gram table must page from SSD via mmap
-fit offNot usedREQUIREDllama.cpp auto-fit mis-sizes this architecture
--min-p 0.0Usually defaultMust be explicitllama.cpp defaults min-p to 0.05 if unset, the model wasn’t trained with it

The --no-mmap one is the most dangerous. On dense models you want everything pinned in memory, but here the 38 GB n-gram table needs to be paged from disk. Using --no-mmap forces it all into RAM and you’ll crash.

#Reasoning Budget: Preventing Thinking Overflow

In long coding agent sessions, the model’s thinking can consume the entire output token budget. You get responses where reasoning_content has thousands of tokens of internal deliberation and the actual content is empty. Symptoms:

  • Coding agent sees stopReason: "length" with zero content
  • 400: request (N tokens) exceeds the available context size
  • Retry loops: “Your previous response was empty” repeating indefinitely

The fix is --reasoning-budget 8192, which caps how many tokens the model can spend thinking before it’s forced to produce an answer. Combined with --reasoning-budget-message "Let me provide my answer now.", you get a graceful transition from thinking to responding instead of a hard truncation.

#The Full Configuration

Here’s the exact llama-swap entry I’m running in production:

macros:
  "llama_server_qwen38fn_base": >
    /usr/bin/toolbox run --container llama-rocm-10.0-qwen38fn-hipcub
    /usr/local/bin/llama-server --host 0.0.0.0 --port ${PORT}
  "qwen38fn_model": >
    ~/Secondary/Models/Qwen3.8-Flash-Next/AD-4.27bpw/
    Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/
    Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf
  "qwen38fn_mmproj": >
    ~/Secondary/Models/Qwen3.8-Flash-Next/mmproj-Qwen3.8-Flash-Next-F16.gguf
  "qwen38_fixed_template": "~/llama-swap/templates/chat_template.jinja"

models:
  "qwen3.8-flash-next":
    cmd: >
      ${llama_server_qwen38fn_base}
      -m ${qwen38fn_model}
      --mmproj ${qwen38fn_mmproj}
      -fa on -ngl 999 -b 2048 -ub 2048
      -fit off
      --ctx-size 131072
      --cache-type-k q8_0 --cache-type-v q8_0
      --jinja
      --chat-template-file ${qwen38_fixed_template}
      --reasoning-format deepseek
      --reasoning-budget 8192
      --reasoning-budget-message "Let me provide my answer now."
      --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
      --presence-penalty 0.0 --repeat-penalty 1.0
      --chat-template-kwargs "{\"preserve_thinking\":true}"
    aliases:
      - "openai/qwen3.8-flash-next"
      - "local/Qwen3.8-Flash-Next"

Sampling parameters are from Qwen’s official model card:

ParameterThinking ModeNon-thinking Mode
temperature1.00.7
top_p0.950.80
top_k2020
min_p0.00.0
presence_penalty0.01.5
repetition_penalty1.01.0

#Steady-State Memory

Once the model is loaded and running:

ComponentUsage
GPU via GTT (active weights)~55-60 GB
N-gram table (mmap pages, variable)Up to 38.4 GB
System + other processes~10 GB
KV cache at 131K (QSA-bounded)~3.5 GB
Estimated free RAM~17-22 GB
Total system memory used~109 / 125 GB
Swap usage5.4 / 8.0 GB (not maxed)

The 131K context window is essentially free compared to 32K. QSA’s fixed attention budget means you’re paying about 3 GiB for the jump from 32K to 131K, not the 12+ GiB a standard attention model would cost. You could probably push to 262K (only ~8 GiB more) if you needed it.

#What’s Next

There are a few promising directions for pushing this further:

  • EngramHalo.cpp: A llama.cpp fork specifically tuned for Qwen 3.8 Flash Next on Strix Halo. It includes hipCUB, true QSA sparse gather, and working MTP. Community reports claim 39.3 tok/s, which would be nearly 2x what I’m seeing. I haven’t validated this yet.
  • STRIX_LEAN ROCmFP4: kingjones777’s Strix Halo-optimized quantization at 4.78 bpw with ROCmFP4 experts. Benchmarks show 22.87 tok/s at 8K context and verified 262K on 128 GB. Requires the ROCmFPX fork, not stock llama.cpp.
  • MTP speculative decoding: The model ships with a 4B MTP head, but as a 6B-active MoE it’s already fast enough that speculative decoding might not help (MoE models that exceed ~15-20 tok/s tend to get slower with speculation, not faster).