Reddit threads and the halogen-flash-server repo have been loud about a dedicated inference engine for Qwen 3.8 Flash Next on Strix Halo: ~1200 tok/s prefill, ~42 tok/s decode, end-to-end claims of 2–4× faster than “everything else.” A backend roundup post lists halogen alongside other Strix Halo forks with similar headline numbers.

I installed halogen 0.5.8, pulled the dedicated .hgn weights (~121 GiB), stopped llama-swap, and ran a production-realistic A/B against my current Nathanw stack: 8k / 32k / 55k context bands, agentic and code prompt classes, thinking on, three repeats per cell. The headline numbers never lined up with what I actually run day to day.

Verdict: I kept Nathanw v0.7.5. Halogen is genuinely fast at cold prefill, but at the context depth my agents actually live in (~55k tokens), Nathanw decodes 17–70% faster depending on workload. The gap is not “halogen is broken.” It is that the community benchmarks and halogen’s README compare against a much slower baseline than what I actually run today.

#Background: Flash Next on This Box

If you have not followed my earlier posts: I run Qwen 3.8 Flash Next (125B MoE, 6B active) on a 128 GB Strix Halo workstation. The setup post covers quant selection (AtomicChat AD-4.27bpw), the hipCUB fix, and chat template traps. The EngramHalo post validated a community fork at 28–38 tok/s on shallow prompts, with decode regressing past 32k.

Production has moved on since those posts. My default backend is now Nathanw v0.7.5 on Vulkan via llama-swap, not stock llama.cpp and not EngramHalo. Strix Halo kernel work is consolidating in halo-box/strix-llama.cpp; Nathanw still ships portable releases from strix-halo-llamacpp while that migration finishes. The config that matters for this comparison:

  • MTP speculative decoding with Unsloth’s shared draft sidecar (--spec-type draft-mtp,ngram-mod --spec-draft-n-max 6)
  • Large prefill batch (-b 8192) and prefix cache reuse (--cache-reuse 1) for deep agent sessions
  • Mandatory mmap layout for the 51B n-gram table (--load-mode mmap, lazy tensor reads)
  • 262144 context, q8_0 KV, froggeric fixed template, --reasoning-budget 8192
  • QSA pooled-block-keys in v0.7.5 (indexer keys computed once per block instead of every decode step)

That stack measures ~47 tok/s warm decode at 55k agentic in controlled benchmarks and ~26–30 tok/s in a week of live agent traffic at 50k+ prompt depth. That matters when you read halogen’s 41.7 tok/s @ 32k and think it is twice as fast as what you could run on llama.cpp today.

#What Halogen Actually Is

Halogen is not another llama.cpp fork you drop into llama-swap. It is a standalone ROCm inference engine with its own weight format (peonist-ai/halogen-qwen3.8-flash-next, ~5.53 bpw) and a containerized server on port 8731. It pins ~68 GiB of weights, owns the GPU, and expects the host mostly to itself. You stop llama-swap, compact memory, run the pod, benchmark, tear it down.

That exclusivity is part of why the marketing numbers look good: no concurrent embedding models, no LiteLLM proxy, no mmap-heavy multi-profile juggling. Fair comparison, but not a drop-in replacement for my daily stack.

There is also a licensing split worth naming. The halogen-flash-server repo publishes Docker wiring, docs, and benchmark tooling, but the engine itself ships as a closed-source binary in the container image. You can run it and publish benchmarks under Peonist’s EULA, but you cannot inspect, patch, or upstream kernel fixes the way you can with llama.cpp. Nathanw’s path is the opposite: open releases on strix-halo-llamacpp, with active Strix Halo work landing in public on halo-box/strix-llama.cpp. On a platform that lives on community kernel churn, that matters as much as the tok/s tables.

#The A/B Setup

Nathanw (production)Halogen (candidate)
RuntimeVulkan v0.7.5, build 10677ghcr.io/peonist-ai/halogen-flash-server:0.5.8
WeightsAtomicChat AD-4.27bpw GGUFDedicated .hgn stack
Context262144, q8_0 KV262144 native, prompt cache mode 2
SpeculationMTP n-max 6 + ngram-modEngine-internal
Integrationllama-swap + LiteLLMStandalone :8731
Host IOMMUamd_iommu=pt (NPU required)Authors used amd_iommu=off (+13–16% prefill)

Benchmark protocol: I ran both backends through the same matrix: 8k, 32k, and 55k context bands; agentic and code prompt classes; thinking mode on; froggeric fixed chat template; --reasoning-budget 8192; three repeats per cell. Each cell uses the same padded prompt shape on both sides.

Two numbers matter and they measure different things:

  • Warm decode (repeat 2+, identical prompt): KV cache already holds the full context. This matches turn 2+ in a long agent session where the prompt grows slowly.
  • Cold prefill (repeat 1): full prompt fill from scratch. This matches the first message after load or a large prompt rewrite.

Nathanw timings come from llama-server’s built-in timings fields. Halogen timings come from HTTP wall clock on the same request shapes. Halogen depth cells used non-streaming HTTP because SSE streaming wedged the engine at 32k+.

#Warm Decode: Nathanw Wins at Depth

This is the number that decides agent throughput. Turn 2+ in a 55k-token session is where my daily workloads spend most of their wall clock.

BandClassNathanwHalogenDelta
8kagentic44.238.2−14%
8kcode46.251.5+11%
32kagentic38.624.1−37%
32kcode54.849.1−10%
55kagentic46.738.9−17%
55kcode119.435.3−70%

The 55k code row is the story. Nathanw with --spec-draft-n-max 6 and high MTP acceptance on structured output hit 119 tok/s. Halogen measured 35 tok/s on the same band. That is not a rounding error. It is MTP speculative decoding on code-shaped prompts at depth, which is exactly where the Nathanw stack is tuned to win.

At 55k agentic (my median live prompt depth), Nathanw still leads 47 vs 39 tok/s. Halogen only wins decode at 8k code (+11%), and even that disappears once you factor in integration cost.

Compare to real traffic: live agent requests at 50k+ averaged 26.4 tok/s (median), about 44% below the 55k agentic benchmark figure. Real sessions mix cold fills, partial rewrites, and variable MTP acceptance, so production always runs below a perfect cached repeat. Halogen’s published 41.7 tok/s @ 32k landed between 24 tok/s @ 32k agentic (halogen in this A/B) and 55 tok/s @ 32k code (Nathanw). Prompt class matters more than the marketing tables suggest, and halogen is not 2× ahead of a tuned stack at depth.

#Cold Prefill: Where Halogen Earns Its Hype

First fill of a long document is halogen’s win condition.

BandClassNathanwHalogenDelta
8kagentic4901082+121%
55kagentic362758+109%
55kcode408608+49%

758 vs 362 tok/s at 55k agentic is real, though not universal: halogen’s cold prefill at 32k agentic was 352 vs 439 tok/s (slower), likely from n-gram page-cache state. If your workload is “ingest a 50k-token doc once and ask one question,” halogen’s cold prefill advantage at depth is meaningful. Halogen also has aggressive prompt caching: repeat 2+ prefill on an identical prompt is near-instant (~3–7 s wall at 55k).

My agents do not look like that. They append a few hundred tokens per turn and decode for minutes. Warm decode dominates wall clock, and Nathanw wins there.

#Why the Community Numbers Look Better Than My Halogen Run

This is the core of the post. The hype is not fabricated. It is mis-aimed.

#Wrong baseline

Halogen’s README “Measured” section compares against EngramHalo, ROCmFP4, CIRU, and official llama.cpp, not a tuned Nathanw fork with MTP and production flags at 55k agent depth. The Strix Halo backend roundup cites ~50 tok/s decode and ~1200 tok/s prefill for halogen while listing ~20 tok/s decode for official llama.cpp on Flash Next. Kyuz0’s toolbox benchmark grid tells a similar story for general llama.cpp backends (Vulkan RADV/AMDVLK, ROCm variants) but has no halogen row and no Flash Next numbers at 55k agent depth. That matches what I measured back in August before switching to Nathanw and dialing in the config above.

When Nathanw already does 47 tok/s @ 55k agentic, halogen’s 42 tok/s @ 32k served is parity, not a revolution.

#Cold vs warm measurement

Halogen marketing emphasizes prefill (1424 tok/s @ 32k in author docs; >1000 tok/s in early user reports). My A/B agrees: halogen crushes cold prefill. But agent loops care about cached decode, and the v0.7.5 QSA pooled-block-keys fix specifically targets decode at depth. On my box, moving from v0.7.4.1 to v0.7.5 roughly doubled warm decode at 55k agentic (23 → 47 tok/s under the same benchmark protocol). That is the baseline halogen had to beat, not the 20 tok/s stock llama.cpp numbers in the README tables.

#MTP jackpot bands

Community decode numbers often come from code-class prompts at moderate depth where MTP acceptance is high. Nathanw at 119 tok/s @ 55k code is an MTP jackpot. Halogen at 35 tok/s on the same cell suggests the dedicated engine is not matching llama.cpp’s speculative path for structured generation at depth, or the harness shape (reasoning tokens, max_tokens=128) penalizes it differently.

#Host and weight confounders

FactorEffect
amd_iommu=pt vs offUp to −16% halogen prefill vs published; I keep pt because NPU breaks without IOMMU
Different quantsAD-4.27 GGUF vs halogen 5.53 bpw .hgn; not quality-matched
Exclusive memoryHalogen ~72 GB MemAvailable after runs vs Nathanw ~44 GB; different accounting, same 128 GB box
Streaming at 32k+Halogen SSE streaming wedged the engine (180s watchdog shutdown); depth cells used non-streaming HTTP

#Operations: Why I Cannot Just “Switch”

Even if decode were closer:

  1. Exclusive host: halogen stops llama-swap. No concurrent local models, no embedding server sharing the box.
  2. ~7 minute cold load: acceptable for batch, painful for interactive default.
  3. Streaming wedge: SSE hung at 32k+; clients must use non-streaming HTTP at depth.
  4. Separate weight tree: 121 GiB .hgn files alongside 93 GiB GGUF; not interchangeable.
  5. Closed engine: the inference binary is not open source; no path to patch, audit, or contribute fixes upstream.

Nathanw stays in my multi-model routing stack alongside embeddings and other local models. See local LLM infrastructure for the full setup.

#When Halogen Still Makes Sense

I am not declaring halogen useless.

  • Dedicated inference box with amd_iommu=off, no NPU, no mixed services.
  • Batch document ingest where cold prefill dominates (legal discovery, log dumps, RAG index builds).
  • Offline prefill pipelines where decode throughput is secondary.

For interactive agent loops at 50k+ cached context, Nathanw remains the correct default.

#Production Configuration

This is the full stack I benchmarked against halogen. It is what produced the Nathanw numbers in the tables above.

#Prerequisites

  1. Backend: Download the Nathanw v0.7.5 Vulkan portable release (strix-halo-llamacpp-vulkan-portable.tar.gz). Extract so llama-server lives at ~/Code/strix-halo-llamacpp-v0.7.5/llama-server. Watch halo-box/strix-llama.cpp for where active Strix Halo development lands next.
  2. Main model: AtomicChat AD-4.27bpw Q4_K_M-M64 (33-shard GGUF; point -m at shard 00001-of-00033).
  3. MTP sidecar: Unsloth shared MTP Q8_0 (~2.6 GB).
  4. Vision projector: mmproj-Qwen3.8-Flash-Next-F16.gguf from the same AtomicChat repo.
  5. Chat template: froggeric fixed template v22+ saved as ~/llama-swap/templates/chat_template.jinja.

Flash Next requires -fit off and mmap for the 51B n-gram table. Do not use --no-mmap. The --load-mode mmap block below is mandatory on Nathanw Vulkan.

#Standalone llama-server command

~/Code/strix-halo-llamacpp-v0.7.5/llama-server \
  --host 0.0.0.0 --port 8080 \
  -m ~/Secondary/Models/Qwen3.8-Flash-Next/AD-4.27bpw/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  -md ~/Secondary/Models/Qwen3.8-Flash-Next/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --mmproj ~/Secondary/Models/Qwen3.8-Flash-Next/mmproj-Qwen3.8-Flash-Next-F16.gguf \
  --image-min-tokens 1024 \
  -fa on -ngl 999 -b 8192 -ub 2048 -t 4 -fit off \
  --ctx-size 262144 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --cache-reuse 1 \
  --spec-type draft-mtp,ngram-mod --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --load-mode mmap --no-host --no-repack --tensor-read-lazy on \
  -ot per_layer_token_embd.weight=CPU --n-cpu-moe 0 --parallel 1 \
  --slot-save-path ~/llama-swap/slots \
  --jinja \
  --chat-template-file ~/llama-swap/templates/chat_template.jinja \
  --reasoning-format deepseek \
  --reasoning-budget 8192 \
  --reasoning-budget-message "Let me provide my answer now." \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --chat-template-kwargs '{"preserve_thinking":true}'

Sampling matches Qwen’s official model card for thinking mode. Nathanw documents the mandatory Flash Next memory flags in Running Qwen3.8-Flash-Next.

#Closing the Prefill Gap Without Halogen

Halogen’s cold prefill win (~2× @ 55k) points at the same bottleneck Nathanw still has: uncached PLE / n-gram page faults on diverse text. The upstream fix to watch is llama.cpp PR #28136 (lazy PLE direct reads). Author reports 2–3× real-world prefill on diverse prompts; it is not in Nathanw build 10677 yet. That is the path to halogen-like first-fill speed without abandoning llama-swap, MTP, or my existing quant.

Until that lands, my planning numbers stay:

ScenarioNathanw (measured)
Warm decode @ 55k agentic (benchmark)47 tok/s
Warm decode @ 50k+ (live traffic)26–30 tok/s
Cold prefill @ 55k agentic (first fill)362 tok/s
Cold prefill @ 13k+ (production median)450 tok/s

Halogen cold @ 55k was 758 tok/s. Impressive, but it does not buy back 119 vs 35 tok/s on the decode path my agents actually use.

#Takeaway

If you are on stock llama.cpp or early EngramHalo at 20–25 tok/s, halogen’s marketing compares fairly against you and the upgrade looks huge. If you have already put in the work to run Nathanw with MTP, large prefill batch, cache reuse, and the correct mmap flags for Flash Next’s n-gram table, you are the baseline halogen claims to beat, and on warm decode at production depth, Nathanw still wins.

Run the A/B on your box. Compare at your prompt depth and prompt class. Do not swap production on README tables that benchmarked EngramHalo and “official llama.cpp.”