A post on LocalLLaMA claimed Tencent’s Hy3 is “the real deal on 128 GB.” The model is a 299B MoE with ~17B active parameters per token, and the community has been running it on DGX Spark, M3 Max 128 GB, and Strix Halo boxes via IQ2_M quantization at around 90-100 GB on disk. The interesting part isn’t just that it loads, it’s that PR #25395 merged native hy_v3 support with draft-MTP speculative decoding into mainline llama.cpp, and DGX Spark users reported +27% throughput from the MTP head with --spec-draft-p-min 0.75.

I’ve had good results with MTP on Strix Halo before, 4.8x on Qwen 3.6 27B. So the question was whether Hy3’s single-depth MTP head would deliver a similar boost on this hardware, or whether the 94 GB model weight footprint would leave too little headroom for the speculative context.

#The Setup

ComponentSpecification
CPU/GPUAMD Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo)
RAM128 GB LPDDR5X unified memory (~218 GB/s)
BackendVulkan (Mesa RADV, GFX1151), cooperative matrix enabled
llama.cppMainline b10065, built from source with -DGGML_VULKAN=ON -DGGML_VULKAN_COOPMAT=ON
Modelsatgeze/Hy3-1M-GGUF, hy3-1M-MTP-IQ2_M.gguf (94 GB)
Templatechat_template_llamacpp.jinja from the same repo

The first obstacle was that my production toolbox (llama-rocm-7.2.4) didn’t recognize the hy_v3 architecture at all. PR #25395 only merged on July 14 and the toolbox container hasn’t been rebuilt since. So I pulled mainline llama.cpp, built it on the host with Vulkan (same approach as my earlier MTP experiments, minus the fork requirement since hy_v3 is now upstream), and ran the benchmark server directly.

#Baseline: It Works, and It’s Usable

Server launch:

~/llama.cpp/build-vk/bin/llama-server \
  --host 127.0.0.1 --port 18081 \
  -m ~/Secondary/Hy3/hy3-1M-MTP-IQ2_M.gguf \
  -fa on --no-mmap \
  --ctx-size 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1 --jinja \
  --chat-template-file ~/llama-swap/templates/chat_template_llamacpp.jinja

This loaded without issues. The model’s auto-fit placed layers across the available Vulkan device memory and system RAM. Three measured runs after a warmup, same prompt (“Explain lock-free ring buffers for systems programmers in one concise paragraph”), temperature=0, max_tokens=256:

RunCompletion tokensWall timetok/s
Warmup24324.94 s9.74
115915.23 s10.44
215915.21 s10.45
315915.57 s10.21

Stable throughput: ~10.2-10.5 tok/s.

For context, this is a 299B-parameter model generating at roughly the same speed as my Qwen 3.6 27B baseline (6.2 t/s) multiplied by 1.7x. That comparison isn’t entirely fair since Hy3 only activates ~17B parameters per token (it’s a massive sparse MoE), but it does mean the routing overhead plus the larger weight footprint don’t kill performance. Ten tokens per second is slow for interactive chat compared to my Qwen3.6-35B-A3B daily driver at 33+ t/s, but it’s perfectly usable for longer-form tasks where you can tolerate a few seconds of latency.

#MTP: The Part That Didn’t Work

The whole reason I downloaded this specific GGUF (the -MTP- variant from satgeze) rather than the MTP-less quants is that it ships with the draft head baked in, same as the MTP approach that gave me 4.8x on Qwen 3.6. The recommended flags from the PR discussion:

--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75

The p-min 0.75 is load-bearing here. Hy3’s MTP head is trained single-depth, and without the confidence gate, speculation is a net slowdown (38.6% acceptance measured by the PR author, vs 97.3% with the gate). On CUDA hardware, this translates to +26-37% decode speed. On Apple Silicon, it’s neutral (bandwidth-bound, the q8 MTP head reads offset the accepted drafts). On Strix Halo, I expected something in between.

What I got instead was an allocation failure:

ggml_vulkan: Device memory allocation of size 1283457024 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
failed to create MTP context

The MTP context requires additional compute buffers for the speculative verification pass, on top of the already-loaded 94 GB model weights and the KV cache. On Vulkan with unified memory, there simply isn’t enough device-addressable headroom left after the base model fills it.

#The Sweep

I tried everything reasonable to squeeze MTP in:

AttemptContextKV typeBatchnmaxResult
Default32768q8_0auto3OOM: failed to create MTP context
Reduced context8192q4_064/643OOM: same error
Minimal4096q4_032/321Hung in “Loading model” indefinitely
Explicit ngl 5008192q4_064/643OOM: failed to allocate Vulkan0 buffer
Explicit ngl 4008192q4_064/643OOM: same
Explicit ngl 3008192q4_064/643OOM: same
Explicit ngl 2008192q4_064/643OOM: same

The pattern is clear. Reducing -ngl doesn’t help because the model itself (94 GB) already fills device memory. The MTP verification graph needs a contiguous ~1.3 GB allocation that simply cannot be satisfied once the model is loaded. There’s no combination of context, KV precision, batch size, or offload ratio that creates enough room.

#Why This Is Different From Qwen MTP

My Qwen 3.6 27B MTP setup uses a 15.4 GB GGUF (Q4_K_M) or 29 GB (Q8_0). Even the larger Q8_0 leaves 100 GB of headroom on a 128 GB system. The MTP heads add maybe 500 MB. The speculative verification buffers fit trivially.

Hy3 IQ2_M at 94 GB leaves roughly 34 GB of addressable space after the model loads, but the Vulkan memory allocator can’t provide a contiguous 1.3 GB block from what remains. This might be a fragmentation issue, a Vulkan driver limit, or a genuine hard constraint on this hardware. The error is deterministic across every configuration I tried, which points to the latter.

The DGX Spark results (+27% with MTP) come from a system with 273 GB/s unified memory and presumably a CUDA allocator that handles the MTP context differently. The M3 Max MTP results (neutral, no speedup but no crash) come from Metal, which has its own memory model. Vulkan on Strix Halo is the first reported case where MTP simply refuses to initialize.

#What 10 tok/s Gets You

Hy3 at IQ2_M is a 2.7-bit quantization of a 299B model. The quality claims from the model card are genuinely interesting, needle-in-a-haystack certified to 786K context, thinking mode with reasoning_effort control, tool calling with a bespoke tag format. At 10 tok/s, this is viable for:

  • Long-form generation where you fire and forget (draft reviews, summaries, analysis)
  • Batch processing pipelines where latency isn’t the constraint
  • Quality comparisons against smaller models to see if 299B matters for your specific use case

It’s not viable as an interactive daily driver. My Qwen3.6-35B-A3B at 33 t/s through llama-swap gives a much better chat experience, and with MTP the 27B dense variant hits ~30 t/s for quality-sensitive work. Hy3 lives in a different niche: maximum parameter count that physically fits on consumer hardware, at a speed that’s slow but not unusable.

#What Would Change This

Three paths to faster Hy3 on this hardware:

  1. Smaller quant: The IQ1_M (62 GB) would leave substantially more headroom for MTP buffers. The quality tradeoff is steep (thinking mode reportedly collapses at 1.7-bit), but for baseline-only use it might be worth testing.
  2. ROCm instead of Vulkan: My Vulkan build hit the allocation wall, but ROCm has a different memory allocator that might handle the contiguous buffer request. Once the llama-rocm-7.2.4 toolbox picks up a build with hy_v3 support, this becomes testable.
  3. Upstream optimization: The MTP verification graph for this 192-expert model is likely unoptimized for memory. If future llama.cpp commits reduce the compute buffer requirements for large-expert-count architectures, MTP might become feasible without changing anything else.