Colibri is a pure-C inference engine that runs GLM-5.2, a 744-billion-parameter Mixture-of-Experts model, on consumer hardware by streaming routed experts from disk. The key insight is that GLM-5.2 activates only ~40B parameters per token, and only ~11 GB of those change between tokens (the routed experts). The dense backbone (~9.9 GB at int4) stays resident in RAM while 19,456 experts (~370 GB total) live on your NVMe and get paged in on demand through an LRU cache.

I wanted to see what this looks like on my Strix Halo workstation with its 128 GB of unified LPDDR5X and a secondary PCIe 4.0 NVMe.

#Setup

ComponentSpecification
CPU/GPUAMD Ryzen AI MAX+ 395 / Radeon 8060S (Strix Halo)
RAM128 GB LPDDR5X unified memory
Model storage/home/zetaphor/Secondary (469 GB ext4, PCIe 4.0 NVMe)
Modelmateogrgic/GLM-5.2-colibri-int4-with-int8-mtp (~384 GB)
EngineColibri v1.0, built with gcc 16, ARCH=native (avx512-vnni)
OSFedora, kernel 7.x

The model download took about 70 minutes over HF Hub. All 100 main shards plus three int8 MTP head files, a tokenizer, and config landed at around 384 GB total.

#Build and Validation

cd ~/Code/colibri/c
./setup.sh          # gcc 16, 32 cores, OpenMP ok, ARCH=native
make test-c         # all C tests pass (avx512-vnni kernels validated)

coli doctor --ram 100 --json passed all required checks. The only warning is that the binary is CPU-only while an NVIDIA RTX 4070 Super is present, which is expected since colibri’s CUDA tier targets expert pinning (not useful here with only 1.7 GB usable VRAM after reserves).

#Resource Plan

coli plan reports the tiered placement for this machine:

TierRoleBudget
DiskCold backing (immutable)384 GB model on disk
RAMDense resident + warm expert cache100 GB budget, 56 slots/layer
VRAMHot experts (RTX 4070 Super)1.76 GB (93 experts), too small to matter

The expected bottleneck is disk expert misses, which is exactly what we see in practice.

#Disk Throughput

Before running inference I measured the NVMe with colibri’s iobench tool against a real model shard (19 MB random reads, 8 threads, 64 iterations):

ModeThroughput
Buffered2.89 GB/s
O_DIRECT2.99 GB/s

This PCIe 4.0 drive lands right between the community’s “PCIe4 NVMe ~3-5 GB/s” and “PCIe3” tiers. The README’s back-of-envelope table predicts 0.5-1 tok/s for this class of drive, which is consistent with what I measured.

#Benchmark Matrix

I ran a structured cold/warm matrix across the two main tuning knobs: TOPP (adaptive expert top-p, drops low-weight experts to reduce reads) and DRAFT (MTP speculative decoding depth). All runs used greedy decode (--temp 0), --ram 100, and a fixed prompt of 18 tokens generating 24 tokens.

ConfigCold tok/sWarm tok/sExpert hit rateMTP acceptanceRSS
Full router, MTP on (DRAFT=3)0.280.2859.6%52%89.3 GB
Full router, MTP off (DRAFT=0)0.310.3067.6%-85.7 GB
TOPP=0.7, MTP on (DRAFT=3)0.290.2959.6%52%89.3 GB
TOPP=0.7, MTP off (DRAFT=0)0.400.4268.7%-85.9 GB

Best result: 0.42 tok/s warm with TOPP=0.7 and MTP disabled.

#Analysis

#MTP hurts on this hardware

This surprised me. MTP speculation achieved a solid 52% draft acceptance rate (2.67 tokens per forward), which is within the range the project reports as effective. But each verified draft routes to additional experts (~914 expert loads/token vs ~331 with MTP off), and on a disk-bound system that extra I/O more than cancels the forward-pass savings.

The colibri README actually warns about this: “on a cold cache each verified draft routes to extra experts (~660 to ~1100 expert-loads/token), so speculation can be a net time loss until the cache/pin warms up.” On this drive at ~3 GB/s, the extra expert loads dominate.

#TOPP=0.7 is the main lever

Adaptive expert top-p drops low-weight experts from routing (~1.6x fewer reads according to the engine). This cuts expert loads from ~576/token down to ~331/token and directly translates to faster decode. The quality tradeoff is documented as “small” by the project, and the generated text remained coherent in my testing.

#The profile breakdown tells the story

Looking at the TOPP=0.7 DRAFT=0 warm run’s timing:

ComponentTime (s)Share
Expert disk wait22.740%
Expert matmul26.847%
Attention3.87%
Other3.46%

The system is roughly balanced between disk and compute, with a slight lean toward matmul-bound. This matches the community benchmark from a Strix Halo (#124) which reported 1.10 tok/s sustained, though that was on a different drive and with extended warmup.

#Why so much slower than the community Strix Halo result?

Issue #124 on the colibri repo reports 1.10 tok/s on a “Ryzen AI Max+ 395, 128 GB, SK hynix P41 PCIe 4.0” with DIRECT=1 PIPE=1 --topp 0.7 auto-pin. The difference comes down to:

  1. Drive speed: The P41 is a high-end PCIe 4.0 drive (~7 GB/s sequential, probably ~5-6 GB/s random). My Secondary NVMe measures 2.99 GB/s O_DIRECT.
  2. Warmup: That result was “sustained” after warmup. My matrix intentionally clears .coli_usage between config groups and only does one cold + one warm run per config, so the learning cache barely builds.
  3. DIRECT/PIPE: I didn’t enable DIRECT=1 or PIPE=1 in this matrix. Those would likely help.

A longer warmup session with DIRECT=1 PIPE=1 on this drive should push toward 0.5-0.7 tok/s based on the scaling patterns in the community benchmarks.

#Comparison to My Normal Stack

For context on what “0.42 tok/s from a 744B model” means relative to the models I normally run through llama-swap:

ModelActive paramsThroughput
Qwen3.5-35B-A3B (MoE)3B33.5 t/s
Qwen3.5-9B (dense)9B15.1 t/s
Qwen3.5-27B (dense)27B5.4 t/s
Gemma 4 31B + MTP31B22.9 t/s
GLM-5.2 via colibri~40B active0.42 t/s

It’s two orders of magnitude slower than my MoE daily driver, and 13x slower than a comparably-sized dense model. But GLM-5.2 is a 744B frontier-class model with 256 experts per layer and 19,456 total routed experts. The fact that it runs at all, producing coherent output at ~2.4 seconds per token on consumer hardware, is the point.

#What Would Make It Faster

Based on the colibri community data and the project’s scaling predictions:

  1. Faster NVMe: Moving to a PCIe 5.0 drive (Samsung 9100 PRO class, ~11 GB/s O_DIRECT) would likely 2-3x the throughput on cold reads.
  2. Extended warmup: The learning cache (DIRECT=1 PIPE=1 with many prompts) builds a .coli_usage histogram that pins hot experts in RAM. Several community users report going from 0.06 to 1.1 tok/s sustained over many turns.
  3. More RAM budget: With PIN_GB=60-80 and extended usage history, more experts stay resident and disk wait drops further.
  4. Future kernel work: The engine uses AVX-512 VNNI kernels on this CPU. Matmul accounts for 47% of decode time, so faster quantized matmul (AVX-512 BF16, or AMX on future chips) would push the matmul-bound ceiling higher.

#Running It Yourself

If you have a Strix Halo (or any x86-64 Linux box with >= 25 GB RAM and ~400 GB NVMe free):

git clone https://github.com/JustVugg/colibri.git
cd colibri/c && ./setup.sh

# Download the pre-converted model with int8 MTP heads
hf download mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
  --local-dir /path/to/fast/nvme/GLM-5.2-colibri-int4

# Best config for Strix Halo (128 GB, moderate NVMe):
COLI_MODEL=/path/to/fast/nvme/GLM-5.2-colibri-int4 \
  TOPP=0.7 DRAFT=0 DIRECT=1 PIPE=1 ./coli chat --ram 100

The first few prompts will be slow (~0.3 tok/s). As the learning cache builds over sessions, hot experts get pinned in RAM and throughput climbs. The engine literally gets faster the more you use it.

#Bottom Line

Colibri does exactly what it claims: it runs a 744B frontier model on consumer hardware by treating your SSD as the bottom tier of a memory hierarchy. On Strix Halo with a mid-range PCIe 4.0 drive, that works out to about 0.4 tok/s with the best settings, roughly 2.4 seconds per token. It’s not interactive, but it’s a real 744B model producing real output on a machine that costs less than a single H100.

The practical use case is long-form generation where you can fire off a prompt and come back to it, think overnight code reviews or batch document summarization. For interactive chat, my normal 27-35B models through llama-swap are 80-130x faster. But there’s something satisfying about watching a model with more parameters than GPT-4 answer questions on a desktop workstation.