Changes to llama.cpp for AMD's Ryzen AI Max+ 395 and two Radeon RX 7900 XTX eGPUs on Vulkan/RADV, each merged through a pull request with a before and after number from this one machine. The negative results are here too.
Three setups: Qwen3.8-Flash-Next, a 180B sparse MoE, layer-split across the integrated GPU and both XTX (40 tok/s end to end on real agent turns); dense Qwen3.8-27B pipelined over both XTX for many concurrent agent streams; and the 27B on one XTX. None of this is in upstream llama.cpp.
180B sparse MoE (512 experts, top-10) with hyper-connections and a sparse-attention indexer, C2T8 quant, MTP depth 3. Production splits the layers three ways: iGPU, XTX#1 (with the MTP drafter) and XTX#2. 39.9 tok/s end to end on real TB2.1 turns. It also runs on the iGPU alone at 28.5–29.6 tok/s (see below).
run-flashnext.sh · SPLIT=25,11,13 · -ub 256 · np1 · 256k context · q8_0 KVSPLIT=25,11,13 -ub 256 · 34 real TB2.1 turns, end to end| Same 34 TB2.1 turns | iGPU only | iGPU + two XTX |
|---|---|---|
| End to end | 28.5–29.6 tok/s | 39.9 tok/s (1.37×) |
| Decode | 36–40 tok/s | 52–55 tok/s |
| Prefill, real turns | 213–222 tok/s | 298 tok/s |
| Prefill, 10k-token prompt | 381 tok/s | 480 tok/s |
On the iGPU alone the chip sits at 93–94 °C and throttles; the low end of each range is a throttled run. Both layouts run the same build; commands under Reproduce it.
Where Flash-Next prefill time goes at ub 256 (before #62): MoE expert matmuls 40%, two skinny f32 hyper-connection products 23% (fixed by #62), dense projections 17%, attention 6%, elementwise 7%, the gated delta-net recurrence only 1.3%. Also merged: int8 coopmat1 MMQ ported from upstream (#61; flat on Flash-Next prefill), coupling and reasoning budgets with backend sampling (#59), per-command-buffer GPU timestamps (#60).
Findingsper decode step when slots in one verify batch draft different lengths. The batch splits into small ubatches. This, not gating itself, is why per-token p_min gating lost every test on this box.
of a Flash-Next decode step at np1 is the 8-bit dense path and the MoE experts respectively. With several streams the expert union takes over.
per token of context on top of a 41.53 ms step: Flash-Next decode at np1 is linear in KV length, with no cliffs.
decode from requantizing Flash-Next's 8-bit dense path, measured without a drafter and then in production with the MTP drafter at depth 3. Quality can match UD-Q4_K_XL (perplexity ratio 0.997), but drafter acceptance still drops 2–3 points, only at draft positions 2 and 3. A no-drafter benchmark overstates any requant of a model served with MTP. Write-up.
live draft acceptance after re-fitting the MTP head to the requantized model it serves. The head was trained against the bf16 weights; 20 minutes of training on the iGPU against C2T8's own top-20 distribution, unrolled over draft depths 1–3, recovers the loss and passes UD's 0.687. Write-up.
One copy of the weights layer-split 48/52 over both cards; two server instances on it run out of phase so both cards stay busy. Every stream reserves its own 131k KV. Built for throughput over many concurrent agent streams.
scripts/two-xtx/run-qwen38-2x.sh · 2 instances × np4 · MTP depth 2, drafters on card 1 · q4_0 KV · write-upaggregate decode on the same 8-stream test in one day: 115 → 185 tok/s from removing host stalls, then 259–269 from retraining the dock links. Draft acceptance stayed 0.85. Write-up.
Terminal-Bench 2.1 tasks scored by 77.5 minutes (34 passed), against 23 for the same two cards before these changes and 17 for one XTX. Decode per stream 23.7 vs 13.7 tok/s, prefill 434 vs 274 tok/s per request, same pass/fail outcomes.
was every host↔GPU transfer on both TB5 docks: the controller→GPU link inside each dock trained at 2.5 GT/s (Target Link Speed Gen1). Stepping the port's kernel bandwidth-controller cooling device 1→0 retrains it at 16 GT/s. Note.
of the decode threads' time, before #55, went to waiting on fences for transfers: 38% output readback from card 2 (the Vulkan host buffer was pinned for card 1 only), 33% before each cross-card copy, 12% synchronous input uploads.
Dense 27B, Q4_K_S, the whole model on one card. Single-stream speed for interactive use: DFlash2 or MTP drafting with confidence-gated depth and ngram drafts.
run-qwen38.sh · np1–np2 · these numbers predate the Gen4 dock retrainMeasured and recorded so nobody has to run them again.
| Idea | Result |
|---|---|
| iGPU · Flash-Next | |
| ngram-mod drafting on Flash-Next | −1.7% to −48% Negative in all five tests. A 512-expert top-10 MoE pays for the union of every wide verify batch. |
| Speculative lookahead: draft the next batch during verify | −7.1% On real traffic, after fixing four concurrency bugs in the prefetch worker. |
| Sparse flash attention at prefill | −18% at 16k, −10% at 32k Pays only past ~33k context (+7% at 64k). |
| Adaptive MTP depth on Flash-Next at np ≥ 2 | −5% to −9% per slot Fixed depth wins at every concurrency on this model. |
| Naive q4_K requant of Flash-Next's dense tensors | +6.4% perplexity For +23% single-stream decode, shrinking to about +7% at np4. |
| Importance-matrix requant of the dense tensors, served with MTP | +2% to +4% Quality equal to UD, but the drafter accepts less. A re-fit head wins the acceptance back (#52). |
| MTP draft depth 2 instead of 3 on Flash-Next at np1 | −5.3% On UD-Q4_K_XL (−0.9% on C2T8). |
Upstream MoE-aware mul_mat_id tile selection (ggml-org #29182) | −6% prefill At 4k and 16k on Flash-Next. Not picked. |
| A faster gated delta-net prefill kernel, as in the ROCm forks | ≤1.3% GDN is 1.3% of Flash-Next prefill here; halo-box measured +0.4–0.5% end to end from their Vulkan port. Not attempted. |
| MUL_MAT_ID tile chosen from rows per expert instead of total tokens | −10% to −30% Small tiles lose more than the padding at ~5 rows per expert costs. |
| MoE gather kernel on the down projection (k 640–768) | 361 vs 438 tok/s Slower than the matmul there; #62 keeps gather to k ≥ 1280. |
| Drafter temperature calibration and support truncation | no effect Temperature scale 0.7–1.5 and top-k 10/15/32 left acceptance unchanged. |
| One XTX · Qwen3.8-27B | |
| Scaling up a post-hoc n-gram (Engram-style) table for agent work | −0.3% perplexity On agent command blocks after 14.7 h of training (−5.7% on its training mix). Only 1.6% of TB2.1 steps were fixable by it. |
| Two XTX · Qwen3.8-27B pipelined | |
Device→device copies with no CPU wait (dma-buf bridge + sync_fd semaphore) | 113 vs 191 tok/s A queue waiting on another device's fence stalls that device. Once everything else is async, the blocking host-staged copy is cheap. |
| Importing host buffers into every GPU (userptr) | 26% of thread time in the submit ioctl amdgpu revalidates userptr pages on every command submission. |
| Micro-batching one context's decode over the layer split | 61.7 → 39.4 tok/s Every micro-batch re-reads the weights. Run several instances instead (#56). |
GPU sampling for the target (-bs) to cut the logits readback | acceptance 0.81 → 0.73 Throughput unchanged (117.8 vs 115.4 tok/s). |
| One compute queue per context; forced per-device ordering | no gain 185 vs 190 tok/s; ordering gave nothing once the stages were balanced. |
cmake -B build -DGGML_VULKAN=ON -DGGML_VULKAN_MMV_MAX_COLS=32 -DCMAKE_BUILD_TYPE=Release cmake --build build -j
MMV_MAX_COLS=32 matters for Flash-Next: the default of 16 puts a decode cliff at verify batches above 16.
# Flash-Next: model flags shared by both layouts FN="-m Qwen3.8-Flash-Next-C2T8-00001-of-00004.gguf \ -md mtp-Qwen3.8-Flash-Next-Q4DRAFT-refit0930.gguf --spec-type draft-mtp \ --spec-draft-n-max 3 --spec-coupled -ngl 99 -ngld 99 -fa on \ -ctk q8_0 -ctv q8_0 -lm mmap -lzm auto -c 262144 -np 1 --jinja" # iGPU only build/bin/llama-server $FN -dev Vulkan2 -devd Vulkan2 -ub 2048 # iGPU + two 7900 XTX (production): 24 / 11 + drafter / 12 + output layers GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 build/bin/llama-server $FN \ -dev Vulkan2,Vulkan0,Vulkan1 --split-mode layer --tensor-split 25,11,13 \ -devd Vulkan0 -ub 256
Device numbers are this machine's (Vulkan0/1 = the XTX cards, Vulkan2 = the iGPU); check yours with --list-devices. The tensor split counts 49 slots (48 layers + output). The vidmem variable is needed when a card came up with a 256 MiB BAR (hot-plugged dock).
Always pass -lm mmap for Flash-Next. With auto, one device that lacks mmap turns it off for all of them, and the load then needs the whole file in RAM.
# Qwen3.8-27B pipelined over two 7900 XTX (one endpoint on :8090) MODEL=Qwen3.8-27B-Q4_K_S.gguf DRAFT=mtp-Qwen3.8-27B-Q4_K_M.gguf TMPL=qwen38.jinja \ scripts/two-xtx/run-qwen38-2x.sh --reasoning-format deepseek -rea on --reasoning-effort medium
Check the dock links first: cat /sys/bus/pci/devices/*/current_link_speed on the GPU's upstream port should read 16 GT/s. See the dock note.