Micron Document
Cross-stack bench cleanup, 24/24 unanimous on every engine

2026-05-10 - v0.3.2 - improvement


Re-ran the full cross-stack matrix after patching two bench-driver bugs (C/TS token-decode fallback, vllm REPS=1 noise). All three engines ร— six client languages now produce byte-identical Codec frames per cell, including vllm, which previously read as 0/24 unanimous in the post-mortem.


The 2026-05-08 cross-stack run had a ยง7 post-mortem flagging three sources of variance; vllm reading as 0/24 unanimous on Codec cells was the loudest. Ran it down to three issues:

1. C and TS demos were emitting 'tokens_emitted=0' for compressed cells (the 'tokens' field was only populated on 'identity' decode), which threw off the unanimity check.
2. vllm at 2 K tokens has ~10-20 % wire-byte variance from non-deterministic batching even at temperature=0; needs โ‰ฅ2 reps to land a stable median.
3. JSON-SSE rows have 10-16 B per-client framing-accounting drift that's structural, not noise.

Patches shipped (commits '7c12286' (https://github.com/wdunn001/Codec/commit/7c12286), 'eb574b6' (https://github.com/wdunn001/Codec/commit/eb574b6)) and the bench re-ran on the same lab box (vinez@192.168.1.88, 2ร— RTX 3090).

Cross-language unanimity, all engines:

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Engine โ”‚ Codec cells unanimous โ”‚ Notes โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ sglang โ”‚ 24/24 โ”‚ clean โ”‚
โ”‚ vllm โ”‚ 24/24 โ”‚ was 0/24 in earlier run โ”‚
โ”‚ llama.cpp โ”‚ 24/24 โ”‚ only โ‰ค5 B drift remains on JSON-SSE rows โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Headline reduction at 2 K tokens (msgpack + gzip, Python row):

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ Engine โ”‚ JSON-SSE โ”‚ Codec msgpack + gzip โ”‚ Reduction โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ sglang โ”‚ 485.2 KB โ”‚ 354 B โ”‚ 1,404ร— โ”‚
โ”‚ vllm โ”‚ 517.8 KB โ”‚ 3,874 B โ”‚ 137ร— โ”‚
โ”‚ llama.cpp โ”‚ 529.2 KB โ”‚ 16.1 KB โ”‚ 33ร— โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

The 'Benchmarks' panel on the front page now points at the new run and the engineRows numbers are refreshed accordingly.


Links

MATRIX.md, 2026-05-09T17-09-35Z (https://github.com/wdunn001/Codec/blob/main/packages/bench/results/2026-05-09T17-09-35Z/MATRIX.md)

Earlier post-mortem (now resolved) (https://github.com/wdunn001/Codec/blob/main/packages/bench/results/2026-05-08T01-15-02Z/MATRIX.md)

Token-decode fix (commit) (https://github.com/wdunn001/Codec/commit/7c12286)