Cross-stack bench cleanup, 24/24 unanimous on every engine
2026-05-10 - v0.3.2 - improvement
Re-ran the full cross-stack matrix after patching two bench-driver bugs (C/TS token-decode fallback, vllm REPS=1 noise). All three engines ร six client languages now produce byte-identical Codec frames per cell, including vllm, which previously read as 0/24 unanimous in the post-mortem.
The 2026-05-08 cross-stack run had a ยง7 post-mortem flagging three sources of variance; vllm reading as 0/24 unanimous on Codec cells was the loudest. Ran it down to three issues:
1. C and TS demos were emitting 'tokens_emitted=0' for compressed cells (the 'tokens' field was only populated on 'identity' decode), which threw off the unanimity check.
2. vllm at 2 K tokens has ~10-20 % wire-byte variance from non-deterministic batching even at temperature=0; needs โฅ2 reps to land a stable median.
3. JSON-SSE rows have 10-16 B per-client framing-accounting drift that's structural, not noise.
Patches shipped (commits '7c12286' (https://github.com/wdunn001/Codec/commit/7c12286), 'eb574b6' (https://github.com/wdunn001/Codec/commit/eb574b6)) and the bench re-ran on the same lab box (vinez@192.168.1.88, 2ร RTX 3090).
Cross-language unanimity, all engines:
โโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Engine โ Codec cells unanimous โ Notes โ
โโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ sglang โ 24/24 โ clean โ
โ vllm โ 24/24 โ was 0/24 in earlier run โ
โ llama.cpp โ 24/24 โ only โค5 B drift remains on JSON-SSE rows โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Headline reduction at 2 K tokens (msgpack + gzip, Python row):
โโโโโโโโโโโโโฌโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโ
โ Engine โ JSON-SSE โ Codec msgpack + gzip โ Reduction โ
โโโโโโโโโโโโโผโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโค
โ sglang โ 485.2 KB โ 354 B โ 1,404ร โ
โ vllm โ 517.8 KB โ 3,874 B โ 137ร โ
โ llama.cpp โ 529.2 KB โ 16.1 KB โ 33ร โ
โโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
The 'Benchmarks' panel on the front page now points at the new run and the engineRows numbers are refreshed accordingly.
Links
MATRIX.md, 2026-05-09T17-09-35Z (https://github.com/wdunn001/Codec/blob/main/packages/bench/results/2026-05-09T17-09-35Z/MATRIX.md)
Earlier post-mortem (now resolved) (https://github.com/wdunn001/Codec/blob/main/packages/bench/results/2026-05-08T01-15-02Z/MATRIX.md)
Token-decode fix (commit) (https://github.com/wdunn001/Codec/commit/7c12286)