codec-llamacpp (Docker)
Pre-built llama.cpp server with the Codec patches applied and a control plane bolted on, in one GPU container. OpenAI-compatible. Smallest of the three.
Server
'codec-llamacpp' is the easy way to stand up a Codec-speaking inference server on top of llama.cpp. It's a pre-built Docker image bundling:
β’ 'llama-server', statically-linked CUDA binary built from the Codec fork (token-native binary transport on the OpenAI-compatible server, plus server-side ToolWatcher, streaming gzipΒ +Β brotliΒ +Β dict-zstd compression, 'Codec-Zstd-Dict' header negotiation, and '/codec/schema' endpoint).
β’ codec-supervisor, the same FastAPI admin sidecar as codec-sglang, handling model uploads, Hugging Face pulls, hot-swaps, and reverse-proxying the llama-server backend.
β’ Static linking ('GGML_BACKEND_DL=OFF', 'BUILD_SHARED_LIBS=OFF'). The CUDA backend is compiled into the binary, no '.so' plugins to load at runtime, no 'LD_LIBRARY_PATH' config.
This image is ~3.6 GB, an order of magnitude smaller than codec-sglang or codec-vllm because llama.cpp doesn't ship a heavy ML Python stack.
Quick start
Default boot downloads 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M' (~400 MB) and serves it.
code (bash):
docker run -d --gpus all
-p 8080:8080
-v codec-models:/models
-v llamacpp-cache:/root/.cache/llama.cpp
--shm-size 8g
wdunn001/codec-llamacpp:latest
code (bash):
# OpenAI-compatible (JSON-SSE)
curl http://localhost:8080/v1/completions
-H "Content-Type: application/json"
-d '{"model":"x","prompt":"Hello","max_tokens":20}'
# Codec wire format - msgpack frames of token IDs with dict-zstd
curl http://localhost:8080/v1/completions
-H "Content-Type: application/json"
-H "Accept-Encoding: zstd, br, gzip"
-d '{"model":"x","prompt":"Hello","max_tokens":20,"stream":true,"stream_format":"msgpack"}'
The negotiator honors the spec preference order 'zstd > br > gzip > identity' and picks the smallest. On Qwen2.5-0.5B-Instruct fp16 at 2Β K tokens, the dict-zstd path lands at 140Β B, 3,868Γ smaller than the JSON-SSE baseline (529Β KB), with TTFB ~40.8Β ms (within ~1Β ms of the JSON path on the same server).
llama-server ignores the 'model' field for routing (single-model-per-process), so '"x"' is fine.
β GPU prereq: NVIDIA Container Toolkit + '--gpus all'. The image is built for compute capability 'sm_86' (RTX 3090); use '--build-arg CUDA_DOCKER_ARCH=<arch>' if you rebuild for a different GPU.
Model spec: HF id or local file
'CODEC_INITIAL_MODEL' accepts two forms; the supervisor's 'LlamaCppBackend' picks '--model' vs '-hf' automatically:
βββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββ
β Spec β What llama-server gets β Use case β
βββββββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββ€
β 'Owner/Repo-GGUF:filename-glob' β '-hf Owner/Repo-GGUF:filename-glob' β Pull a quantized β
β β β GGUF from Hugging β
β β β Face directly. The β
β β β default. β
β '/absolute/path/to/file.gguf' β '--model /absolute/path/to/file.gguf' β Bind-mounted local β
β β β model file. β
βββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββ
Examples
HF GGUF id:
code (bash):
docker run --gpus all -p 8080:8080
-e CODEC_INITIAL_MODEL='Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M'
-v llamacpp-cache:/root/.cache/llama.cpp
wdunn001/codec-llamacpp:latest
Local .gguf file:
code (bash):
docker run --gpus all -p 8080:8080
-e CODEC_INITIAL_MODEL=/models/my-model.gguf
-v /path/to/my-model.gguf:/models/my-model.gguf:ro
wdunn001/codec-llamacpp:latest
Hot-swap via admin API:
code (bash):
curl -X POST http://localhost:8080/admin/load
-H "Content-Type: application/json"
-d '{"name":"Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M","allow_remote":true}'
Configuration
βββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β Variable β Default β Effect β
βββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββ€
β 'CODEC_INITIAL_MODEL' β 'Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M' β Model spec (HF id β
β β β ':filename-glob' or β
β β β absolute '.gguf' path). β
β 'CODEC_BACKEND_ARGS' β '--ctx-size 4096 --gpu-layers 999' β Verbatim arguments to β
β β β 'llama-server'. β
β β β '--gpu-layers 999' β
β β β offloads everything to β
β β β GPU. β
β 'CODEC_PORT' β '8080' β Supervisor port. β
β 'HF_TOKEN' β (unset) β Required only for gated β
β β β GGUF repos. β
βββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββ
Admin API
Source & links
β’ Image: 'wdunn001/codec-llamacpp:latest' (https://hub.docker.com/r/wdunn001/codec-llamacpp) on Docker Hub.
β’ Source: github.com/wdunn001/codec-supervisor (https://github.com/wdunn001/codec-supervisor) (see 'Dockerfile.llamacpp').
See also
β’ codec-sglang, same image story, sglang backend (best throughput on supported models).
β’ codec-vllm, same image story, vLLM backend.