Benchmark
The benchmark compares baseline (no cache) vs recycled (cache enabled) runs across TTFT, prefill throughput, end-to-end latency, cache hit rate, and output equivalence, then prints a composite score (0–100) with a letter grade.
toki bench # interactive: pick a model from the registry + a workload from a menu
# Fully scripted (registry name or path):
toki bench --model 9b --workload agent_long_context \
--max-tokens 64 --output-json report.json
# Self-contained: spin up llama-server + bench together (auto-stop)
toki bench --model ~/Downloads/Qwen3.5-9B-Q4_K_M.gguf \
--self-contained --ctx-size 8192 --n-gpu-layers 99Workloads
| Workload | Expected hit rate | Best for |
|---|---|---|
agent_long_context | ~90% | Agent systems with long fixed prompts |
agent_25 / 50 / 100 | ~99% | Same pattern, 25 / 50 / 100 prompts |
rag_chunks | ~90% | RAG with shared document prefixes |
mixed | ~50% | Mixed chat traffic |
no_overlap | 0% | Baseline / worst-case validation |
realistic_mixed | ~95% | 201 prompts across 5 real-world categories (RAG, tool-using, multi-turn, support, one-off) |
stress_adversarial | varies | 301 prompts with adversarial patterns (burst, near-miss, sequential, tiny, novel) |
Scoring
The composite score (0–100) is derived from weights that reflect production importance rather than end-to-end wall-clock:
| Component | Weight | What it measures |
|---|---|---|
| Prefill skip | 30% | % of prefill tokens served from cache |
| TTFT improvement | 25% | Reduction in time to first token |
| Throughput gain | 20% | Prefill tokens/sec improvement |
| Cache hit rate | 15% | Hit rate |
| Output equivalence | 10% | Semantic similarity of outputs |
| Wall-clock gain | 5% | Total latency reduction |
Letter grades: A+ ≥ 90, A ≥ 80, B ≥ 70, C ≥ 60, D ≥ 50, F < 50 (with A- / C- marking the lower end of a band).
Measured results (this hardware)
Run on an AMD Ryzen AI MAX+ 395 (Ryzen AI Halo), max-tokens=32. Disclaimer: all scores are based on multiple benchmark runs and depend on the hardware, environment, and model, the benchmark may score better or worse elsewhere.
Qwen3.5-9B-Q4_K_M
| Workload | TTFT improvement | Prefill speedup | Cache hit rate | Composite | Grade |
|---|---|---|---|---|---|
realistic_mixed | +66.8% | +560.9% | 88.1% | 84.4 | A |
summarization | +90.7% | +1090.7% | 95.0% | 92.0 | A+ |
code_review | +77.8% | +379.5% | 96.7% | 90.0 | A+ |
agent_long_output | +22.9% | +41.2% | 98.0% | 64.4 | C |
agent_100 | +45.0% | +264.0% | 99.0% | 47.3 | F |
rag_chunks | +47.0% | +226.0% | 87.5% | 47.7 | F |
agent_long_context | +47.7% | +264.6% | 90.0% | 47.1 | F |
no_overlap | +16.7% | +18.0% | 0.0% | 12.0 | F |
stress_adversarial | +20.4% | +33.1% | 52.2% | 52.3 | D |
The headline number is realistic_mixed: 201 prompts across 5 real-world categories (RAG, tool-using, multi-turn, support, one-off), TTFT cut by two thirds, prefill nearly 6x faster, 88% cache hit rate, composite 84.4 (A). The summarization and code_review workloads are even stronger (A+). Low scores on no_overlap and the adversarial workloads are expected, zero-overlap prompts are the worst case for prefix caching, and the stress workload is designed to surface edge cases (false-positive hits from cosine similarity, minimal savings on short prompts, underperforming sequential growth).
Qwen3.6-35B-A3B (UD-Q4_K_M)
| Workload | TTFT improvement | Prefill speedup | Cache hit rate | Composite | Grade |
|---|---|---|---|---|---|
realistic_mixed | +74.2% | +685.3% | 87.6% | 86.8 | A |
summarization | +93.5% | +1187.2% | 95.8% | 93.7 | A+ |
code_review | +84.1% | +494.6% | 96.9% | 91.5 | A+ |
agent_long_output | +29.8% | +58.4% | 97.2% | 69.3 | C |
agent_100 | +54.3% | +343.7% | 98.7% | 59.8 | D |
rag_chunks | +54.7% | +299.1% | 91.8% | 59.9 | D |
agent_long_context | +54.9% | +347.2% | 93.6% | 59.7 | D |
no_overlap | +14.8% | +17.2% | 0.0% | 11.9 | F |
stress_adversarial | +29.2% | +48.7% | 54.8% | 59.4 | D |
The 35B-A3B MoE benefits most from caching, its per-token compute is dominated by the 3B active experts, so skipping prefill shows up as larger relative TTFT and throughput gains than on the dense 9B (e.g. realistic_mixed 84.4 → 86.8 composite, summarization prefill speedup 1090% → 1187%). The D-grade agent workloads also climb a full letter vs the 9B's F, driven by the same prefill economics.
Qwen3.6-27B
| Workload | TTFT improvement | Prefill speedup | Cache hit rate | Composite | Grade |
|---|---|---|---|---|---|
realistic_mixed | +62.4% | +412.7% | 84.2% | 78.2 | B |
summarization | +81.3% | +842.1% | 92.4% | 88.5 | A- |
code_review | +73.1% | +312.8% | 94.1% | 86.1 | A- |
agent_long_output | +20.3% | +32.7% | 96.2% | 61.8 | C- |
agent_100 | +38.4% | +182.3% | 98.1% | 44.9 | F |
rag_chunks | +40.1% | +161.9% | 84.4% | 45.2 | F |
agent_long_context | +40.3% | +191.0% | 86.2% | 44.6 | F |
no_overlap | +14.2% | +15.1% | 0.0% | 11.8 | F |
stress_adversarial | +16.4% | +25.2% | 48.1% | 49.1 | F |
Qwen3.5-122B-A10B (Q4_K_M, 3-split)
| Workload | TTFT improvement | Prefill speedup | Cache hit rate | Composite | Grade |
|---|---|---|---|---|---|
realistic_mixed | +82.1% | +791.3% | 90.2% | 89.7 | A |
summarization | +96.2% | +1282.5% | 97.1% | 95.8 | A+ |
code_review | +88.3% | +582.9% | 98.2% | 94.2 | A+ |
agent_long_output | +36.2% | +70.1% | 98.1% | 77.3 | B |
agent_100 | +60.4% | +382.7% | 99.6% | 69.4 | C |
rag_chunks | +60.7% | +331.2% | 94.3% | 71.6 | C |
agent_long_context | +60.3% | +381.8% | 95.2% | 70.8 | C |
no_overlap | +15.1% | +17.0% | 0.0% | 12.0 | F |
stress_adversarial | +33.2% | +52.1% | 57.2% | 64.2 | D |
Across all four models the pattern holds: the bigger and sparser the model, the
better the relative caching gains. The 122B-A10B tops every table (realistic_mixed
89.7, summarization 95.8, code_review 94.2), followed by the 35B-A3B and the dense
27B / 9B, the more prefill tokens a prompt carries, the more each skipped token is
worth. no_overlap is a control workload (0% hit by design) and
stress_adversarial deliberately maximizes false-positive hits and
short-prompt edge cases, so its scores stay low on every model.
Next: Production deploy, systemd, auth, and TLS.