Toki Recur documentation

A standalone KV cache reuse system for LLM inference. Drop-in prefix caching that works across CUDA, ROCm, SYCL, OpenVINO, Vulkan, Metal, and CPU without any model modifications.

What it does. When a new prompt arrives at an LLM server, the KV cache is rebuilt from scratch, even if 90% of the prompt is identical to one we just processed. Toki Recur detects exact prefix matches, reuses the saved KV state, and skips redundant prefill compute.

Real measured results

Run on an AMD Ryzen AI MAX+ 395 (Strix Halo, UMA, 125 GB RAM) with Qwen3.5-9B-Q4_K_M via llama-server, cache enabled vs. baseline:

Metric	Baseline	Recycled	Δ
Total latency (avg)	2307.8 ms	2198.1 ms	+5.0%
Time-to-first-token	353.4 ms	171.4 ms	+51.5%
Prefill tokens/sec	731.3	2819.3	+285.5%
Cache hit rate	0.0%	90.0%	+90.0%

The headline metric is TTFT, a 51.5% reduction, achieved by serving 2,241 of 2,575 prefill tokens from cache. Total wall-clock improves 5% because generation dominates the 64-token output; for prefill-bound workloads (long inputs, short outputs) the speedup is closer to 2×.

Guide

Install

pipx, llama-server binary, model management, platform support.

Quick start

Six commands to your first cached chat, plus small-RAM notes and opencode wiring.

Command reference

Every toki command: stack lifecycle, bench, cache, diagnostics, web UI.

Launcher & shells

The full-screen launcher, interactive shell, TUI chat, and web chat UI.

Architecture

The 5-step recycling algorithm, targets, and KV save/restore mechanics.

API reference

OpenAI-compatible endpoints, cache admin endpoints, auth, client wiring.

Benchmark

Workloads, scoring weights, and measured results on this hardware.

Production deploy

systemd/launchd units, auth + TLS for the daemon and web UI.

Development

Repository layout, test suite, and license.