Architecture

┌─────────────────┐      ┌──────────────────┐      ┌────────────────┐
│  Client / agent │ ───► │  toki (daemon)   │ ───► │  llama-server  │
│  (OpenAI API)   │      │                  │      │   (ROCm/HIP)   │
└─────────────────┘      └──────────────────┘      └────────────────┘
                         │  embedding + index │
                         │  SQLite + blobs    │
                         └────────────────────┘

tokid (the daemon) is a small FastAPI service that sits in front of llama-server. It intercepts requests and runs the 5-step recycling algorithm:

1. Embed the incoming prompt.
2. Find the most similar cached prompt via cosine similarity.
3. Verify exact prefix match on token IDs.
4. If hit: load the saved KV state via ?action=restore and decode only the novel tokens.
5. If miss: full prefill, persist KV state via ?action=save for the future.

Targets

• ROCm + llama-server (first-class, current focus)
• AMD Ryzen AI MAX+ 395 APU with UMA (GGML_HIP_UMA=ON)
• gfx1151 (RDNA 3.5)
• Up to 128 GB unified memory

How KV save/restore works (no patch needed)

As of llama.cpp b3524 (mid-2025), the upstream llama-server already exposes the endpoints we need behind the --slot-save-path flag:

POST /slots/{id_slot}?action=save     body: {"filename": "..."}
POST /slots/{id_slot}?action=restore  body: {"filename": "..."}
POST /slots/{id_slot}?action=erase    body: {"filename": "..."}

The Debian-packaged llama-server 9413 already has these. LlamaServerBackend uses them directly. The patches/ directory contains a historical patch for users on older llama.cpp and a README.md explaining the situation.

Architectural improvements

Beyond the basic cache, the tool ships with:

1. Tree-structured KV cache, KV tensors are tracked in BLOCK_SIZE chunks. Two prompts sharing a prefix share the same on-disk blocks (content-addressed by token hash). Provides a fast tree-hit lookup that bypasses embedding entirely.
2. Persistent slot pool, the top-N most-frequently-used prefixes get dedicated llama-server slots that never clear. Skips the save/load cycle entirely for hot prefixes.
3. Honest hit definition (MIN_REUSE_FOR_HIT = 10), a "hit" requires at least 10 tokens of prefix reuse. Below that, the save/load overhead exceeds the prefill savings, so it's reported as a soft_hit instead of inflating the hit rate.
4. Prefix-length-weighted similarity (BruteForceIndex.search), among candidates within 5% cosine similarity, prefers the one with the longest token list. Dampens the "cosine finds a 1-token lookalike" pathology.
5. Substantive divergence classification, output divergence is classified into identical / whitespace-only / restore-noise / substantive via sentence-transformers semantic similarity. Reports the percentage of divergent pairs that are real (different meaning) vs cosmetic.
6. Embedding chunking fallback (LlamaServerBackend.embed), long prompts are split into overlapping segments and re-averaged, with a deterministic hash-based fallback when embedding fails entirely.

In-memory cache (current behavior)

Prefix reuse is in-memory only: the engine routes similar prompts to the same llama-server slot and lets the slot's KV cache serve the shared prefix (cache_prompt=true). This is lossless, outputs are byte-identical to a no-cache run. Disk persistence via ?action=save/?action=restore was removed after benchmarks showed 88% substantive divergence from baseline outputs; the .trkv sidecar layer remains in the codebase but is dormant.

Next: API reference, endpoints and auth.