Architecture

┌─────────────────┐      ┌──────────────────┐      ┌────────────────┐
│  Client / agent │ ───► │  toki (daemon)   │ ───► │  llama-server  │
│  (OpenAI API)   │      │                  │      │  (any backend) │
└─────────────────┘      └──────────────────┘      └────────────────┘
                         │  embedding + index │
                         │  SQLite metadata   │
                         └────────────────────┘

tokid (the daemon) is a small FastAPI service that sits in front of llama-server. It intercepts requests and runs the 5-step recycling algorithm:

1. Embed the incoming prompt.
2. Find the most similar cached prompt via cosine similarity.
3. Verify exact prefix match on token IDs.
4. If hit: route to the llama-server slot that already holds that prefix (cache_prompt=true); only the novel tokens are prefilled.
5. If miss: the normal full prefill runs, and that slot's KV stays in memory for the next request.

Targets

• ROCm + llama-server (first-class, current focus)
• AMD Ryzen AI MAX+ 395 APU with UMA (GGML_HIP_UMA=ON)
• gfx1151 (RDNA 3.5)
• Up to 128 GB unified memory

Why there is no disk layer

Upstream llama-server exposes slot save/restore endpoints behind --slot-save-path. Earlier builds used them to persist KV state to disk and restore it later. That was removed: restored KV corrupted precision in the benchmark (88% of divergent pairs were substantive, not cosmetic).

LlamaServerBackend still carries the endpoint helpers and patches/ keeps a historical patch for older llama.cpp, but the engine does not call cache_save / cache_load. Starting llama-server with or without --slot-save-path changes nothing about behavior.

Architectural improvements

Beyond the basic cache, the tool ships with:

1. Persistent slot pool, the most-reused prefixes get slots held across requests instead of being recycled after every turn. The pool is LRU-bounded: when it is full, the least-recently-used slot is evicted. Ownership is an in-memory routing hint; no save/restore round-trip is made.
2. Honest hit definition (MIN_REUSE_FOR_HIT = 10), a "hit" requires at least 10 tokens of prefix reuse. Below that, the lookup and routing cost exceeds the prefill saving, so it's reported as a soft_hit instead of inflating the hit rate.
3. Prefix-length-weighted similarity (BruteForceIndex.search), among candidates within 5% cosine similarity, prefers the one with the longest token list. Dampens the "cosine finds a 1-token lookalike" pathology.
4. Substantive divergence classification, output divergence is classified into identical / whitespace-only / restore-noise (a legacy name for KV-restore artifacts) / substantive via sentence-transformers semantic similarity. Reports the percentage of divergent pairs that are real (different meaning) vs cosmetic.
5. Embedding chunking fallback (LlamaServerBackend.embed), long prompts are split into overlapping segments and re-averaged, with a deterministic hash-based fallback when embedding fails entirely.

In-memory cache (current behavior)

Prefix reuse is in-memory only: the engine routes similar prompts to the same llama-server slot and lets the slot's KV cache serve the shared prefix (cache_prompt=true). This is lossless, outputs are byte-identical to a no-cache run. A restart, a slot spill or an eviction loses it. The cache/tree.py index and the .trkv sidecar remain in the codebase but are not on the request path, and disk persistence via ?action=save/?action=restore was removed after the benchmark showed 88% substantive divergence from baseline outputs.

Next: API reference, endpoints and auth.