Development

Repository layout

tokirecur/
├── src/tokirecur/
│   ├── core/        # The 5-step algorithm, recycling engine
│   ├── index/       # BruteForce + HNSW index, SQLite metadata
│   ├── backend/     # llama-server HTTP adapter (uses upstream endpoints)
│   ├── cache/       # KV blob storage (mmap'd, header-versioned)
│   ├── api/         # FastAPI: OpenAI-compatible + admin endpoints
│   ├── bench/       # The benchmark CLI (toki bench)
│   └── cli/         # `toki` admin CLI
├── patches/         # Historical patch for older llama.cpp
├── docker/          # ROCm Dockerfile + docker-compose
├── scripts/         # build_llama_cpp.sh
├── tests/           # unit, integration, benchmarks
└── benchmark/       # configs, reports, corpus

Tests

pytest tests/unit/ -v        # fast (seconds)
pytest tests/ -v             # includes integration (needs a live llama-server)

375+ unit tests cover the recycling engine, the index, the rate limiter, the tree-cache wiring, the persistent-slot pool, the .trkv sidecar format, the CLI config writers, and the OpenAI-compatible API (chat, completions, and Responses). CI runs the unit-test matrix on Python 3.11 / 3.12 / 3.13 on every push and PR, and executes the Linux and macOS system installers end-to-end.

Benchmark harness

Run on AMD Ryzen AI MAX+ 395 with Qwen3.5-9B-Q4_K_M via llama-server 9413, --slot-save-path /tmp/toki/slot-save:

License

MIT.