Command reference

Shell

toki                  # run benchmark (default)
toki interactive      # enter interactive REPL

Chat interfaces

toki tui                                      # full-screen TUI chat (terminal)
toki tui --auth-token <token>                 # if daemon requires auth
toki tui --daemon-url http://other:8787       # custom daemon URL
toki diff --system "<long>" "<user prompt>"  # one-shot A/B comparison

# Web UI on http://127.0.0.1:7777 (production-ready single-page chat)
toki web                                      # foreground
toki web --detach                             # background, log → /tmp/toki/web.log
toki web --tls-cert cert.pem --tls-key key.pem  # HTTPS + wss, Secure cookies
toki web --port 8888 --daemon-url http://other:8787  # custom endpoint

In-chat slash commands (TUI chat): /mode raw|cached|compare · /system <text> · /clear · /metrics · /max <n> · /temp <f> · /raw <msg> · /cached <msg> · /compare <msg> · /quit

Stack lifecycle

toki up                        # foreground (Ctrl+C stops)
toki up --detach               # background, returns PIDs
toki up --skip-llama           # only start the daemon
toki up --model /path/to/other.gguf --ctx-size 16384  # custom args
toki down                      # stop everything
toki down -y                   # skip confirmation
toki status                    # show stack state

Model registry

toki model list                                 # registry + port + status
toki model status                               # per-model stack state
toki model use <name>                          # boot, wire clients at it
toki model use <name> --keep-others            # don't stop the other big stacks
toki model stop <name>                         # stop llama-server + daemon
toki model stop <name> --force                # stop even unmanaged stacks on the ports
toki model pull <hf-repo> --use                # download, register, and start
toki model pull <hf-repo> --file name.gguf    # pick a specific GGUF (or split)
toki model pull <hf-repo> --name my9b --ctx-size 16384
toki model remove <name>                      # drop from the registry
toki model remove <name> --purge             # also delete weights + runtime state
toki model remove <name> --force              # allow removing the active model
toki model repair                               # backfill binary/env on legacy entries

toki model pull detects multi-file GGUF splits (the upstream llama.cpp convention -NNNN-of-NNNN.gguf) and downloads every sibling part automatically, picking the recommended Q4_K_M variant no longer leaves llama-server unable to find the other shards. The model's registry entry always points at the first part (the entry point llama-server loads); the rest land beside it. toki model remove defaults to keeping weights and state on disk so you can re-add the same name with toki model pull without re-downloading; pass --purge to free the disk.

Benchmark (the headline tool)

toki bench   # interactive: pick a model from the registry + a workload from a menu

# Or fully scripted (registry name or path):
toki bench --model 9b --workload agent_long_context \
           --max-tokens 64 --output-json report.json

# Self-contained: spin up llama-server + bench together (auto-stop)
toki bench --model ~/Downloads/Qwen3.5-9B-Q4_K_M.gguf \
           --self-contained --ctx-size 8192 --n-gpu-layers 99

Available workloads:

Workload	Expected hit rate	Best for
`agent_long_context`	~90%	Agent systems with long fixed prompts
`agent_25 / 50 / 100`	~99%	Same pattern, 25 / 50 / 100 prompts
`rag_chunks`	~90%	RAG with shared document prefixes
`mixed`	~50%	Mixed chat traffic
`no_overlap`	0%	Baseline / worst-case validation
`realistic_mixed`	~95%	216 prompts across 5 real-world categories
`stress_adversarial`	varies	301 prompts with adversarial patterns

Cache management

toki stats                    # hit rate, bytes, entries
toki evict --target-mb 4096   # LRU evict to fit budget

Diagnostics

toki doctor                          # llama-server, ROCm, GPU
toki compare report.json report2.json # diff two bench reports

OpenAI-compatible API server

Running under toki up (default 127.0.0.1:8787):

POST http://127.0.0.1:8787/v1/completions
POST http://127.0.0.1:8787/v1/chat/completions
POST http://127.0.0.1:8787/cache/lookup
DELETE http://127.0.0.1:8787/cache/{id}
GET  http://127.0.0.1:8787/cache/stats
GET  http://127.0.0.1:8787/healthz

Next: Launcher & shells, the interactive UIs.