The headline of Ariya’s Kokoro writeup is quality, but the operational detail is the real story: the GPU stays fully reserved for LLM inference while an 82M-parameter model synthesizes multilingual speech on the CPU. In a voice agent, that’s not a nicety — it’s the difference between one accelerator per stage and one accelerator shared across the whole pipeline.
Anyone who has built a voice loop knows the latency budget is the enemy. ASR, the LLM turn, and TTS all compete for the same clock and often the same silicon. Pushing TTS to the CPU changes the arithmetic in a few concrete ways worth calling out:
- 🎯 GPU contention drops — speech synthesis stops fighting the LLM for VRAM and compute
- ⚡ 82M params is small enough to keep per-utterance latency low without a dedicated card
- 🔌 OpenAI speech-API compatibility makes it a drop-in swap for a cloud TTS call, not a rewrite
- 🔍 ~50 voices, multiple languages covers most conversational-AI needs without per-voice licensing
- 📦 Container ships pre-baked (~5GB) so the deploy story is a
podman run, not a model-download dance
The privacy angle gets the top billing in the post, and it’s real. But for production conversational AI the sharper win is that local, API-compatible TTS collapses a vendor dependency into a container you already control — and frees the GPU you’re paying for to do the one thing only it can. The HN discussion has people benchmarking real-time factors on modest CPUs, which is exactly the number that decides whether this ships.
If your voice stack still routes TTS through a hosted API, what’s actually keeping it there — quality, or inertia?