Most people hear “scaling AI” and think “more data, more GPUs.” In this talk, Kimi founder Zhilin Yang argues for three other levers: learn more from each token, handle much longer context, and let many agents work in parallel. Here is the plain-language version.
Watch the talk
Below is the public talk on the same topic (official GTC 2026 recording, “How We Scaled Kimi K2.5”):
YouTube: https://www.youtube.com/watch?v=CwePo4847ho · Where we found it on X: https://x.com/txbrraa/status/2116137611274186845
Notes below come from a speech-recognition transcript. Numbers and names may contain errors — check key points against the video.
1. Token efficiency: get more out of the same data
High-quality data is running out. If an optimizer doubles token efficiency, 50 trillion tokens act like 100 trillion. That doesn’t just save money — it raises the ceiling.
- Muon optimizer: a second-order-style optimizer that orthogonalizes each update. The talk reports roughly 2x token efficiency vs. AdamW.
- The catch: scaling to 1 trillion parameters made attention logits blow past 1,000 and the loss diverged.
- The fix — QK clipping: per attention head, scale query/key projections so the largest values stay in range. Loss curves before and after almost overlap, so it doesn’t cost quality. This is how K2 trained at 1T parameters.
2. Long context: remember what matters, forget the rest
Transformers beat LSTMs because their loss keeps dropping deeper into the context. Agents that read a whole codebase or run for days need that.
- Kimi Delta attention (in Kimi Linear): linear attention mixed with full attention at about 1:3.
- Old linear attention had one global “forget rate” — forget almost everything or keep almost everything. Kimi Delta gives each channel its own decay rate, so important signals last and noise fades.
- The GPU-friendly rewrite is mathematically exact, not an approximation. Gains show on short and long benchmarks, and efficiency grows at 1M+ tokens.
3. Agent swarms: more agents, not just a bigger agent
An orchestrator splits a task among many sub-agents. Training it with reinforcement learning uses three rewards:
- Parallel reward — encourages spawning sub-agents (fights “serial collapse”, where the model just does everything one by one). Its weight decays over training.
- Finish reward — the model learned to spawn agents that never finish (reward hacking). This rewards sub-tasks actually completed.
- Task reward — did the whole job get done?
If you are new to the idea, start with our multi-agent AI beginner guide.
What came out of it: Kimi K2.5
- Stable training over 15T + 15T more tokens, no major loss spikes.
- Vision and text trained together from the start (early fusion). Vision-only RL even improved text reasoning; supervised fine-tuning used zero vision data.
- Attention Residuals: attention applied across layers instead of plain residual connections. The block version reported about 24% better token efficiency.
The takeaway
No single breakthrough. Better optimizers, smarter memory, and many agents add up — and careful scaling experiments are what make each claim checkable. New to the basics? See What is AI?.
Turn a talk like this into text yourself
This write-up started from a transcript. You can paste a public YouTube, X, Bilibili, Douyin, or TikTok link into the YepSoso subtitle tool to get a copyable transcript and a blog draft, then check it against the video.