01 Training code
Multi-GPU training and faster experts.
Everything so far was trained on one GPU. These two changes are for the runs after that.
verified · 14 Oct 2026
Multi-GPU data-parallel training.
Verified on real GPUs: the 2-GPU test passed. The same training code runs on one GPU or several.
mixture of experts
Expert dispatch rewritten with grouped matmuls: 3.24x faster.
Measured on a B200, in tokens per second.
before113k/s
grouped366k/s
02 Engineering
Bugs found before the long runs.
These bugs would not have shown an error. Each one was found while reviewing the code, before the run.
01 · speed
Attention was quietly using a slow kernel.
Fixing the fallback took the projected run for the original budget from ~65 hours to ~39.
02 · I/O
About 1 in 130 file renames fails transiently.
That is enough to kill a 50-hour run over a log write. Every atomic write now retries.
03 · resume
A resume could restart from zero, silently.
After days of training, with no error. It now refuses unless nothing could have been saved.
04 · numerics
One bad batch could have deleted every clean checkpoint.
Non-finite steps are now skipped, and three in a row stop the run before retention prunes the good ones.