01 Next
What's next.
- donePretraining at 114M3B tokens in 45 hours on one laptop GPU. Weights are public.
- done1B mixture of experts with a code-aware tokenizer4.26B tokens, ten languages and code, $18.75 in total. The whitespace result above is why the tokenizer changed.
- doneChat fine-tune and execution-verified distillationHuman-written conversations, then teacher answers that pass their tests: MBPP 1.9% → 7.4%.
- doneDistillation v2A 7B teacher, 638,342 answers, every one tested; 498,813 verified answers kept (78.1%).
- doneData v3 and multi-GPU training10B tokens from 42 open sources for $2.51; data-parallel training verified on 2 GPUs; MoE dispatch 3.24x faster.
- nextContinued pretraining on data v3Monthly runs on free compute credits, with a decayed checkpoint released every two months.
- nextChat fine-tune on the verified answersAll 498k execution-verified answers from distillation v2.
- plannedLong-context phase: 2,048 → 8,192 tokensAbout 1B tokens of training at 8k context with a larger RoPE base (~500k). The demo already stretches to 4,096 without retraining, at under 0.3% loss cost on the first 2,048 positions; this trains the model on longer text instead of only stretching it.
- laterA ~1B-active modelWhen bigger compute is available.