01 Data v3
Ten billion tokens of pretraining data, built for $2.51.
The next pretraining batch: 10,000,003,101 tokens from 42 open, licence-checked sources, built on Modal CPUs. Benchmark problems (HumanEval and MBPP) were filtered out, as before.
What's in it
Mostly code and educational text, with maths, reference text, nine languages from FineWeb-2, and a small slice of the verified teacher answers from distillation v2.
02 Data · quipu-114m
Three billion tokens: mostly text, one-fifth code.
80% FineWeb-Edu, 20% code from github-code-clean. Only permissive licences are kept: MIT, Apache-2.0, BSD, ISC, CC0 and Unlicense. Minified files, vendored directories and files over 16k tokens are filtered out. That filter alone removed as many tokens as the entire code budget.
HTML is capped at 10% so markup doesn't crowd out the languages the model should actually learn to write.