01 Distillation
Keep only the answers that pass their tests.
The chat model could barely write code: greedy decoding solved 5 of MBPP's 257 problems. So an open-weight teacher, Qwen2.5-Coder-1.5B-Instruct (Apache-2.0), answered 5,000 coding problems, and every answer was run against the problem's unit tests in a sandbox. Only the 3,277 that passed were kept. The chat fine-tune was then repeated with those added.
About 4x better, still far behind bigger models.
SantaCoder (1.1B) reports around 35% on MBPP and Qwen2.5-Coder-1.5B about 69%, with different settings. Quipu uses 149M parameters per token and saw 4.26B tokens. It is not competing with them yet.
The same idea, at a much bigger scale.
A 7B teacher, Qwen2.5-Coder-7B-Instruct (Apache-2.0), generated 638,342 answers, and every one was run against its unit tests in a sandbox. 498,813 passed and were kept: 78.1%.
Generation ran on a rented RTX 3090, the laptop and Modal H100s; verification ran on Modal CPUs.
The demo runs on a small CPU machine that sleeps when idle: the first visit can take about 15 seconds to wake it.