4 posts
After learning to infer (252× GPU speedup), TatuEngine went for the Master-Apprentice cycle: a Qwen2.5-3B teacher with LoRA distilling structured reasoning ([THOUGHT]/[ANSWER]) into the BitMamba-2 1B student. Six versions in one day (v0.17→v0.22.2), 381 examples, and the convergence that wouldn't come until the system message fix.
Depois de aprender a inferir (GPU 252×), o TatuEngine partiu pro ciclo Mestre-Aprendiz: um teacher Qwen2.5-3B com LoRA destilando raciocínio estruturado ([THOUGHT]/[ANSWER]) pro student BitMamba-2 1B. Seis versões num dia (v0.17→v0.22.2), 381 exemplos e a convergência que não vinha até o fix do system message.
The saga of making Student BitMamba-1B learn without exploding: Adafactor vs AdamW, saving 4 GB of VRAM, and the 4 bugs that produced NaN during bf16 training.
A saga de fazer o Student BitMamba-1B aprender sem explodir: Adafactor vs AdamW, a economia de 4 GB de VRAM, e os 4 bugs que produziam NaN no treinamento em bf16.