Google turned Gemma into a diffusion model without training from scratch
The new technical report on DiffusionGemma: convert a pretrained autoregressive model into a parallel text diffusion model for under 10% of the original training budget, and keep the accuracy.
Google rebuilt Gemma as a diffusion model for under 10% of its original training budget. Weights are Apache 2.0.
Google DeepMind published the technical report for DiffusionGemma in August, and the method matters more than the model. Instead of training a text diffusion model from scratch, the team converted the existing Gemma 4 26B A4B checkpoint into one. The conversion cost less than 10% of the original pretraining token budget, and the result generates around 1,500 tokens per second on a single H100 at accuracy comparable to the source model. The weights are Apache 2.0 on Hugging Face.
How the conversion works
Autoregressive models write one token at a time, left to right. DiffusionGemma refines blocks of 256 tokens in parallel instead, drafting and correcting a whole span at once. The retrofit runs in two stages: supervised fine-tuning teaches the model to reconstruct clean text from noised text, then a combined sampler distillation and reinforcement learning stage trades off output quality against sampling speed. Because generation is bidirectional inside a block, the model can fix an early mistake after seeing what came later, something a left-to-right decoder structurally cannot do.
Why the budget number is the story
Text diffusion has looked promising for years, but training a frontier-size diffusion model from zero priced almost everyone out of trying. A conversion recipe at under a tenth of pretraining cost changes who can afford the experiment: any lab or company sitting on a good autoregressive checkpoint. The report says the released model already runs in production at Interfaze for speech recognition and in radiology research.
Why a build studio cares
Speed is a feature users feel. 1,500 tokens per second is the difference between streaming a reply and just showing it, and 3 to 4x cheaper serving changes which product ideas clear the unit-economics bar. The honest caveat: diffusion language models are young, the tooling around them (structured output, tool calls, long context behavior) is thinner than for standard decoders, and Apache weights do not fix that by themselves. We would prototype on it today and think hard before betting a production feature on it this quarter.
Next step: read The Decoder's writeup and the technical report, then pull the weights from Hugging Face and time it against your current model on your own prompts. If you want help running that comparison for a real workload, write to us at hello@gattyworks.com.