Latest developments

Illustration of memory modules feeding a glowing grid of attention blocks that connects to a graphics chip

llama.cpp v0.9.0 adds FP8 quantization, faster Gemma 4 small-model prompts and multi-node tensor parallelism

The v0.9.0 release notes list a new OCP FP8 E4M3 tensor type, a flash-attention change that makes the unified KV cache efficient for multi-sequence use, and large prompt-processing gains for Gemma 4 E2B and E4B. The bundled ggml update adds initial NCCL-based multi-node tensor parallelism and fused mixture-of-experts kernels for CUDA. New model support includes Mistral Large 4, and NVFP4 files now work for LFM3.

Why it matters

If you run llama.cpp directly, this is a worthwhile update for Gemma 4 small-model speed and for serving several chats at once. Multi-node support is early, but it points toward linking more than one machine for models too big for a single box. Check the release notes for your own backend before upgrading.

Source: llama.cpp Releases (GitHub) github.com
  • software
  • gpu
  • memory

More news

All news