Latest developments

Illustration of glowing blue cubes linked like a neural network above a dark circuit board

ggml 1.0, the engine under llama.cpp, adds FP8 support and faster MoE paths on CUDA and Metal

The ggml tensor library that powers llama.cpp and many local AI apps reached version 1.0.0. The release adds FP8 support, fused mixture-of-experts work on CUDA and Metal, a Vulkan prefill FlashAttention path and a scheduler that skips unused inputs. It also removes deprecated APIs and bumps the RPC protocol to v9, which rejects older peers.

Why it matters

Faster MoE handling helps the large sparse models people now run on single GPUs, Macs and unified-memory PCs. If you split models across machines with llama.cpp RPC, update every node together, since v9 will not talk to older versions.

Source: ggml release notes (GitHub) github.com
  • software

More news

All news