Latest developments

Illustration of a dark terminal window beside a stream of glowing blue blocks compressing into smaller cubes

llama.cpp adds FP8 KV-cache support in build b12274

llama.cpp merged support for storing the KV cache, the memory a model uses to hold your conversation and documents, in the OCP FP8 format, with optional per-tensor scales. It shipped in build b12274 on November 10, and the project cites NVIDIA's Mistral-Medium-3.5-128B as a model that uses a scaled FP8 cache.

Why it matters

An 8-bit cache takes about half the memory of a 16-bit one, so long chats and big documents leave more room for the model itself on a GPU or unified-memory machine.

Source: llama.cpp release notes (GitHub) github.com
  • software
  • memory
  • gpu

More news

All news