llama.cpp adds FP8 KV-cache support in build b12274
llama.cpp merged support for storing the KV cache, the memory a model uses to hold your conversation and documents, in the OCP FP8 format, with optional per-tensor scales. It shipped in build b12274 on November 10, and the project cites NVIDIA's Mistral-Medium-3.5-128B as a model that uses a scaled FP8 cache.
Why it matters
An 8-bit cache takes about half the memory of a 16-bit one, so long chats and big documents leave more room for the model itself on a GPU or unified-memory machine.