New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — ...

Nvidia says it can shrink LLM memory 20x without changing model weights

Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory costs and time-to-first-token by up to 8x for multi-turn AI applications.
Semiconductor Engineering

Freeing Up Near-Memory Capacity For Cache Using Compression Techniques In A Flat Hybrid-Memory Architecture

A technical paper titled “HMComp: Extending Near-Memory Capacity using Compression in Hybrid Memory” was published by researchers at Chalmers University of Technology and ZeroPoint Technologies.
Nature

Cache Performance and Memory Hierarchy Optimization

The dynamic interplay between processor speed and memory access times has rendered cache performance a critical determinant of computing efficiency. As modern systems increasingly rely on hierarchical ...
EDN

Efficient checks for cache-coherency verification in complex SoCs

Cache, in its crude definition, is a faster memory which stores copies of data from frequently used main memory locations. Nowadays, multiprocessor systems are supporting shared memories in hardware, ...
Android Authority

What is cache memory - Gary explains

System-on-a-Chip (SoC) designers have a problem, a big problem in fact, Random Access Memory (RAM) is slow, too slow, it just can’t keep up. So they came up with a workaround and it is called cache ...
PC World

How does CPU memory cache work?

In the eighties, computer processors became faster and faster, while memory access times stagnated and hindered additional performance increases. Something had to be done to speed up memory access and ...