Apple Workshop on Privacy-Preserving Machine Learning: Matrix Factorization DP-FTRL outperforms DP-SGD for cross-device federated learning and centralized training

Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements. While recent work has largely addressed KV cache reduction via compression and eviction along the temporal axis, we argue that the depth dimension offers…