JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference

Chengyu Sun

doi:10.1145/3774934.3786428

JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference

2026-01-01

0

PRE

AI

DOI：10.1145/3774934.3786428

Original

Original request for help

Save

Abstract

En 中文

Long-context large language models (LLMs) have seen wide-spread adoption in recent years. However, during inference, the key-value (KV) cache-which stores intermediate activations-consumes significant memory, particularly as sequence lengths grow. Quantization offers a promising path to compress KV cache, but existing 2-bit approaches fall short of achieving optimal inference efficiency due to hardware-unfriendly algorithms and system implementations. We present JanusQuant, a 2-bit KV cache quantization system that achieves both high accuracy and end-to-end efficiency through algorithm-system co-design for long-context generation tasks. At its core is RtSmooth, a novel runtime smoothing quantization algorithm that mitigates outlier-induced accuracy loss via adaptive transformation. Building on RtSmooth, JanusQuant further enhances quantized inference with a series of optimizations: a fast absmax positioning technique for lightweight quantization, a memory-efficient data structure for managing recent tokens, and a custom mixed-precision attention kernel to accelerate computation. Across representative LLMs, JanusQuant preserves 99% of FP16 accuracy, reduces KV cache memory usage by up to 5.3x, and delivers up to 4.45x faster decoding throughput compared to state-of-the-art methods, while scaling efficiently to long-context inference.

Keywords:

Long-context Inference

KV Cache Quantization

JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference

Abstract

Journal

Organization