.png){kind=icon}
.png){kind=icon}
Return
Learnable Cloud-Guided LLM Quantization for Resource-Constrained Edge Devices
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-032-10459-5_33.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Model quantization is crucial for deploying large language models (LLMs) on resource-constrained edge devices. However, in cloud-edge collaboration, edge devices (EDs) often lack the resources for on-device quantization. Moreover, existing Post-Training Quantization (PTQ) methods employ a static parameter approach, which fails to adapt to diverse local data distributions. To address these challenges, we propose a novel method, Learnable Quantization Guided by Distribution Correction (LQGDC), for generating an optimal, lightweight model that can be delivered to the ED for local inference within a cloud-edge collaborative framework. In this framework, edge devices upload a small amount of local data to the cloud as a calibration set. The cloud server then selects a suitable pre-trained model from a Model Pool and applies LQGDC to quantize the model. LQGDC introduces learnable parameters for weights, activations, and key-value (KV) cache. LQGDC employs a composite loss function that combines Mean Squared Error (MSE), cosine similarity, and Kullback-Leibler (KL) divergence to fine-tune parameters, thereby matching each devices unique data distribution. Experiments on seven datasets demonstrate that LQGDC outperforms all three current baselines in both language generation and zero-shot tasks. Specifically, when quantizing LLaMA-13B to W4A4KV4, LQGDC reduces average perplexity (PPL) by 1.92 and improves zero-shot task accuracy by 2.14% compared to the best baseline. This approach shows promise for single AI task implementation on resource-constrained EDs (e.g., complex voice command processing on smartphones, real-time visual defect detection on industrial drones, and document analysis with long-term context).
Keywords:
LLMs
Model quantization
Local Inference
Cloud-Edge Collaboration
Post-Training Quantization
Journal
N
IF:
0
Papers:
29
Citations:
0
View more organizations