.png){kind=icon}
Return
Drop2Sparse: Improving Dataset Distillation via Sparse Model
.png){kind=icon}
PRE
AI
DOI：10.1109/TCSVT.2025.3552047.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
The success of modern deep learning algorithms requires large amounts of training data, which leads to high computational and storage costs. Dataset Distillation (DD) is a rising research field that resolves this issue by synthesizing a compact training dataset from a large one. Recent gradient matching DD methods have achieved remarkable results. However, these methods typically utilize weak models for DD performance improvement, while well-trained models are often considered inferior choices due to their lower performance. Conversely, our study provides new insights into the role of well-trained models in DD, particularly under high-storage budget scenarios. We identify a previously overlooked design principle—a positive correlation between model capability and storage budget. Based on this principle, we propose Drop2Sparse, an approach that randomly sparsifies well-trained models to create efficient models for various storage budget scenarios. Drop2Sparse concurrently infuses significant model diversity and regularization effects into DD, outperforming previous state-of-the-art methods by up to 3.8% on CIFAR and 3.6% on ImageNet-subset. Moreover, our method exhibits remarkable cross-architecture generalization and achieves promising results even under challenging scenarios, such as using an extremely reduced model pool or highly accelerated training.
Keywords:
Dataset compression
dataset distillation
gradient matching
model sparsification
Journal
IF:
11.1
Papers:
845
Citations:
3.1W
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSaveShareSave
ShareSave
ShareSave
View more