.png){kind=icon}
Return
Enhancing progressive ensemble learning via normalized extra-Gradient initialization
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2026.108733.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Deep learning models continue to demand significant computational resources as their scale grows. Progressive training, which incrementally enlarges models during training, offers an effective remedy. Among its most popular applications is progressive ensemble learning, a method trains ensembles of models in a stage-wise manner. To make progressive ensemble learning more efficient, a copy initialization method has been proposed to initialize newly added modules by duplicating parameters from earlier models. Nevertheless, copy initialization compromises training stability, and few works focus on designing novel initialization methods based on learning theory. To bridge this gap, we formulate progressive ensemble learning as a functional optimization problem. Building upon this theoretical framework, we introduce Normalized Extra-Gradient (NEG) Initialization, a novel method designed to enhance the performance of progressive ensemble learning. We deliver rigorous theoretical insights, including convergence guarantees and an Edge-of-Stability analysis, which explain how NEG Initialization benefits both enhancing training efficiency and stability. To validate the superiority of our method, we conduct proof-of-concept experiments on synthetic data, and apply progressive ensemble learning to Vision Transformer (ViT) models on ImageNet-200 and ImageNet-1K. Extensive empirical results highlight the advantages of our approach over baselines and underscore its broad applicability.
Keywords:
progressive ensemble learning
normalized extra-gradient initialization
deep learning
model training
functional optimization
Journal
IF:
6.3
Papers:
8.2K
Citations:
3.0W
Organization
View more organizations
Cited Papers
ShareSaveSelf-supervised anomaly detection in computer vision and beyond: A survey and outlook.png)
NEURAL NETWORKS
OAAI
NEURAL NETWORKSIF6.3
OAAIShareSaveSNN-BERT: Training-efficient Spiking Neural Networks for energy-efficient BERTNEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
ShareSaveProgressive fine-to-coarse reconstruction for accurate low-bit post-training quantization in vision transformersNEURAL NETWORKS
NEURAL NETWORKSIF6.3
PREAI
ShareSaveShareSaveExtragradient and extrapolation methods with generalized Bregman distances for saddle point problems
PREAI
ShareSave View more