.png){kind=icon}
返回
Transformer-BLS: An efficient learning algorithm based on multi-head attention mechanism and incremental learning algorithms
.png){kind=icon}
PRE
AI
DOI：10.1016/j.eswa.2023.121734.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Due to its efficient model calibration given by unique incremental learning capability, broad learning system (BLS) has made impressive progress in image analytical tasks such as image classification and object detection. Inspired by this incremental remodel success, we proposed a novel transformer-BLS network to achieve a trade-off between model training speed and accuracy. Specially, we developed sub-BLS layers with the multi-head attention mechanism and combining these layers to construct a transformer-BLS network. In particular, our proposed transformer-BLS network provides four different incremental learning algorithms that enable the proposed model can realize the increments of its feature nodes, enhancement nodes, input data and sub-BLS layers, respectively, without the need of the full-weight update in this model. Furthermore, we validated the performance of our transformer-BLS network and its four incremental learning algorithms on a variety of image classification datasets. The results demonstrated that the proposed transformer-BLS maintains classification performance on both the MNIST and Fashion-MNIST datasets, while saving 2/3 of the training time. These findings imply that the proposed method has the potential in significant reducing model training complexity with this incremental remodel system, while simultaneously improving the increment learning performance of the original BLS within such contexts, especially in the classification task of some datasets.
Keyword:
Broad learning system
Multi-head attention mechanism
Transformer structure
Incremental learning algorithms
期刊
IF:
7.5
论文数:
3.0W
被引数:
10.2W
机构
引用论文
Gradient-based learning applied to document recognition基于梯度的学习在文档识别中的应用.png)
PROCEEDINGS OF THE IEEE
PROCEEDINGS OF THE IEEEIF25.9
PREAI
分享收藏
查看更多内容