Embeddings hidden layers learning for neural network compression

Jia Cheng Hu

doi:10.1016/j.neunet.2025.107794

2025-07-01

Jia Cheng Hu

Roberto Cavicchioli

Alessandro Capotondi

DOI：10.1016/j.neunet.2025.107794

原文链接

原文求助

摘要

En 中文

• 嵌入可以学习隐藏层，而不会损害标记和学习的质量。 • EPS和VEPS能够实现极小模型的构建。 • ShareBERT在参数量减少21.9倍的情况下，达到了BERT的95.5%的精度。 • 为资源受限设备引入了轻量级ShareBERT变体。 • 在不同任务上均有效，并且与现有的压缩方法正交。

Keyword:

Parameter-sharing

Parameter reduction

Compression

Embeddings

BERT

期刊

Neural Networks

IF:

6.3

论文数:

8.2K

被引数:

3.0W

机构

暂无机构信息

引用论文

Right-truncatable Neural Word Embeddings

IF0

2016-01-01

PREAI

Jun Suzuki; Masaaki Nagata

Long Short-Term Memory 长短期记忆

Neural Computation

IF0

1997-11-01

PREAI

Sepp Hochreiter; Jürgen Schmidhuber

GhostBERT: Generate More Features with Cheap Operations for BERT

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

IF0

2021-01-01

OAAI

Zhiqi Huang; Lu Hou; Lifeng Shang; Xin Jiang; Xiao Chen; Qun Liu

Self-Critical Sequence Training for Image Captioning

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

IF0

2017-07-01

OAAI

Steven J. Rennie; Etienne Marcheret; Youssef Mroueh; Jerret Ross; Vaibhava Goel

Glove: Global Vectors for Word Representation 手套: 单词表示的全局向量

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

IF0

2014-01-01

PREAI

Jeffrey Pennington; Richard Socher; Christopher Manning

MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model MicroBERT：将基于MoE的知识从BERT迁移到一个更轻量级的模型

Applied Sciences

IF0

2024-07-16

OAAI

Dashun Zheng; Jiaxuan Li; Yunchu Yang; Yapeng Wang; Patrick Cheong-Iao Pang

LAD: Layer-Wise Adaptive Distillation for BERT Model Compression

SENSORS

IF3.5

2023-01-28

OAAI

Lin, Ying-Jia; Chen, Kuan-Yu; Kao, Hung-Yu

Placing search in context

Proceedings of the 10th international conference on World Wide Web

IF0

2001-04-01

PREAI

Lev Finkelstein; Evgeniy Gabrilovich; Yossi Matias; Ehud Rivlin; Zach Solan; Gadi Wolfman; Eytan Ruppin

Uncertainty-Driven Knowledge Distillation for Language Model Compression

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING

IF5.1

2023-01-01

PREAI

Huang, Tianyu; Dong, Weisheng; Wu, Fangfang; Li, Xin; Shi, Guangming

Channel Pruning for Accelerating Very Deep Neural Networks 用于加速非常深的神经网络的通道修剪

2017 IEEE International Conference on Computer Vision (ICCV)

IF0

2017-10-01

OAAI

Yihui He; Xiangyu Zhang; Jian Sun

查看更多内容