Binary mask tuning on gradient: Towards multi-data question answering

doi:10.1016/j.knosys.2025.113505

2025-06-01

PRE

DOI：10.1016/j.knosys.2025.113505

原文链接

原文求助

摘要

En 中文

预训练语言模型已被广泛应用于问答任务。为在未见数据集上实现更好的泛化能力，先前的一些研究通常在多个数据集上训练单个模型。然而，由于不同数据集间的噪声和分布差异，模型在多个数据集上训练时倾向于过度调整其权重。这种偏离初始预训练状态的情况导致模型在特定数据上表现优异，但在其他数据集上过拟合。最终，模型丧失了对新数据的泛化能力。本文从约束模型权重的角度出发，提出了一种新颖的微调方法——二进制掩码调优（BMT）。我们采用一个精心设计的、与数据分布密切相关的二进制掩码向量，用于屏蔽反向传播产生的梯度，从而从巨大参数空间中实现对拟合数据所需子空间参数的精确对齐。该方法旨在通过更精准的微调来增强模型对数据分布的适应性并提高参数效率。我们的实验表明，BMT不仅有效缓解了模型过度调整权重的倾向，还在不同数据集的问答任务中更好地捕捉了跨数据集规律和特定数据集属性。

Keyword:

Multi-data fine-tuning

Question answering

Binary mask tuning

Few-shot transfer learning

期刊

Knowledge-Based Systems

IF:

7.6

论文数:

1.3W

被引数:

4.5W

机构

暂无机构信息

引用论文

Zero-Shot Relation Extraction via Reading Comprehension

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)

IF0

2017-01-01

OAAI

Omer Levy; Minjoon Seo; Eunsol Choi; Luke Zettlemoyer

MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Proceedings of the 2nd Workshop on Machine Reading for Question Answering

IF0

2019-01-01

OAAI

Adam Fisch; Alon Talmor; Robin Jia; Minjoon Seo; Eunsol Choi; Danqi Chen

Efficient Parametrization of Multi-domain Deep Neural Networks

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

IF0

2018-06-01

OAAI

Sylvestre-Alvise Rebuffi; Andrea Vedaldi; Hakan Bilen

NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

IF0

2022-01-01

OAAI

Chuhan Wu; Fangzhao Wu; Tao Qi; Yongfeng Huang

Adaptive weighted over-sampling for imbalanced datasets based on density peaks clustering with heuristic filtering

Information Sciences

IF0

2020-05-01

PREAI

Xinmin Tao; Qing Li; Wenjie Guo; Chao Ren; Qing He; Rui Liu; JunRong Zou

UNIFIEDQA: Crossing Format Boundaries with a Single QA System

Findings of the Association for Computational Linguistics: EMNLP 2020

IF0

2020-01-01

OAAI

Daniel Khashabi; Sewon Min; Tushar Khot; Ashish Sabharwal; Oyvind Tafjord; Peter Clark; Hannaneh Hajishirzi

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

IF0

2017-01-01

OAAI

Mandar Joshi; Eunsol Choi; Daniel Weld; Luke Zettlemoyer

A backpropagation learning algorithm with graph regularization for feedforward neural networks

Information Sciences

IF0

2022-08-01

PREAI

Yetian Fan; Wenyu Yang

RACE: Large-scale ReAding Comprehension Dataset From Examinations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

IF0

2017-01-01

OAAI

Guokun Lai; Qizhe Xie; Hanxiao Liu; Yiming Yang; Eduard Hovy

Transformers: State-of-the-Art Natural Language Processing

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

IF0

2020-01-01

OAAI

Thomas Wolf; Lysandre Debut; Victor Sanh; Julien Chaumond; Clement Delangue; Anthony Moi; Pierric Cistac; Tim Rault; Remi Louf; Morgan Funtowicz; Joe Davison; Sam Shleifer; Patrick von Platen; Clara Ma; Yacine Jernite; Julien Plu; Canwen Xu; Teven Le Scao; Sylvain Gugger; Mariama Drame; Quentin Lhoest; Alexander Rush

查看更多内容