.png){kind=icon}
返回
Rethinking Resource Competition in Multi-Task Learning: From Shared Parameters to Shared Representation
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3429281.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
The core idea of Multi-Task Learning (MTL) is to develop neural networks with a shared feature extraction backbone and multiple prediction heads, each capable of inferring a different task simultaneously. Parameters in the backbone contribute to all tasks while those in the prediction heads contribute to only one or fewer tasks. Challenges arise when multiple tasks compete for resource. Existing methods focus on resource competition in shared parameters and proposed explanatory factors of task conflicts, task dominance, and gradient stability. However the fundamental nature of MTL is still understudied. In this paper, instead of following the existing methodology research directions, we carry out large-scale empirical study and provide deeper insight on understanding MTL. In particular, instead of focusing on resource competition in the shared parameters in the backbone, we shift our attention to resource competition in the backbone output, which is the embedded representation that is shared by all prediction heads. We show that the existing explanatory factors display weak causal relationship with model performance. We propose a novel measurement, which we term Feature Disentanglement, and show that understanding MTL problems from the perspective of how the shared representation is leveraged by different prediction heads, is a more faithful and reliable way than that from the perspective of how supervision signals from different tasks are interfering in the shared parameters. Additionally, it has been a commonly employed technique to replace gradients w.r.t. shared parameters with gradients w.r.t. shared representation for reduced computation. We conduct a comprehensive study and show that unless a theoretical analysis could be developed, there is not general guarantee that this fast approximation technique would work in practice.
Keyword:
Task analysis
Feature extraction
Training
Magnetic heads
Computational modeling
Explainable AI
Optimization
Attention mechanism
computer vision
deep learning
explainable AI
machine learning
multi-objective optimization
multi-task learning
neural networks
representation
robotics
期刊
IF:
3.6
论文数:
9.8W
被引数:
29.4W
机构
引用论文
FDG-PET/CT Imaging Predicts Histopathologic Treatment Responses after Neoadjuvant Therapy in Adult Primary Bone Sarcomas.png)
Sarcoma
OAAI
SarcomaIF0
OAAI分享收藏分享收藏Thermal comfort, perceived air quality, and cognitive performance when personally controlled air movement is used by tropically acclimatized persons当热带适应的人使用个人控制的空气运动时，热舒适性，感知的空气质量和认知表现Indoor AirOAAI
Indoor AirIF0
OAAI分享收藏Analysis of Whole-Brain Resting-State fMRI Data Using Hierarchical Clustering ApproachPLoS ONEOAAI
PLoS ONEIF0
OAAI分享收藏
查看更多内容