.png){kind=icon}
.png){kind=icon}
Return
Transformer Meets Gated Residual Networks to Enhance PICU's PPG Artifact Detection Informed by Mutual Information Neural Estimation
.png){kind=icon}
PRE
AI
T
C
K
S
P
R
DOI：10.1109/tnnls.2026.3656756.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
This study delves into the effectiveness of various learning methods in improving Transformer models, focusing mainly on the Gated Residual Network (GRN) Transformer in the context of pediatric intensive care units (PICUs) with limited data availability. Our findings indicate that Transformers trained via supervised learning are less effective than MLP, CNN, and LSTM networks in such environments. Yet, leveraging unsupervised and self-supervised learning (SSL) on unannotated data, with subsequent fine-tuning on annotated data, notably enhances Transformer performance, although not to the level of the GRN–Transformer. Central to our research is analyzing different activation functions for the gated linear unit (GLU), a crucial element of the GRN structure. We also employ Mutual Information Neural Estimation (MINE) to evaluate the GRN’s contribution. Additionally, the study examines the effects of integrating GRN within the Transformer’s attention mechanism versus using it as a separate intermediary layer. Our results highlight that GLU with sigmoid activation stands out, achieving 0.98 accuracy, 0.91 precision, 0.96 recall, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$0.94~F1$ </tex-math></inline-formula>-score. The MINE analysis supports the hypothesis that GRN enhances the mutual information (MI) between the hidden representations and the output. Moreover, using GRN as an intermediate filter layer proves more beneficial than incorporating it within the Attention mechanism. This study clarifies how GRN boosters GRN–Transformer’s performance surpasses other techniques. These findings offer a promising avenue for adopting sophisticated models like Transformers in data-constrained environments, such as PPG artifact detection in PICU settings.
Keywords:
Clinical PPG signals
Gated Residual Networks (GRNs)
imbalanced classes
mutual information (MI)
transformers
Journal
IF:
8.9
Papers:
7.5K
Citations:
7.2W
Organization
Cited Papers