.png){kind=icon}
.png){kind=icon}
Return
Temporal modeling with reversible transformers
.png){kind=icon}
PRE
AI
DOI：10.1007/s00521-025-11707-3.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Memory efficiency is a critical bottleneck in deep learning models for sequence processing, particularly in long-range dependencies and continuous data streams. In the effort to solve this, we introduce a new architecture denoted as Reversible Temporal Transformer (TempVerseFormer). TempVerseFormer integrates reversible transformer blocks uniquely with a time-agnostic backpropagation strategy that decouples the memory footprint from the temporal depth and enables efficient training on long prediction time ranges. We validate our model on both a procedurally generated dataset designed to probe specific temporal dynamics and on standard spatio-temporal benchmarks, including KTH Action and TaxiBJ. The predictive accuracy of TempVerseFormer is competitive compared to other tested baselines, with memory consumption being practically independent of the time-to-predict. This substantial gain in memory efficiency, achieved while maintaining strong performance on synthetic and real-world benchmarks, places the TempVerseFormer as an indicative candidate for scalable temporal sequence modeling, allowing for real-time adaptation or video analysis in resource-constrained environments, and thereby leading to more fiscally responsible and temporally wise resource AI systems that are capable of working with changing and evolving environments.
Keywords:
Artificial intelligence
Deep learning
Memory efficiency
Reversible architectures
Transformer
Backpropagation
Journal
IF:
4.5
Papers:
838
Citations:
3.2W
