.png){kind=icon}
Return
Off-Chip Memory Allocation for Neural Processing Units
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3352900.png)
.png){kind=icon}
.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Many modern Systems-on-Chip (SoCs) are equipped with specialized Machine Learning (ML) accelerators that use both on-chip and off-chip memory to execute neural networks. While on-chip memory usually has a hard limit, off-chip memory is often considered large enough to hold the network's inputs, outputs, weights, and any intermediate results that may occur during model execution. This assumption may not hold for edge devices, such as smartphones, which usually have a limit on the amount of memory a process can use. In this study, we propose a novel approach for minimizing a neural network's off-chip memory usage by introducing a tile-aware allocator capable of reusing memory occupied by parts of a tensor before the entire tensor expires. We describe the necessary conditions for such an off-chip memory allocation approach and provide the results, showing that it can save up to 33% of the peak off-chip memory usage in some common network architectures.
Keywords:
Tensors
Memory management
Resource management
Neural networks
Mathematical models
Deep learning
System-on-chip
NPU
memory allocation
neural network runtime
tiling
strip-packing problem
Journal
IF:
3.6
Papers:
9.8W
Citations:
29.4W
Organization
No organization information available
Cited Papers
ShareSaveMachine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods.png)
IEEE ACCESSOAAI
IEEE ACCESSIF3.6
OAAIShareSaveShareSaveShareSaveShareSaveShareSaveShareSave
View more