.png){kind=icon}
.png){kind=icon}
Return
Off-Chip Memory Allocation for Neural Processing Units
.png){kind=icon}
OA
AI
DOI：10.1109/ACCESS.2024.3352900.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Many modern Systems-on-Chip (SoCs) are equipped with specialized Machine Learning (ML) accelerators that use both on-chip and off-chip memory to execute neural networks. While on-chip memory usually has a hard limit, off-chip memory is often considered large enough to hold the network's inputs, outputs, weights, and any intermediate results that may occur during model execution. This assumption may not hold for edge devices, such as smartphones, which usually have a limit on the amount of memory a process can use. In this study, we propose a novel approach for minimizing a neural network's off-chip memory usage by introducing a tile-aware allocator capable of reusing memory occupied by parts of a tensor before the entire tensor expires. We describe the necessary conditions for such an off-chip memory allocation approach and provide the results, showing that it can save up to 33% of the peak off-chip memory usage in some common network architectures.
Keywords:
Tensors
Memory management
Resource management
Neural networks
Mathematical models
Deep learning
System-on-chip
NPU
memory allocation
neural network runtime
tiling
strip-packing problem
Journal
IF:
3.6
Papers:
9.7W
Citations:
29.4W
Organization
No organization information available