.png){kind=icon}
.png){kind=icon}
Return
BATCH-DNN: Adaptive and Dynamic Batching for Multi-DNN Accelerators
.png){kind=icon}
PRE
AI
DOI：10.1007/978-3-031-99857-7_8.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Multi-DNN accelerators enable the simultaneous execution of multiple DNN workloads which improves performance by overlapping computations and memory accesses of multiple DNN workloads. However, on-chip memory must accommodate the footprint of all workloads. Batching allows DNN inferences using the same model to share weights which improves weight reuse and reducing off-chip access costs over a batch. Batching determines the batch size statically, leading to stalls when there is not enough on-chip memory available at runtime. This paper introduces BATCH-DNN, a dynamic method for adapting batch size on a layer-by-layer basis to available on-chip memory. It employs two techniques: adaptive cascaded sub-batching and adaptive sub-batch merging. Offline profiling establishes the footprint, while runtime adjustment establishes the maximum batch size on a layer-by-layer basis based on available on-chip memory. BATCH-DNN can improve the utilization of accelerator compute fabrics by 60%, which increases throughput by up to 27% and by 6%, on average, for batched multiDNN workloads.
Keywords:
Multi-DNN accelerator
Batching
On-chip memory anagement
Journal
E
IF:
0
Papers:
24
Citations:
0