Model Parallelism Optimization for Distributed Inference Via Decoupled CNN Structure

Jiangsu Du

doi:10.1109/TPDS.2020.3041474

Return

Model Parallelism Optimization for Distributed Inference Via Decoupled CNN Structure

2020-01-01

PRE

DOI：10.1109/TPDS.2020.3041474

Original

Original request for help

Save

Abstract

En 中文

It is promising to deploy CNN inference on local end-user devices for high-accuracy and time-sensitive applications. Model parallelism has the potential to provide high throughput and low latency in distributed CNN inference. However, it is non-trivial to use model parallelism as the original CNN model is inherently tightly-coupled structure. In this article, we propose DeCNN, a more effective inference approach that uses decoupled CNN structure to optimize model parallelism for distributed inference on end-user devices. DeCNN is novel consisting of three schemes. Scheme-1 is structure-level optimization. It exploits group convolution and channel shuffle to decouple the original CNN structure for model parallelism. Scheme-2 is partition-level optimization. It is based on channel group to partition the convolutional layers, and then leverages input-based method to partition the fully connected layers, further exposing high degree of parallelism. Scheme-3 is communication-level optimization. It uses inter-sample parallelism to hide communications for better performance and robustness, especially in the weak network connections. We use ImageNet classification task to evaluate the effectiveness of DeCNN on a distributed multi-ARM platform. Notably, when using the number of devices from 1 to 4, DeCNN can accelerate the inference of large-scale ResNet-50 by 3.21x, and reduce 65.3 percent memory footprint, with 1.29 percent accuracy improvement.

Keywords:

Parallel processing

Kernel

Convolution

Computational modeling

Optimization

Performance evaluation

Task analysis

Intelligent applications

distributed deep learning

distributed inference

model parallelism

decoupled CNN structure

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.