EASTER: Learning to Split Transformers at the Edge Robustly

doi:10.1109/TCAD.2024.3438995

EASTER: Learning to Split Transformers at the Edge Robustly

2024-11-01

0

PRE

AI

DOI：10.1109/TCAD.2024.3438995

Original

Original request for help

Save

Abstract

En 中文

Prevalent large transformer models present significant computational challenges for resource-constrained devices at the Edge. While distributing the workload of deep learning models across multiple edge devices has been extensively studied, these works typically overlook the impact of failures of edge devices. Unpredictable failures, due to, e.g., connectivity issues or discharged batteries, can compromise the reliability of inference serving at the Edge. In this article, we introduce a novel methodology, called EASTER, designed to learn robust distribution strategies for transformer models against device failures that consider the tradeoff between robustness (i.e., maintaining model functionality against failures) and resource utilization (considering memory usage and computations). We evaluate EASTER with three representative transformers-ViT, GPT-2, and Vicuna-under device failures. Our results demonstrate EASTER's efficiency in memory usage, and possible end-to-end latency improvement for inference across multiple edge devices while preserving model accuracy as much as possible under device failures.

Keywords:

Deep learning

Navigation

Computational modeling

Memory management

Transformers

Search problems

Robustness

Space exploration

Resource management

Integrated circuit modeling

Deep learning (DL)

design space exploration (DSE)

distributed inference

embedded system

robustness

EASTER: Learning to Split Transformers at the Edge Robustly

Abstract

Journal

Organization