Latency-Efficient LLM Split Inference With Token Reduction

Jiahui Zhao

doi:10.1109/tvt.2026.3742922

Latency-Efficient LLM Split Inference With Token Reduction

2026-10-12

0

PRE

AI

DOI：10.1109/tvt.2026.3742922

Original

Original request for help

Save

Abstract

En 中文

This paper investigates latency-efficient split inference for large language models (LLMs) in multi-user mobile edge networks. An offloading user executes the LLM layers preceding its split layer locally, performs token reduction at the split layer, and transmits the retained hidden states to an edge server, while a fully local user executes all layers on the device. We formulate a mixed-integer nonlinear program that minimizes the weighted sum of prefill latencies by jointly optimizing the split layers, token retention ratios, bandwidth allocation, and edge computing allocation under task-performance constraints. We then develop a heuristic alternating optimization algorithm in which the retention ratio is set to its minimum feasible value, the edge computing allocation is obtained in closed form, and the bandwidth allocation is determined by bisection. Simulations using token-reduction and task-performance models calibrated on Qwen2.5-0.5B-Instruct show that the proposed method reduces latency under the evaluated resource settings, achieving up to a 67.5% mean paired relative reduction versus Fixed Split.

Keywords:

Edge computing

large language models

split inference

token reduction

resource allocation

Latency-Efficient LLM Split Inference With Token Reduction

Abstract

Journal

Organization

Cited Papers