.png){kind=icon}
.png){kind=icon}
Return
Optimizing Timeliness for Distributed Stream Processing via Coflow Transmission
.png){kind=icon}
PRE
AI
DOI：10.1109/TON.2024.3517718.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Distributed stream processing has recently gained much interest due to the need of extracting meaningful results from continuous data stream. To keep the extracted results fresh, the underlying network flows are often required to transmit packets continuously. Otherwise, these results will become stale, and their staleness is determined by the slowest flow. At this point, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">coflows</i> can be semantically comprised. Hence, efficient coflow transmission is critical for streaming applications. However, prior coflow-based solutions have significant limitations. They use a one-shot performance metric—CCT (coflow completion time), which cannot continuously reflect the staleness of the output results for a streaming application. To this end, we propose a new performance metric—<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">coflow age</i> (CA), for coflows generated by distributed streaming applications. The CA tracks the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">longest time-since-last-service</i> among all flows in a coflow. In such a context, we consider a data center network with multiple coflows that continuously transmit packets between their source-destination pairs and address the problem of minimizing the average long-term CA while simultaneously satisfying the throughput constraints from the coflows. To solve this problem efficiently, we design a randomized algorithm and a drift-plus-age algorithm, and show that they can make the average long-term CA to achieve nearly two times and arbitrarily close to the optimal value, respectively. Through extensive simulations, we further demonstrate that both of the proposed algorithms can significantly reduce the CA of coflows, without violating the throughput requirement of any coflow, when compared to the state-of-the-art solution in both scenario with the packet arrival probability being known and unknown a prior.
Keywords:
Distributed stream processing
coflow
age
Journal
I
IF:
0
Papers:
543
Citations:
0
Organization
View more organizations