.png){kind=icon}
返回
Whittle Index-Based Q-Learning for Wireless Edge Caching With Linear Function Approximation
.png){kind=icon}
OA
AI
DOI：10.1109/TNET.2024.3417351.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
We consider the problem of content caching at the wireless edge to serve a set of end users via unreliable wireless channels so as to minimize the average latency experienced by end users due to the constrained wireless edge cache capacity. We formulate this problem as a Markov decision process, or more specifically a restless multi-armed bandit problem, which is provably hard to solve. We begin by investigating a discounted counterpart, and prove that it admits an optimal policy of the threshold-type. We then show that this result also holds for average latency problem. Using this structural result, we establish the indexability of our problem, and employ the Whittle index policy to minimize average latency. Since system parameters such as content request rates and wireless channel conditions are often unknown and time-varying, we further develop a model-free reinforcement learning algorithm dubbed as Q(+)-Whittle-LFA that relies on Whittle index policy. However, Q(+)-Whittle-LFA requires to store the Q(+)-function values for all state-action pairs, the number of which can be extremely large for wireless edge caching. To this end, we approximate the Q-function by a parameterized function class with a much smaller dimension, and further design a Q(+)-Whittle-LFA algorithm with linear function approximation, which is called Q(+)-Whittle-LFA. We provide a finite-time bound on the mean-square error of Q(+)-Whittle-LFA. Simulation results using real traces demonstrate that Q(+)-Whittle-LFA yields excellent empirical performance.
Keyword:
Wireless communication
Indexes
Wireless sensor networks
Servers
Q-learning
Quality of experience
Costs
Wireless edge caching
restless bandits
whittle index policy
reinforcement learning
finite-time analysis
期刊
I
IF:
3.6
论文数:
4.4K
被引数:
9.5K
机构
引用论文
分享收藏分享收藏Dynamic Control of Birth-and-Death Restless Bandits: Application to Resource-Allocation Problems
PREAI
分享收藏Formation of NADP(H) analogs of tobacco-specific nitrosamines in rat liver and pancreatic microsomes.png)
IF0
PREAI
分享收藏
查看更多内容