.png){kind=icon}
返回
CVaR-Constrained Policy Optimization for Safe Reinforcement Learning
.png){kind=icon}
PRE
AI
DOI：10.1109/TNNLS.2023.3331304.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Current constrained reinforcement learning (RL) methods guarantee constraint satisfaction only in expectation, which is inadequate for safety-critical decision problems. Since a constraint satisfied in expectation remains a high probability of exceeding the cost threshold, solving constrained RL problems with high probabilities of satisfaction is critical for RL safety. In this work, we consider the safety criterion as a constraint on the conditional value-at-risk (CVaR) of cumulative costs, and propose the CVaR-constrained policy optimization algorithm (CVaR-CPO) to maximize the expected return while ensuring agents pay attention to the upper tail of constraint costs. According to the bound on the CVaR-related performance between two policies, we first reformulate the CVaR-constrained problem in augmented state space using the state extension procedure and the trust-region method. CVaR-CPO then derives the optimal update policy by applying the Lagrangian method to the constrained optimization problem. In addition, CVaR-CPO utilizes the distribution of constraint costs to provide an efficient quantile-based estimation of the CVaR-related value function. We conduct experiments on constrained control tasks to show that the proposed method can produce behaviors that satisfy safety constraints, and achieve comparable performance to most safe RL (SRL) methods.
Keyword:
Costs
Safety
Optimization
Measurement
Tail
Reactive power
Random variables
Artificial intelligence (AI) safety
conditional value-at-risk (CVaR)
constrained policy optimization
safe reinforcement learning (SRL)
期刊
IF:
8.9
论文数:
7.6K
被引数:
7.2W
机构
引用论文
Miscibility and interactions in a mixture of poly(ethylene oxide) and an aromatic poly(ether amide).png)
Polymer
PolymerIF0
PREAI
分享收藏
Study of bi-directional buck-boost converter topologies for application in electrical vehicle motor drives应用于电动汽车电机驱动的双向buck-boost变换器拓扑研究
PREAI
分享收藏
查看更多内容