.png){kind=icon}
Return
Cautious implicit value learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neunet.2026.109511.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
•
A missing next-state constraint in the V-network loss is revealed, motivating a target V-network and a sample filtering for cautious value and policy updates.
•
Trajectory-position weighting is proposed to leverage trajectory information and dynamically adjust the V-network’s pessimistic/optimistic bias in a temporally fine-grained manner.
•
Negative-action penalty suppresses actions deemed unworthy and stochastically induces an approximate Actor gradient penalty; a floating term is introduced to increase its occurrence.
Abstract
Implicit Q-learning, a prominent approach within the implicit value learning (IVL) paradigm, mitigates distribution shift in offline reinforcement learning (offline RL) by avoiding out-of-distribution action queries. However, current IVL methods lack stepwise training analysis in value learning and commonly assume the behavior policy to be optimal, both of which overlook more realistic suboptimal scenarios. This work presents cautious implicit value learning (CIVL), a novel offline RL method that redefines the advantage weight to identify a previously unrecognized constraint in IVL training. By establishing a formal connection between the identified constraint and the target V-network, CIVL provides theoretical motivation for introducing the target V-network. In policy learning, CIVL introduces a negative-action penalty (NAP) that penalizes actions with negative advantage values, thereby better handling suboptimal actions. Since the NAP can be shown to stochastically induce an approximate gradient penalty on the Actor, CIVL introduces a floating term that synergistically increases the probability of the gradient penalty occurring. Finally, CIVL further incorporates trajectory-position weighting to provide temporally fine-grained control over the V-network’s bias. Benchmark experiments demonstrate that CIVL outperforms existing IVL methods in terms of normalized score and training stability.
Keywords:
Offline reinforcement learning
Implicit value learning
Target V-network
Advantage weight
Gradient penalty
Journal
IF:
6.3
Papers:
8.2K
Citations:
3.0W
Organization
Cited Papers
.png){kind=icon}
ShareSave