Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation

Ensuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we address this conflicting relationship by leveraging the theory of gradient manipulation. We analyze the conflict between reward and safety gradients and propose a soft switching policy optimization method, with convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. We evaluate the effectiveness of our method on Safety-MuJoCo and Safety Gymnasium. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization.