Ensuring Safety in Real-World Reinforcement Learning: Tsinghua’s Safe Exploration Equilibrium Mechanism

Recent work by Tsinghua University published in IEEE TPAMI tackles the critical problem of safe exploration in real‑world reinforcement learning (RL). Unlike simulation where unlimited trial‑and‑error is possible, physical systems cannot afford unsafe actions that may damage equipment or endanger people.

The authors define safe exploration as requiring every intermediate policy during training to remain within a feasible zone —a region derived from an environment model that is robust to model uncertainty. As the model improves, the feasible zone can expand, forming a “snowball” process of data collection, model refinement, and zone enlargement.

Prior approaches, such as those by Andreas Krause’s team (Lyapunov‑based feasible zones) and Claire Tomlin’s team (Hamilton‑Jacobi reachability), leave an open question: does this expanding process converge, and if so, to what limit? The Tsinghua paper proves that the process inevitably converges to a well‑defined point called the Safe Exploration Equilibrium , where the feasible zone cannot grow further because the model error is already minimized.

The equilibrium consists of two elements: (1) the Maximum Feasible Zone , the largest safe region achievable under the current model, and (2) the Least Uncertain Model , the most accurate model attainable given all data inside that zone.

To locate this equilibrium, the authors propose the Safe Equilibrium Exploration (SEE) algorithm, which alternates two steps:

Step 1 – Zone Computation: With the current model fixed, solve a Risky Bellman Equation to obtain the Maximum Feasible Zone.

Step 2 – Model Computation: With the newly computed zone fixed, cast the search for the Least Uncertain Model as a Clique Decision Problem and solve it approximately in polynomial time.

Theoretical analysis shows that each iteration monotonically reduces model error, monotonically expands the feasible zone, and guarantees convergence to the equilibrium.

Numerical experiments on three classic control benchmarks—2‑D linear double integrator regulation, 2‑D nonlinear inverted‑pendulum balancing, and 3‑D nonlinear unicycle obstacle avoidance—demonstrate the claimed properties. The SEE algorithm achieves strict zero‑constraint violation and reaches the equilibrium within a few iterations (e.g., 10 iterations for the unicycle task, attaining 95.78% zone recall). Images illustrate the monotonic expansion of the feasible zone for each task.

The authors argue that this equilibrium perspective provides a solid mathematical foundation for safe real‑world RL, encouraging future work to integrate more expressive function approximators or to deploy the framework on high‑DOF humanoid robots.