Ensuring Safety in Real-World Reinforcement Learning: Tsinghua’s Safe Exploration Equilibrium Mechanism

Problem

Real‑world reinforcement learning must keep every intermediate policy strictly safe; any violation can damage hardware or cause injury.

Feasible‑Zone Framework

Safety is enforced by restricting exploration to a feasible zone computed from an environment model. The model is initially uncertain; the algorithm assumes worst‑case model error, yielding a robust feasible zone that guarantees safety as long as the agent stays inside it.

The learning loop is a “snowball” process: data collected inside the current feasible zone are used to refine the model, which in turn expands the feasible zone.

Open Question

Prior work (e.g., Krause et al. on Lyapunov‑based zones and Tomlin et al. on Hamilton‑Jacobi reachability) left unanswered whether this expanding process converges and, if so, to what limit.

Theoretical Result

The paper proves that the iterative process inevitably converges to a well‑defined Safe Exploration Equilibrium . At equilibrium the feasible zone cannot grow further because additional data cannot reduce model error, and the model cannot be refined further within that zone.

Maximum Feasible Zone : the largest safe region supported by the current model.

Least Uncertain Model : the model with minimal achievable error inside that region.

SEE Algorithm

Zone Computation : With the model fixed, solve a Risky Bellman Equation to obtain the maximum feasible zone.

Model Update : With the zone fixed, cast the search for the least uncertain model as a Clique Decision Problem and solve it approximately in polynomial time.

Convergence Properties

Repeated alternation of the two steps yields monotonic reduction of model error, monotonic expansion of the feasible zone, and guarantees convergence to the equilibrium point.

Numerical Validation

SEE was evaluated on three benchmark control tasks:

2‑D linear double integrator.

2‑D nonlinear inverted pendulum.

3‑D nonlinear unicycle obstacle‑avoidance.

All experiments exhibited strict zero‑constraint violation. On the unicycle task the feasible‑zone recall reached 95.78 % after only ten iterations, demonstrating rapid approach to the theoretical limit.

Code example

来源：机器之心
本文
约2500字
，建议阅读
5
分钟
让智能体在零约束违反的前提下逐步扩展可行区域。