How Humanoid Robots Sense Invisible Forces: Momentum Observers + Learned Residuals for Robust Gait
This article details a robust gait control system for humanoid robots that estimates external forces without force sensors by combining a momentum observer with a learned residual network, enabling the robot to recover from pushes during walking.
1. Why Invisible Forces Cause Falls
Early in the project, a reinforcement-learned walking policy was stable on flat ground but collapsed when a person lightly tapped the robot's shoulder. The 50 Hz policy should have reacted in time, yet the robot fell sideways. Logs showed the policy's joint torque commands were normal, but the body had already developed a lateral velocity deviation. The core issue: the policy could not distinguish state changes caused by its own actions from those imposed by external forces. During training, the policy only saw self-generated state transitions; when an external disturbance broke that pattern, the policy applied its normal gait correction logic — designed for self-generated motions — which mismatched the disturbance dynamics, amplifying errors between footfalls until balance was lost.
2. What Sensors Are Missing
Typical humanoid sensors — joint encoders, IMU, joint current sensors — all measure "effects" not "causes." An external push on the trunk produces IMU linear acceleration, joint position offsets, and increased motor torques to maintain trajectory. But identical signals arise from the robot's own acceleration, turning, or ground reaction forces on uneven terrain. Without force sensors, these signals are coupled and cannot be separated into an "external force" variable. Early attempts using joint current differentials worked for single-joint tests but failed during whole-body walking because ground impact forces create high-frequency current peaks indistinguishable from external forces, and some disturbances (e.g., a sideways tap) produce only a sustained acceleration bias without sharp current spikes.
3. What the Policy Actually Needs
From a control perspective, the policy needs a quantified, action-separable external force estimate — not a binary collision flag. Ideally, a vector of external forces/torques on each rigid body. However, for a tightly coupled bipedal system, the policy only requires an equivalent influence: the external force/torque near the center of mass (CoM) and a coarse hint of which body segment is affected. The policy's goal is whole-body balance, so it cares about how the external force changes CoM state trends, not the exact application point.
4. External Torque Observer Design and Implementation
4.1 From Rigid-Body Dynamics to Momentum Observer
The foundation is the generalized momentum observer, a decades-old technique in manipulator control. Starting from the rigid-body dynamics equation: M(q)q̈ + C(q,q̇)q̇ + G(q) = τ + τ_ext where q is joint positions, M inertia matrix, C Coriolis/centrifugal terms, G gravity, τ motor torque commands, τ_ext external torque. Defining generalized momentum p = M(q)q̇ yields: ṗ = τ + τ_ext + Cᵀ(q,q̇)q̇ - G(q) Instead of directly computing acceleration, the observer subtracts known model terms and motor commands from the measured momentum rate to obtain an external torque estimate. This filters some model errors and high-frequency noise but critically depends on dynamics model accuracy. Any inertia, friction, or transmission model mismatch introduces large drift, rendering the estimate unusable.
4.2 Learning Residuals: Data-Driven Substitute for Precise Modeling
Precise identification of all friction and motor transmission models proved infeasible — each joint's friction curve differs and drifts with temperature, wear, and load. Instead of discarding the physics observer, its output is used as a baseline, and a lightweight neural network learns the "residual" between the physics observer and the true external force.
Implementation pseudocode:
# Pseudocode: external torque estimation data flow
def estimate_external_torque(state, cmd_torque, model):
# Physics baseline: generalized momentum observer
mom_obs = momentum_observer(state, cmd_torque, model)
# Learned residual input: sliding window of historical states and commands
feat = build_feature_window(history, cmd_torque)
residual = residual_net(feat) # small MLP
# Fused output, low-pass filtered
ext_est = lowpass_filter(mom_obs + residual)
return ext_estThe network input is a 10-step (200 ms) history of joint positions, velocities, IMU linear acceleration, angular velocity, and motor torque commands — a few hundred dimensions. Output is 3D external force and 3D external torque in CoM frame, plus a discrete confidence for whether the force acts on trunk/arm/swing leg. The network is deliberately tiny: two-layer MLP, ~90k parameters, to learn only the residual, not the entire physics.
Training data comes from simulation with diverse disturbances: varying direction, duration, magnitude pushes and random collisions. Ground-truth external forces are available in sim; subtracting the momentum observer's output gives the residual supervision target. The key sim-to-real gap is that dynamics error distributions differ. Domain randomization addresses this: inertia parameters, friction, mass distribution, and joint torque latency are randomized so the residual network learns a general mapping from "current sensor state" to "physics observer error estimate" rather than a specific error pattern.
4.3 Injecting Estimates into Policy Observation Space
The PPO policy's original observation space included joint positions, joint velocities, IMU orientation, linear velocity, angular velocity, previous action, and gait phase. The external torque observer's output is concatenated directly, forming a new observation vector. Policy and value networks share a bottom feature extractor. This small change yielded a clear behavioral shift: the policy learned to distinguish "this displacement came from my own walking" from "this displacement came from an external push." Post-training, the policy reacts faster to pushes and walks more stably on flat ground because it no longer overreacts to noise in the external force estimate.
5. Training and Transfer: Sim-to-Real Pitfalls
5.1 Simulation Environment and Domain Randomization
Simulator: MuJoCo, 50 Hz control, 0.5 ms physics step. Three disturbance categories were injected during policy training:
Short impulsive forces: 50–200 ms, 10–80 N, random direction (simulating taps/collisions).
Sustained forces: 0.5–2 s, 5–40 N, random direction (simulating continuous pushing or wind).
Random lift forces: upward or lateral force at a trunk point (simulating backpack snag or light lift).
Force magnitudes were calibrated from real-world push tests using a handheld rod with a cheap strain-gauge sensor. Typical adult pushes ranged 15–45 N, with impact peaks ~60 N — lower than expected, so training bounds were widened.
Domain randomization ranges: joint friction torque ±40%, link mass and CoM ±10%, IMU noise std 1–3× real hardware, control latency 1–3 control cycles. Latency randomization proved critical: without it, the policy learned to predict based on zero latency, causing slight oscillations on the real robot with ~40 ms latency.
5.2 Reward Design to Avoid Policy Exploiting Wrong Signals
Initial reward: "keep CoM near target" + external torque estimate as observation. The policy learned a degenerate behavior: upon any external force estimate, it moved rapidly opposite to reduce CoM error, regardless of force direction, becoming hypersensitive to noise.
Reward was split into two parts:
Proprioceptive reward: encourage desired gait phase, joint trajectories, small orientation error.
External force response reward: instead of directly rewarding resistance, penalize "external force direction opposite to policy-induced acceleration direction." This encourages the policy to "yield" to the force rather than fight it.
Result: the policy behaves more like a human — pushed from left, it yields slightly left while stepping right to recover. Hard resistance almost always causes falls on the real robot; yielding is stable.
5.3 Real-Robot Calibration and Validation Before Deployment
Before deployment, the robot was fixed on a weighted stand, suspended, and manually pushed on each body segment while recording true force vs. observer estimate. Three issues emerged:
Peak delay: observer estimate lagged true force peak by ~80 ms due to low-pass filter phase lag. Cutoff frequency raised from 2 Hz to 6 Hz; noise increased slightly but policy learned to ignore high-frequency noise.
Residual network input distribution shift on real robot, especially during high-speed swing leg motion where IMU vibration caused false external force detections. Added sliding-window mean on raw IMU data, window length increased from 20 ms to 60 ms.
Joint position tracking error larger on real robot due to gear friction and flexible transmission (sim used ideal torque sources). Added first-order inertia delay to all joint torque commands in sim and increased observation noise randomization.
Final pre-deployment step: offline noise injection test. Real-robot logged observations replayed through the observer and policy to check for high-frequency action jitter. If jitter appeared, observation denoising or reduced policy gradient dependence on the estimate was added.
6. Experimental Comparison: With vs. Without External Force Perception
6.1 Simulation Disturbance Tests
Protocol: robot walks straight at 0.8 m/s; every 5 s a random 150 ms force applied to trunk, magnitude ramped 10–90 N, 20 trials per magnitude. Three policies compared:
Group A: baseline, no external force estimate in observation.
Group B: abstract collision flag (1/0) added, no magnitude/direction.
Group C: full proposed solution, external torque estimate as observation.
Results (image included): Group C maintains >60% success rate at 50 N, sufficient for typical human pushes. Merely knowing a collision occurred (Group B) is far from enough; direction and magnitude are essential.
6.2 Real-Robot Push Experiments
Three directions: front, side, rear. Colleagues of varying weight pushed 20 times each while robot walked 2 m. Success categories: experiential success (stopped but didn't fall), recovery success (stumbled but reached goal), failure (fell).
Max tolerable forces: front ~45 N, side ~38 N, rear ~28 N. Front/rear asymmetry due to larger ankle/hip range and torque capacity in sagittal plane; lateral balance relies on side-stepping; backward walking has smallest margin.
Continuous disturbance test: two people randomly push from all sides, recording how many consecutive pushes robot withstands. Group C averaged 6 pushes; Group A averaged <2. This demonstrates improved closed-loop adaptation to continuous disturbances, not just single-event response.
6.3 Instructive Failure Cases
High-speed walk (1.2 m/s, beyond tested 0.8 m/s) pushed from rear-side. Observer latency 80 ms + policy inference 20 ms = 100 ms total response, too long for short support phases at high speed. Increasing estimation frequency to 125 Hz would mitigate.
On 10° upslope, robot misclassified terrain normal reaction force as forward external force, stepped prematurely onto a small stone, slipped and fell. Estimator validated only on flat terrain; no calibration for terrain variation.
After sustained push removed, policy maintained "crouched" anti-push posture ~0.5 s. Policy learned to lower CoM and abduct hips during sustained push; waited for external torque estimate to fully return to zero before restoring upright stance. Added fast-decay gating on estimate: when estimate drops continuously, policy recovers default stance earlier.
7. Lessons Learned and Next Steps
7.1 Unsolved Problems
Online calibration drift: motor temperature rise changes friction, shifting residual network input distribution. Current workaround: lightweight online adaptation every 5 minutes using no-external-force data to refit residual network bias. Requires reliable detection of "external force interruption" events.
Contact location identification: estimator only gives CoM-equivalent force. Low-confidence body-part classification insufficient for generating arm avoidance motions when arm is pulled. Next step: add "contact probability" prediction heads for arms/legs, trained in sim without extra physical sensors.
Distributed force sensors debate: for truly general humanoid applications, at least foot 6-axis force sensors are still needed because foot contact forces are too critical for balance; learning-based estimation cannot yet meet industrial safety/precision standards. But for rapid control algorithm validation, the pure observer approach is sufficient.
7.2 Practical Advice for Practitioners
Calibrate control latency first, before any observer. Every 10 ms added latency significantly degrades external force observer phase accuracy. Final solution: randomize 1–2 step delays per joint in sim to match real robot.
Do not use external torque estimate directly in reward. Use two-stage training: freeze estimator, train policy to convergence, then unfreeze for end-to-end fine-tuning. Joint training from scratch causes estimator scale and reward to fight, extremely slow convergence.
During real-robot continuous disturbance testing, include "weak disturbances" like pedestrian airflow, door drafts, floor vibrations. These can be masked by robot's own motion or picked up by estimator at low magnitude. Without constraints, policy overreacts, causing frequent adjustments and stiff gait. Solution: add random tremor noise in sim training, forcing policy to ignore external force estimates below a threshold.
The project's deepest insight: enabling a humanoid to "see" external forces is not about adding a sensor, but about giving the policy a crucial missing piece of the "state puzzle." External disturbances are the control system's natural enemy, but if the policy knows where the enemy is, its direction, and its magnitude, balance becomes far easier to maintain. Future work will extend this state information to complex terrain and dynamic object interaction.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
