Hexapod, part 4 of 4
Learning to walk
The scripted gait from part 2 assumes six working legs. Here the robot learns to walk by trial and error in simulation, first finding a tripod gait on its own, then learning to keep going when a leg gives out.
The setup
Training uses PPO in NVIDIA Isaac Lab, with 4,096 copies of the robot simulated in parallel on one GPU. I started from Isaac Lab's velocity-tracking task, the same one used to train ANYmal, Go1 and Spot, and rescaled it. Those robots walk at about 1 m/s, while this one walks at 0.1-0.2 m/s, so the velocity targets, noise, pushes and reset speeds are all about five times smaller.
| Task | Follow a random velocity command: forward −0.1 to 0.2 m/s, sideways ±0.1 m/s, turning ±1 rad/s |
|---|---|
| Observations | Body velocity, gravity direction, the command, the 18 joint positions and velocities, the previous action |
| Actions | 18 joint position offsets from the standing pose |
| Policy | Multilayer perceptron, 256-128-128, ELU; trained with rsl-rl |
| Servo model | XL430 torque-speed curve: 1.4 N·m stall torque, 6 rad/s no-load speed |
| Randomised per robot | Foot friction (rubber, 0.9–1.2), body friction (PETG, 0.3–0.5), added mass, centre of mass ±1 cm, random pushes |
A bug that broke every foot
The robot comes from the same URDF description as the ROS packages. A script turns it into a training model, with servo limits, a fixed head, and collision shapes only on the feet, legs and body. One problem hid in that import step, and it would have been easy to mistake for a reward problem.
Isaac Sim's URDF importer nests each link inside its parent, but Isaac Lab's contact sensors expect every body to sit directly under the robot. With the nested layout, only the body reported contacts. Every reward term that depends on foot contact (air time, slip, support) was silently computing zero. I wrote a second script that flattens the bodies into a single USD file, and added sanity checks that run before any training: does the robot stand, and is the load balanced? With 3.3 kg on six feet, each foot should carry about 5.4 N.
Shaping a gait
I deliberately didn't impose a gait clock. A fixed tripod rhythm stops making sense the moment a leg is lost, and the damage study below would be biased if the policy were built around one. So the gait had to come out of the rewards. It took several runs:
| Run | Change | Resulting gait | Falls, 1 leg lost |
|---|---|---|---|
| v1 | First setup | Skating and shuffling | 85% |
| v2 | Rubber feet, self-collision, slip penalty | Walks on two diagonal feet | 19% |
| v3 | Penalty for fewer than 3 feet down; air time capped at 0.35 s | Limping tripod | 62% |
| v4–v5 | Foot clearance penalty | Failed to learn | – |
| v6 | v3 + left-right symmetry | Clean symmetric tripod | 78% |
| v7–v9 | Rewards for standing on six feet when idle | Action noise grew without bound | – |
The diagonal gait of v2 came from the reward Isaac Lab's template uses for quadrupeds: "time in the air". Combined with penalties on slip and power for every foot on the ground, it made balancing on two feet, with quick taps of the others, the best deal. A penalty for having fewer than three feet loaded fixed that. Insects keep three feet down at walking speed for the same stability reason.
Two failures were as instructive as the successes:
- Foot clearance (v4–v5). A penalty for feet moving while close to the ground, active from the first iteration, made falling over the cheapest option, since a fallen robot ends its episode early and stops collecting penalties. It's still in the code, with weight zero, waiting for a curriculum.
- Standing still (v7–v9). Rewarding six feet down at zero command made the policy's action noise grow without limit, both from scratch and when fine-tuned. Instead, standing is handled outside the policy: below a small command, the robot blends into the standing pose, just like the ROS gait controller does.
Symmetry
The robot is left-right symmetric, but nothing forces a policy to be. PPO here trains on a mirrored copy of every sample: legs 1↔6, 2↔5 and 3↔4 are swapped and the sign of the coxa angles is flipped. I checked the mirroring in simulation to 0.01 mm. Symmetry was the change that turned the limping tripod of v3 into the clean gait of v6.
Losing a leg
Policy A (run v6) walks well but falls 78% of the time when one leg stops working. Policy B is trained for that case:
- Damage. At every reset, either no leg or one random leg is disabled. Its servo gains are set to zero, as if the servo lost power.
- Memory instead of being told. The policy never sees which leg is disabled. It sees the last 0.5 s of observations (25 steps, as in Rapid Motor Adaptation), so it can work out the damage from how the body responds.
- Privileged critic. During training only, the value network is told which leg is out. That makes learning easier without the policy relying on it.
- Bigger network, longer training. 512-256-128 and 3,000 iterations. Symmetry is kept, which is why B uses an observation history instead of a recurrent network: rsl-rl's symmetry augmentation doesn't support recurrent models.
Both policies were evaluated on 22 conditions: intact, each of the 6 single legs, and each of the 15 pairs. Each condition ran 50 episodes at 0.15 m/s.
| A · standard | B · damage-aware | |
|---|---|---|
| Falls, one leg lost | 78% | 0.3% |
| Speed with one leg lost, vs intact | usually falls | 102% |
| Falls, two legs lost (never seen in training) | 87% | 72% |
| Energy, intact | 71 J/m | 97 J/m |
The hardest test is losing a leg mid-walk. Policy A falls immediately; policy B continues and covers 1.19 m in the next 8 seconds. Robustness has a cost, though: B uses about 35% more energy per metre when intact. And it doesn't generalise to two lost legs, which it never saw in training.
What's next
- Per-leg policies. This work is part of a study asking whether insect-like control, with one small policy per leg, makes a hexapod more robust to damage it has never seen. A third policy, C, will test that with the same symmetry and training budget, so that B and C differ only in network structure.
- Sim-to-real. The servo gains in the model are first guesses and need to be identified on the robot. Body velocity is observed in simulation but not measurable on the real robot, so it has to move to the critic before any policy runs on hardware.
Takeaways
Most of the work in RL locomotion is in the details. The robot model, the contact setup and the reward terms decide whether the policy learns a real gait or finds a loophole, and a silent sensor bug can look exactly like a reward problem. Gait diagrams and systematic evaluation over many episodes say far more than a single good-looking video.