- In locomotion RL, rewards define what behavior the policy should learn. Instead of directly telling the robot how to move each joint, we define reward terms that encourage desired behaviors and penalize undesirable ones. These rewards can be roughly classified into three main categories:
- Task tracking rewards: encourage the robot to follow the command, such as linear velocity tracking, angular velocity tracking, and upright torso tracking.
- Gait and contact rewards: encourage natural and stable stepping behavior, such as foot air time, foot clearance, foot slip, stumble penalty, and contact-force penalty.
- Regularization and safety rewards: encourage smooth, safe, and physically reasonable motion, such as action smoothness, joint-limit penalty, angular momentum penalty, and self-collision penalty.
For reward tuning, it’s good to start with the main task rewards first, like velocity tracking, upright. Then based on the behavior of the robot, rewards can be further fine tuned: add feet air time to encourage foot lifting, and other gait and contact rewards to make the walking pose better. Then to make it robust and smooth, regularization and safety can be tuned.
Figure4. Asimov rewards list