How do Asimov training tasks? What’s the scalable ways to do it?
Let’s first understand what are the approaches out there to enable a robot to perform desired tasks and behaviors and the difference between them.
In robotics, the methods that can generate robot motions can be broadly divided into explicit models (dynamics-based) and implicit models (learning-based).
-
Explicit Models: The robot’s behavior is generated using equations or rules written directly into the controller, such as kinematics, dynamics, contact constraints, or optimization objectives. The controller explicitly computes how the robot should move rather than learning the complete mapping from observations to actions from data.
-
Implicit Models: Learning-based methods learn the relationship between sensor observations, robot states, and control actions from data rather than relying entirely on manually specified dynamics. The makes the model represents robot–environment interactions implicitly through learned parameters.
Robot motion can be roughly classified into 3 categories: 1. manipulation, 2. Locomotion , 3. Whole-body control. These motion types involve different dynamic constraints and therefore typically require different control methods.
- Inverse Kinematics:
Inverse kinematics (IK) controls a robot by converting a desired end-effector pose, such as a target hand position and orientation (x, y, theta), into the joint angles needed to reach it: q_des = IK(x_des, y_des, theta_des). For example, when commanding a robot arm to reach a cup, we specify where the hand should go, and IK calculates how the shoulder, elbow, and wrist should move. These desired joint angles are then sent to a low-level controller, such as PD control, which drives each motor to track them: torque = Kp*(q_des - q) + Kd*(dq_des - dq). In simple terms, IK decides the target joint configuration geometrically, while the low-level controller makes the physical robot follow it. This approach is well suited to slow, quasi-static manipulation and lightweight robots such as Bittle, where IK generates joint or foot trajectories and PD control tracks them. However, IK alone is insufficient for dynamic locomotion for heavy robots because it does not account for joint torques, contact forces, momentum, friction, or balance.
- Model predictive control:
Model predictive control (MPC) generates robot motion by repeatedly solving a finite-horizon optimization that minimizes future tracking and control costs subject to dynamics, (x_{k+1}=f(x_k,u_k)), and constraints such as torque limits, contact forces, and friction. At each timestep, it estimates the current state, optimizes a trajectory, executes only the first action, and replans. MPC is widely used in legged robotics because it explicitly handles dynamics and constraints, but robust bipedal control may require evaluating many contact sequences, model variations, and disturbance scenarios online, making computation grow with the horizon and number of scenarios. RL shifts most of this computation offline by training across randomized simulated conditions and encoding the resulting state-to-action mapping in a neural-network policy. Runtime control then requires only a forward pass, although robustness is limited to the training distribution and constraints are not guaranteed explicitly.
- Imitation Learning:
Imitation learning is a learning-based methods that learns control policies from expert demonstrations. In behavior cloning, a neural network is trained to minimize the difference between predicted actions from the policy and expert actions. By minimizing this loss, the policy gradually learns to reproduce the expert’s behavior. However, for dynamic locomotion, limited demonstration coverage can cause distribution shift: once the robot deviates from the expert trajectory, it may encounter unseen states and produce poor recovery actions, which causes errors to compound. So for Humanoid locomotion, imitation learning is not preferred.
- Reinforcement Learning
In reinforcement learning (the graph above demonstrates how PPO, an RL algorithm works), the robot observes the environment o_t, chooses an action a_t = pi(o_t), and receives a reward r_t. The goal is to maximize the long-term return R_t = r_t + gamma*r_(t+1) + gamma^2*r_(t+2) + .... During training, the simulator exposes the robot to many randomized conditions, such as pushes, uneven terrain, friction changes, payload variation, and sensor noise. Instead of manually defining a controller for every possible edge case, the policy learns an implicit feedback strategy from these experiences.
Reinforcement learning is therefore good for locomotion because the desired behavior is encoded through rewards, and the policy can collect large amounts of interaction data in simulation under randomized pushes, contacts, friction, terrain, payloads, and initial states. This allows it to learn balance and recovery strategies implicitly, although it will only generalize to edge cases that are sufficiently represented by the simulator and training distribution.