Kp and Kd setting
In position control, each actuator is commanded to track a desired joint position:
τ = Kp(q_target - q) - Kd q_dot
A higher Kp makes the actuator track the target position more aggressively, improving position accuracy and reducing steady-state error. However, if Kp is too high, the joint may become overly stiff and causing oscillation& jitter behavior. A higher Kd adds damping, which suppresses oscillation and improves stability, especially during fast motion or impacts.
We can model each joint actuator as a second-order spring-damper system, where the armature J represents the reflected motor inertia at the joint:
J * q_ddot + Kd * q_dot + Kp * (q - q_target) = 0
By matching the coefficients of standard second-order system, we get:
omega_n = sqrt(Kp / J)
zeta = Kd / (2 * sqrt(J * Kp))
For stability and reduced oscillation, the system is usually set to be critically damped or overdamped. Critical damping means zeta = 1, while overdamping means zeta > 1. In our case, zeta = 2.0, so the response is intentionally overdamped. The desired natural frequency is set to 10 Hz for a fast and still stable actuator response.
With omega_n = 10 * 2pi rad/s and zeta = 2.0, the gains are computed as:
Kp = J * omega_n^2
Kd = 2 * zeta * J * omega_n
for each joint.
Effort limit (rated torque/peak torque)
Rated torque is the torque the motor can safely output continuously without overheating, while peak torque is the maximum torque it can output briefly during short high-load situations.
For BuiltinDcMotorActuator, voltage_limit limits the maximum voltage command from the internal controller, and effort_limit caps the final output torque. The actuator first converts position error into a voltage command, then the native DC motor model converts voltage into torque:
tau = K * (V - K * omega) / R
As joint velocity increases, back-EMF reduces the available torque. Therefore, voltage_limit determines the available motor drive, while effort_limit acts as the torque safety cap.
Action scale
The policy does not directly output the final joint target position. Instead, the NN output action term is scaled and added to the default joint angle to generate target_dof_pos, which is then sent to the actuator as the position command:
target_dof_pos = default_angles + action * action_scale
And action_scale determines how much each joint is allowed to move around its default pose.
Since different joints have different stiffness and torque limits, using the same action scale for all joints would not be physically consistent. A small position change on a stiff joint can generate a large torque, while the same position change on a softer joint generates less torque.
Hence here we estimated the action scales based on each joints’ stiffness and effort limit:
action_scale = 0.30 * effort_limit / stiffness
This means the maximum action roughly corresponds to a fraction of the joint’s available torque capacity. Joints with higher torque capacity or lower stiffness get a larger action range, while joints with lower torque capacity or higher stiffness get a smaller action range.
This helps normalize the action distribution across joints, so the neural network output has a more consistent physical meaning for every actuator instead of over-driving some joints and under-driving others.
Actuator delay
There’s always latency in real deployment, by inference, communication and motor-driver processing. In order to make the policy deployable in real, we need to inject actuator delay in simulation to model the real behavior and make the policy safer and more robust.
In our setting:
DELAY_MIN_LAG = 0
DELAY_MAX_LAG = 5
This means the actuator command can be delayed by a random amount between 0 and 5 simulation integration steps. 0 means the command is applied immediately, while 5 means the command is applied after 5 simulation substeps. With a simulation frequency of 200 Hz, each substep is 5 ms, so the delay range is 0–25 ms.