Last week we shared that our pretrained encoder learned good representations but didn’t improve downstream policy performance. Our hypothesis was that the issue was the training objective rather than the architecture, so we explored new latent objectives that preserve object and scene information, tighter policy integration, and better diagnostics.
The offline metrics improved, but the policy still didn’t.
We added object and scene supervision, redesigned the encoder so the robot state conditions what it extracts from images, and evaluated several ways of feeding the latent into the policy, including replacement, concatenation, FLARE-style auxiliary alignment losses, and unpooled tokens. Offline, the latent eventually matched raw image patches on our probing tasks.
However, after two more rounds of downstream evaluation, policy performance still failed to improve. Several variants actually hurt performance. On our cleanest benchmark, the vision baseline achieved ~0.90 success, the state-blind encoder reached near parity (~0.85), while the more sophisticated state-conditioned encoder dropped to ~0.54. Replacing vision entirely with the latent failed outright (0.00). The biggest surprise was that a better offline representation consistently produced a worse control policy.
A deeper literature review helped explain why. Frozen pretrained encoders generally provide little benefit for reactive behavior cloning when abundant demonstrations are already available, which closely matches our setting. Based on both our results and prior work, we’re retiring “frozen encoder for direct reactive control” as our primary research direction.
Instead, we’re focusing on three settings where pretrained representations are much more likely to help:
-
Robustness under distribution shift. Does the representation degrade less than raw vision under changes in lighting, camera viewpoint, or distractors? This is the cheapest experiment since it can be run on existing checkpoints.
-
Low-data learning. Re-evaluate with only 10 to 100 demonstrations, where nearly all published gains from frozen encoders have been reported.
-
Latent-space planning. Use the compact 8-token representation as the state for a world model and planner instead of directly driving a reactive policy. This makes planning roughly 10 to 100× cheaper per step and remains largely unexplored for robotic manipulation.
Alongside these experiments, we’re also testing the one policy integration method with consistent evidence in the literature (letting the action head attend to latent tokens rather than injecting them into the backbone), and rebuilding the encoder to model temporal and multi-sensor observations with better collapse detection during training.
Whatever the outcome, we’ll learn something useful. Either we identify where pretrained representations genuinely add value, or we produce a rigorous negative result with strong baselines and clearly defined evaluation criteria that helps the community avoid the same dead ends.
As always, we’d love to hear from others who have successfully made pretrained representations work for robotics, particularly in low-data learning or planning rather than reactive control.