One of our goals with Menlo is to share our research process, including experiments that don’t go as expected.
This week, we ran follow-up experiments on Kepler v0.1 to understand why our pretrained encoder wasn’t improving downstream policy performance.
The key takeaway:
The architecture works. The training objective doesn’t.
Under matched training conditions (500 demos, 400 epochs, 2 seeds), our pretrained encoder learned stable latent representations but failed to outperform the baseline policy. Interestingly, our encoder even benefited from additional in-domain pretraining, making the null result stronger.
Digging deeper, we found that the issue wasn’t the encoder itself. We trained it to predict future state, and it did exactly that. The problem is that this objective doesn’t encourage the latent to preserve the object and scene details that manipulation policies actually rely on. Strong pretraining metrics didn’t translate into better downstream control because the representation optimized for the wrong information.
Our next experiments will focus on:
- New latent objectives that preserve object and scene information (e.g. patch reconstruction, DINO distillation, geometry preservation)
- Better integration with the policy using FLARE-style alignment losses
- Diagnostics like SeeSE3 probes to measure whether the latent retains useful 3D structure
Negative results are still useful results. We’d love to hear thoughts on alternative objectives or diagnostics others have found effective for representation learning in robotics.