Hi everyone! I’m Olga, founder of VR Oxygen. We built infrastructure for distributed spatial tasks and are now exploring how it can support Physical AI and robotics data collection.
I’m hoping to learn from teams already building in this space.
For those collecting real-world data today:
What types of data are currently the hardest to source (egocentric video, human demonstrations, teleoperation, household interactions, or something else)? What makes them difficult?
Which parts of the workflow are the biggest operational bottlenecks: finding participants or operators, qualification, scheduling, quality control, hardware, or something else?
Which datasets do you find yourselves collecting repeatedly rather than as one-off projects?
I’d really appreciate hearing about your experiences. Thanks!
I think data that actually has more information than just video, for example, torque, force, tactile, thing that supplement representation that is necessary for a human to operate that is not just vision. It is quite difficult because you need extra hardware to collect those.
Hardware
Data that is corresponding to the embodiment we need to use because the mapping is quite important.
It sounds like the bottleneck isn’t just collecting more data, but collecting the right modalities for a specific embodiment, and that the required hardware makes those datasets much harder to acquire.
Are most teams building custom in-house rigs for this today, or is a more standardized setup starting to emerge?
I agree with Yuuki that hardware is probably the biggest bottleneck, and I’d add that it’s not just about having the sensors, but having a reliable, repeatable data collection pipeline.
One reason is that today’s VLA models are surprisingly sensitive to camera setup. We recently ran a small experiment showing how even relatively minor viewpoint changes can significantly affect model performance. ( Why VLAs are so viewpoint brittle? ) That means camera placement, calibration, and synchronization aren’t just implementation details, they’re effectively part of the data distribution the model is trained on.
Because of that, I don’t think a truly standardized setup has emerged yet. Most teams still end up building custom rigs around their embodiment, sensor suite, and tasks. Even if two teams are both collecting teleoperation data, differences in cameras, calibration, control interfaces, and robot kinematics can make the resulting datasets much less interchangeable than they appear.
This is a really important point, thanks for sharing! It essentially means camera setup is a core decision that has to be locked down across the entire collection pipeline.
If you’re collecting across hundreds of different home environments and operator setups, even small differences in camera angles or heights can confuse the AI even if the human operator performed the task perfectly.
Is this brittleness mostly about camera placement, or does variance in rooms, lighting, and object distance still cause major issues? And do you expect future models to handle these camera variations better, or will strict hardware setups always be necessary?
I think the strongest evidence today is around viewpoint and camera geometry. In our experiment, relatively small changes in camera placement alone were enough to noticeably degrade performance, which suggests current VLA vision encoders are quite sensitive to that distribution shift.
Lighting, background clutter, object appearance, and other environmental factors definitely matter too, but it’s harder to isolate their individual effects.
I do expect future models to become much more robust as we get better pretraining, data augmentation, and larger, more diverse datasets. At the same time, I don’t think standardized hardware will disappear entirely. Even if the model can tolerate more variation, consistent camera placement and calibration make data collection, debugging, and evaluation much more reliable.
Thanks for the detailed explanation! This was really helpful and gives me a clear sense of why the collection protocol itself is such an important part of the dataset.