Why VLAs are so viewpoint brittle?

[Ishneet_Singh]

TL;DR: a tripod nudge can push the policy much further out of distribution than a completely new object layout. Fixing this likely requires multi-view training, viewpoint augmentation, or representations explicitly encouraged to factor viewpoint out.

Why a tripod nudge kills a VLA that handles object variation fine.

Every VLA deployment hits the same wall. The policy handles object positions it has never seen, then fails completely when the camera moves. This is not anecdote: controlled decompositions of the imitation-learning generalization gap rank new camera positions among the hardest distribution shifts — well above new object positions (Xie et al., 2023) — and it’s the failure that viewpoint-augmentation methods exist to patch (Tian et al., 2024; Chen et al., 2024). Placements varied in training; the camera didn’t. We wanted to see this asymmetry directly in the representation, with the smallest experiment that could show it.

The experiment

18 photos of our two-arm rig: the same four objects (a headset shell, a shoe last, a Joy-Con, an orange bezel) in 3 layouts, each shot from 6 camera positions. Cameras A–C are overhead variants; D–F are side views from table height — spanning everything from a mount nudge to a full re-rig.

Same layout across a row of cameras, same camera across a column of layouts — so every pair of photos is exactly one of: objects moved, same camera (18 pairs), camera moved, objects untouched (45 pairs), or both (90).
Encoders — the four ViTs that VLAs actually deploy: SigLIP So400m/14, the vision tower of PaliGemma and therefore π0; SigLIP2, its successor in the GR00T N1 family; DINOv2 ViT-L, half of OpenVLA’s fused encoder; and CLIP ViT-L, the classic VLM tower. All at 224px. The metric is mean cosine distance between corresponding tokens of the 16×16 patch grid — the visual state the language model consumes. For scale, a 10% brightness change scores 0.05 on SigLIP (and 0.001 on DINOv2).

Moving the camera changes more than moving the world — in every encoder

encoder	objects moved	camera moved	both	camera ÷ objects
SigLIP So400m	0.46	0.61	0.63	1.3×
SigLIP2 So400m	0.40	0.54	0.57	1.4×
DINOv2 ViT-L	0.31	0.57	0.60	1.9×
CLIP ViT-L	0.36	0.52	0.54	1.4×

Two things to note. Moving the camera with the world untouched lands within a few percent of “everything changed” — once the viewpoint shifts, rearranging the objects barely adds anything. And the full distance matrix shows the structure directly: the light stripes are the object rearrangements, the dark blocks are camera changes.

Placing all 18 images in 2D by their distances makes it blunt: the representation’s biggest axis is which camera, not what’s on the table. The side-view photos clump together with all three layouts interleaved — world state is a detail inside a camera cluster.

The bands do overlap at the extremes: the smallest camera move (E→F, adjacent side views: 0.31) scores less than the biggest rearrangement (0.57). Distance tracks how much of the image changed — and a camera move changes all of it, a layout change only some of it. That is precisely the problem.

The objects were never lost

The obvious suspicion — the encoder can’t track objects across views — is mostly wrong. Cluster all 4,608 patch tokens with k-means and paint the clusters back: the same content clusters (objects, arms, wood, wall) appear in every view, overhead or side-on.

Single-token matching — ViT patch features as dense descriptors (Amir et al., 2021; Oquab et al., 2023) — shows the same thing, with a caveat that proves the point. Within the overhead family, one token from the orange bezel relocates the bezel in other images. Across the 90° re-rig, similarity drops by a third and the best match drifts to the robot’s orange joints. Object identity survives modest view changes and fades exactly where the viewpoint shift is extreme.

So where does the viewpoint hide?

In every token, one level below the content. For each patch, ask which other image holds its single nearest neighbor. Chance is 5.9% per candidate image. A same-camera partner attracts 15% of an image’s patches; a partner showing the identical scene from another camera attracts 9%; an unrelated image 2% (SigLIP; 14–17% vs 9–10% vs 2% across all four encoders). Content pulls above chance — the objects are recognized — but the camera pulls harder.

Each patch feature is coarsely “what am I looking at” and finely “exactly how this camera frames it” — which slice of wood grain, what perspective, what shadow. That patch tokens carry more than their local content is a documented ViT behavior (Darcet et al., 2023). The policy reads all 256 tokens at once, and 256 small camera stamps add up to the table above.

Why this breaks VLAs

The policy’s state is that token grid. Camera pose is its dominant coordinate, so a viewpoint change relocates the state further than any object rearrangement — into a region the action head has never mapped. And it never mapped it because the camera was bolted down for every demo: placements came with thousands of training variations, viewpoint with zero. The information needed to recognize “same scene, different view” is demonstrably in the tokens; there was just never any training pressure to use it. Which also says what won’t fix this — a better object detector. The camera has to move in training — via multi-view data or synthesized viewpoints (RoVi-Aug, VISTA) — or the representation has to be forced to factor it out.

[Jia_Qi_Yip]

For context, this is how close ABC and DEF are to each other in positions.

We started investigating this because I was annoyed by how strangely sensitive all the VLA models were when it came to camera positions. And in practice there was just a lot of messing with camera mounts until the VLA worked. It doesn’t matter whether it is trained in sim or trained on real data, and even with camera viewpoint augmentation the brittleness stayed. Given that the vision encoders are very often pre-trained for VLA models, I was wondering if maybe something unexpected was going on with the vision embeddings, so we devised this test to create images where we would expect the embeddings to be the same (i.e. small perturbations in the camera position) and just run through the encoder. There are all sorts of other variable that could be at play, but we just ran this quick experiment to check. This is an easy enough experiment to replicate, if anybody else wants to try we can share the code.

[Ishneet_Singh]

Small clarification on the MDS figure above. The thumbnail version was visually intuitive, but it hid some of the structure because the images overlapped and the first two MDS axes only captured part of the distance matrix.

Here is the same SigLIP distance embedding redrawn without thumbnails, plus a third-axis sanity check.

Left: points are colored by camera; marker shape is layout. MDS-1 is basically viewpoint: overhead views A–C sit on one side, table-height views D–F on the other.
Middle: same coordinates, but colored by layout. Fixed layouts do not form their own islands; they stretch across the camera axis.
Right: MDS-1 vs MDS-3. Layout does not cleanly reappear as a hidden third axis either.
So the claim is not “2D MDS perfectly explains everything.” It is narrower: the largest recovered coordinate is camera/viewpoint, and layout is secondary inside that camera geometry.

[Ishneet_Singh]

TL;DR. VGGT-Ω is better in the right way: its patch-token camera/object ratio drops from VGGT-1B’s 2.63× to 1.23×, and its frozen scene/register tokens improve OpenVLA-OFT on LIBERO from 97.1% to 98.5%. It still does not make the token grid viewpoint-invariant; camera moves remain larger than object-layout moves.

I ran the same probe on two geometry-pretrained checkpoints: the public original VGGT-1B and the VGGT-Ω-1B-512. I tested two modes for each. First, single-image encoding: each photo is encoded alone, which is the closest analogue to replacing a monocular VLA vision tower with a geometry-pretrained one.

VGGT feature	brightness	objects moved	camera moved	both	camera / objects
camera token	0.0001	0.0049	0.0294	0.0312	6.00x
register tokens	0.0001	0.0047	0.0299	0.0316	6.36x
patch tokens	0.0005	0.0160	0.0420	0.0447	2.63x
all tokens	0.0005	0.0159	0.0420	0.0447	2.64x

The absolute distances are much smaller than SigLIP/DINO/CLIP, so the scale is not directly comparable, but the ordering does not flip. In monocular VGGT features, camera changes are still larger than object-layout changes. The special camera/register tokens make the camera-vs-layout ratio more explicit.

VGGT’s final token is also a concatenation of a frame stream and a global-attention stream. Splitting it does not change the readout. In single-image encoding, frame-stream patches score 0.0234 for object moves and 0.0624 for camera moves (2.66×), while global-stream patches score 0.0123 and 0.0319 (2.58×).

Second, all-18-view inference: VGGT sees the full grid jointly, which is a reconstruction setting, not VLA training. This makes the separation stronger, not weaker: patch tokens move 8.9× more for camera than layout, and camera/register tokens become explicitly pose-separated. That is expected for reconstruction, but it is not the invariant monocular state a fixed-camera policy would need.

So original VGGT-1B does not make the tokens viewpoint-invariant.

VGGT-Ω feature	brightness	objects moved	camera moved	both	camera / objects
camera token	0.0000	0.0004	0.0044	0.0045	9.90x
register tokens	0.0000	0.0006	0.0063	0.0065	11.12x
patch tokens	0.0758	0.5593	0.6900	0.6895	1.23x
all tokens	0.0746	0.5503	0.6789	0.6785	1.23x

The camera/register tokens are now almost frozen in absolute distance under single-image inference, but their residual variation is still much more camera- than layout-linked. More importantly for a VLA state, the dense patch/all-token grid is not invariant: camera changes still move it farther than object rearrangements. Joint all-view inference makes camera/register pose separation explicit (24× and 51× camera/layout ratios), while patch/all tokens remain camera-sensitive at about 1.3×.

There is one important split inside that patch result. Omega’s final token is a concatenation of a frame stream and a global-attention stream. Separating them does not reveal a hidden invariant state. In single-image encoding, frame-stream patches score 0.2905 for object moves and 0.3741 for camera moves (1.29×), while global-stream patches score 0.6106 and 0.7501 (1.23×). In all-18-view inference, the frame stream is 0.2337 vs 0.3508 (1.50×), and the global stream is 0.5261 vs 0.6648 (1.26×).

The robotics result in the VGGT-Ω paper points the same way. They do not replace OpenVLA-OFT’s vision tower outright; they freeze VGGT-Ω, extract its scene/register tokens, concatenate them with the normal OpenVLA-OFT input tokens, and train OpenVLA-OFT with the standard protocol. Average LIBERO success rises from 97.1% to 98.5%. That is not a proof of mechanism, but it is the right intervention: add geometry-trained scene tokens that reduce the effective viewpoint problem, and the VLA improves.

So the global stream is not less changed in absolute terms; it is more changed. It is a geometry/reconstruction stream, not a camera-invariant patch state by itself. VGGT and VGGT-Ω give structured geometry tokens, but they do not by themselves erase viewpoint from the token grid.

Method caveat: this is an aligned-token representation probe, not a policy evaluation. It asks whether the token at the same grid index moves more under layout or camera changes. It does not test optimal cross-view patch matching, and it does not say a downstream policy cannot learn to use geometry tokens with the right training pressure. It does say that simply swapping in these final patch tokens would still hand the policy a state whose dominant displacement is viewpoint.