Anyscale Uses Ray and a Physics Twin to Probe What a Robot Policy Sees

A frozen policy in a twin of its own rig

In a blog post dated October 9, 2026, Ian Jordan, PhD, describes a study built on LeRobot's pi0.5 laundry-folding checkpoint, published as lerobot/folding_latest. According to the post, the policy is pi0.5 fine-tuned by LeRobot on real bimanual OpenArm t-shirt folding. It combines a 3B-parameter PaliGemma VLM with a 300M flow-matching action expert. It reads three cameras (a head camera at 640 x 480 and two wrist cameras at 1280 x 720), a 16-dimensional joint state, and the instruction "Fold the T-shirt properly".

Anyscale says it never modified or fine-tuned the policy. Every call used the public checkpoint in bf16. The twin runs on Isaac Lab 3.0 with the Newton physics backend, and the shirt is a Newton VBD cloth with 25,610 nodes. The policy runs closed loop at 30 Hz on the twin's rendered images and joint state. According to the post, no real pixel is ever placed into an observation that a rollout consumes.

The experiment: swap one image, count what the policy plans

The method is simple. The team takes an observation the policy saw in the twin and replaces one of its three images with a real frame from the same phase of the task. Everything else stays identical, and the team counts what the policy plans. Anyscale argues that a real robot cannot offer this kind of test, because no two resets match, while a physics twin can.

The company reports three results:

• The policy's lift-or-release decision sits in a few hundred pixels at the jaw tips of its wrist cameras.
• The finding showed what the twin's grasp had to look like.
• It produced what Anyscale calls its first zero-shot fold-overs of a cloth shirt by a frozen VLA in simulation, with no real pixels and no fine-tuning.

The post frames the problem as one of blind data collection. When a VLA that works on a real rig fails after a camera move or a switch to simulation, fine-tuning on more data is the usual answer, but it does not say what to collect. The post cites LIBERO-Plus, where viewpoint or initial-state changes drop success from 95% to under 30%, as a benchmark that shows the failure but not its cause.

How Ray runs it

The whole study is one Ray program. A policy service runs on one GPU, three twins run across three GPUs, and the probes act as concurrent clients of the same service. The policy is deployed with Ray Serve and returns a whole 30-step action chunk per request without keeping per-client state. That statelessness lets many clients share one replica, and concurrent calls, including a probe issuing eighty at once, are batched through a single forward pass.

Sharing matters because, according to the post, three simulators would otherwise each load 3B parameters on their own GPU, and one 24 GB L4 cannot hold Isaac Sim, the cloth solver and the policy together. Each twin is a Ray task holding one GPU and runs as a subprocess, since Isaac Sim's Kit event loop must not live inside a Ray worker. Retries are disabled on purpose, because a silently retried twin would re-run from scratch while overwriting its own frames.

Because a twin boot costs a minute or three and a settled shirt costs another two hundred simulation steps, the notebook starts twins from saved holds. Anyscale says the notebook reproduces the core of the study in six minutes on four L4s.

Limits of the grasp proxy

The post is open about one proxy. A friction pinch on simulated fabric of this thickness does not hold, so when the gripper closes, cloth nodes between the pads are attached to the gripper frame and carried. Anyscale adds a pinch fold, bunching the fabric between the pads, because that is what the wrist camera has to see. As the post puts it, a pinned grasp always holds, so the twin validates reaching, folding strategy and visuomotor control, not grasp robustness.

A stricter test also checks whether the jaw was actually over the fabric when it closed. Only the left jaw passes in the shipped states. The right hand's pin attached nodes from several centimeters away, which the proxy allows and a real gripper cannot. The saved segments therefore exercise the restore path and the shared replica but are not evidence of a two-hand grasp.

What it means

Anyscale's pitch is that once closed-loop evaluation is cheap, a simulator becomes an instrument for asking what a policy depends on. The post says such answers can inform where a camera goes and what a simulator or synthetic-data pipeline must get right for a frozen policy to keep working. The post also notes that activation-level tools for finding where in the network a decision lives are the next step once the twin exists. The results come from one policy and one rig, and the grasp proxy limits what they say about real-world grasping.