GlobeBench: No AI Model Solves Half of Text World Simulation Tasks

What GlobeBench measures

GlobeBench covers five domains and 19 simulation skills. Each task gives a model the full agent trajectory and a final action, then asks it to predict the environment's response. The prediction is compared with a reference derived by actually executing the action in the item's backend.

Patronus AI says 1,137 of the 1,632 items were built with trajectories the team designed itself. The other 495 are adapted from executable open-source backends such as BFCL, TextWorld, τ²-bench and SymPy. Every item passes 12 automatic quality checks covering answer consistency with the visible history, reproducibility of the gold answer, and whether the item is unambiguous and not a duplicate. The paper is on arXiv and was accepted to the NeurIPS 2026 IAEval workshop.

The skills map to four capabilities: State Coherence, Causal Correctness, Interface Fidelity and Constraint Enforcement. A skill is a specific environment behavior, such as remembering a value written many turns earlier.

Why simulators matter

Patronus AI argues that as reinforcement learning scales to software engineering, computer use and scientific discovery, building stable, high-fidelity sandboxes remains inefficient and hard to maintain. Language models used as simulators have improved downstream results in recent work, the company notes, citing Qwen's AgentWorld and a NeurIPS 2026 paper by Patronus AI's Darshan Deshpande.

The company's view is that today's models are not good enough to replace real RL environments. If a simulator is wrong in ways an agent can exploit, training teaches the agent the simulator's mistakes rather than how the real software behaves. Patronus AI says no earlier benchmark showed which parts of simulation break, which is why it built GlobeBench.

One example task involves a Linux shell where a file ends up holding the label russet-willow-54. The final action asks for the file's line count, byte size and a checksum. The correct answer is 17 bytes, the 16-character label plus the trailing newline added by the write command. Nine of the 20 models answered correctly. Kimi K3 and GLM 5.3 predicted 16 bytes, so their checksums were also wrong. Four Claude models refused.

Results across 20 models

The benchmark covered 18 general-purpose large language models and two models built specifically as digital world models. Each model got one answer per item, with no examples in the prompt and no rerolls. The general models ran at their provider's high reasoning setting. Refusals counted as failures.

Scoring checks whether the predicted state is correct, not whether the answer merely looks plausible. Deterministic gates settle predictions that provably match or contradict the correct answer, and an LLM judge handles the rest.

• GPT-6 Astra: 49.57%, or 809 items.
• Grok 4.6: 28.13%.
• Claude Opus 5.5: 26.35%.
• Qwen 3.8 Max: 25.31%.

Astra's lead over second-place Grok is 350 items. The pooled success rate across all models is 19.68%.

Pass rates differ widely by skill. Schema Conformance, with 200 items, is the hardest at about 6%. Content Repetition, with 11 items, is the easiest at about 66%.

Refusals and what the results imply

Four Claude models refused 3,696 answers, all counted as failures. According to Patronus AI, nearly all refusals come from Anthropic's safeguards, which block requests that could enable cyber harm, such as malware, and can also block benign cybersecurity work. Claude Fable 5 had 1,309 items with no response, Claude Opus 5 had 1,048, Claude Fable 5.1 had 750 and Claude Opus 5.5 had 678. Across all models, 3,786 of 32,640 decisions had no response, including 90 empty completions.

For teams considering model-based simulators for agent training, the results point to a clear gap. Even the top model fails about half the tasks, and Patronus AI says the benchmark is meant to show where current shortcomings lie.