Artificial intelligence news.
Straight from the original announcements.

Visual Cipher Bench Finds Frontier AI Agents Struggle With Hidden Rules

November 18, 2026, 11:39By the AI News Bites Editorial TeamSource: PR Newswire
Visual Cipher Bench Finds Frontier AI Agents Struggle With Hidden Rules
Image: PR Newswire

Einsia AI's Navers Lab has launched Visual Cipher Bench, a benchmark for testing whether AI agents can discover hidden rules from visual clues and solve long-horizon tasks. Across 3,480 attempts involving eight frontier models and 29 reasoning configurations, no configuration exceeded 30% accuracy.

Key points
  • Visual Cipher Bench contains 40 puzzles across four categories.
  • GPT-6 Astra (max) led the evaluation with 26.7% accuracy.
  • Rule induction accuracy was 0.6% across 3,480 attempts.
  • Providing core rules increased overall accuracy from 12.2% to 51.1%.

A benchmark without explicit instructions

Visual Cipher Bench is designed to test a capability that conventional visual question-answering benchmarks generally do not measure: solving problems when the instructions and rules are hidden in images. Agents receive one instruction, examine the images, and write a final answer to a file.

The benchmark contains 40 original visual puzzles represented by 67 black-and-white illustrations. To complete them, an agent must identify examples, understand the target problem, infer the required output, and solve each task without explicit rules or immediate correctness feedback.

The puzzles are divided into four categories: spatial operations such as folding, rotation, projection, and geometric transformations; logical planning; rule induction; and knowledge decoding involving Morse code, Braille, and word puzzles.

GPT-6 Astra leads, but rule induction remains weak

GPT-6 Astra (max) achieved the highest result, with 26.7% accuracy and an average of 10.7 correct answers per 40-puzzle run. GPT-6.1 Sol (max) and Claude Opus 5.5 (xhigh) each scored 21.7%.

GPT-6 Luna and Claude Sonnet 5.5 each scored 0.8%, solving one puzzle across 120 attempts. DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro recorded zero correct answers.

Spatial operations were the strongest category, at 22.8% accuracy across all attempts. Knowledge decoding reached 3.3%, logical planning reached 2.4%, and rule induction reached only 0.6%. Models solved five of 870 rule-induction attempts. GPT-6 Astra (max) reached 80% on spatial operations, while no configuration exceeded 13.3% in the other three categories.

Einsia AI said some correct answers appeared to result from thematic guesses rather than systematic rule discovery, further separating answer execution from independent reasoning.

Rule hints sharply improve execution

An additional experiment tested 10 puzzles with GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna. When researchers supplied the core rules, without providing intermediate reasoning or answers, overall accuracy rose from 12.2% to 51.1%. Rule-induction accuracy increased from 0% to 85.2%.

GPT-6 Astra improved from 23.3% to 70.0%, while GPT-6.1 Sol rose from 13.3% to 66.7%. Timeout occurrences fell from 51 to 29 across 90 attempts. A control experiment that supplied information about task-state representation, but not the core rules, produced 16.7% accuracy.

The researchers identified premature conclusions and inefficient search as recurring failure patterns. Agents sometimes accepted rules that explained only part of the evidence, or used extensive tool calls and combinations until they timed out. The findings suggest that discovering, validating, and transferring hidden rules remains a major obstacle for agents expected to handle less structured real-world tasks.

Related news

Cline-Bench ranks AI agents on 47 real-world coding tasks
ResearchNovember 19, 2026, 19:43

Cline-Bench ranks AI agents on 47 real-world coding tasks

Cline has introduced Cline-Bench, a benchmark designed to measure how reliably AI agents complete real user requests in existing software projects. Its initial…

GPT-6 Astra led coding agents, but finished only 26.4% of tasks
ResearchNovember 10, 2026, 17:08

GPT-6 Astra led coding agents, but finished only 26.4% of tasks

OpenAI’s GPT-6 Astra passed an average of 69.9% of hidden acceptance criteria in a benchmark of coding agents, yet fully finished only 26.4% of tasks. The Co-Design…

Research
ResearchNovember 6, 2026

OpenAI introduces BioBasicsBench to test AI agents in wet labs

OpenAI has introduced BioBasicsBench, a benchmark designed with biotech company Ligo to measure how well AI agents interpret routine wet-lab evidence. Its 100 tasks test…