Decision Models Struggle at Tic-Tac-Toe From Screenshots, Arize Finds

What Arize tested

Decision models take a prompt and a fixed list of options and return a probability for each option. According to Arize, that makes them a good fit for a step inside an agent, such as picking a tool, picking a route or deciding whether to stop. Several now accept images: Cloudflare's Clef Flash, Liquid's d1 models and OpenAI's Decisions route handle them, and TypeSafe's Jev 1.14 added experimental image support.

Arize asked whether a model that can read an image and decide from it can also play a game. It built a small browser arcade with three games, tic-tac-toe, Minesweeper and Flappy Bird, and ran eight models through it. Six are decision models, a mix of local and cloud. The other two are LLMs added as a comparison. Arize also ran a text-only version of each game, where the model gets a description of the game state instead of a picture.

The blog's summary of the overall outcome: the models did "OK, but not great, depending on the game." The author adds that they scored better than most of the AIs despite calling themselves a terrible gamer.

The setup for tic-tac-toe

In tic-tac-toe the model plays a perfect minimax opponent, so a draw is the best possible result. A win scores 1, a draw 0.5 and a loss 0. Each model plays 10 games per input type, five as O and five as X. Every move is a single request with no conversation history. In the image version the arcade sends a screenshot of the board, a fixed prompt and the list of empty squares as choices. The text version uses the same positions, with the board written out as rows.

The prompt tells the model to win if it can, block an immediate opponent win otherwise, and else choose a move that preserves a draw. The final move is played by the arcade itself, because decision models generally reject a single decision prompt.

Arize also built deterministic baselines for each game, such as minimax for tic-tac-toe. It notes these are not competitors on equal terms, since they have programmatic access to the game state. They show what a good score looks like.

Results: LLMs far ahead of decision models

Jev 1.14 was the strongest decision model. From images it drew nine of 10 games, losing only one as X. From text it drew four of 10, all as O, and lost all five games as X. Arize points out that this contradicts TypeSafe's own warning in its launch post that "all models (Jev included) perform worse on multimodality over pure text."

The rest fell away quickly:

• Liquid d1 drew four games per input type.
• GPT-6 Luna Decisions drew one game from images and three from text.
• Liquid d1-3B drew one game from images.
• Clef Flash and Liquid d1-omni-600M drew none.

Across the six decision models that adds up to 26 draws in 120 games.

The LLMs did far better. GPT-6 Astra Responses drew all 20 of its games, 10 from images and 10 from text. GPT-6 Luna Responses drew 19 of 20, with nine draws from images, all 10 from text and one image loss as O. Together the two drew 39 of 40 games.

According to Arize, the decision models lose in a nearly identical way: the opponent has two in a row and the model does not block.

What it means

The results suggest that accepting an image does not mean a decision model can reliably use it in a game, even one as small as tic-tac-toe. In this test, general LLMs handled both image and text input much better than most of the decision models. Jev 1.14 was the exception among the decision models, and its image result beat its text result.

Arize frames the work as a fun question rather than a formal verdict. Anyone considering a decision model for an image-based step in an agent can read it as a reminder to test on their own task first. The Minesweeper and Flappy Bird results are not covered here.