Gemini 4 Argon tops Ultralytics benchmark of 26 auto-annotation models

What Ultralytics added

Ultralytics has added Gemini 4 Argon to the class-prompted vision language models in the annotation editor of Ultralytics Platform. On an object detection dataset with 1 to 200 classes, the model looks at each image and returns boxes for the instances of the dataset's classes that it finds. Users then review and save those boxes like any other prediction.

There are two ways to use it. In the annotation editor, users switch to Smart mode, select gemini-4-argon under Google in the model picker, and click Predict. For a whole dataset, Batch Annotation, found under More actions on the dataset page, labels every unlabeled image in a single run.

Gemini 4 Argon sits alongside other auto-annotation options in the platform, from SAM 3.1 point prompts to users' own fine-tuned Ultralytics YOLO models.

How the benchmark worked

Ultralytics argues that a model that handles everyday photos well can still fail on a medical scan or a circuit board. For that reason it used UL37, a set of real-world benchmark datasets the company uses for evaluation, and took its 36 detection datasets. These cover medical imaging, aerial and satellite imagery, PCB inspection, security X-ray, retail shelves, traffic, sports, underwater scenes and thermal footage, among others.

The company sent 108 labeled images, three per dataset, to every class-prompted model in the platform through the same detection code the annotation editor uses, with each dataset's full class list. Returned boxes were scored against ground truth with mAP50. Every provider ran at its default reasoning effort, and according to Ultralytics repeat runs moved scores by only about 0.005 mAP50.

Gemini 4 Argon reached 0.565 mAP50. Behind it came GPT-6.1 Sol (0.491), Gemini 3.8 Flash (0.489), Claude Opus 5.5 (0.476) and GPT-6 Astra (0.470). It completed every request, with a median of about five seconds per image. Ultralytics says the lead does not come from longer thinking: the model used a median of about 150 output tokens per image, about the same as Claude Opus 5.5 and well below GPT-6.1 Sol or Gemini 3.8 Flash.

Across 12 real-world domains, Gemini 4 Argon beat the average of all 26 models in 11 and tied in underwater imagery. Its biggest gains were in roads and traffic (0.59 vs. 0.17) and aerial imagery (0.65 vs. 0.30), and it doubled the average on medical imaging (0.57 vs. 0.28). Industrial PCB and security X-ray images stayed below 0.2 mAP50 even for Gemini 4 Argon.

Cost and setup

Paid models run on the user's own provider API key, so the provider bills each prediction directly. At Google's launch pricing, which runs through November 30, 2026, Gemini 4 Argon cost about $7.60 per 1,000 images in the benchmark. After that date its price doubles to about $15 per 1,000 images at the same usage.

Ultralytics says premium models cost more without scoring higher. GPT-6 Astra and Claude Fable 5.1 cost about $47 and $37 per 1,000 images and scored 0.470 and 0.372. For cheaper options, Claude Haiku 5.5 reached 0.384 mAP50 at about $0.35 per 1,000 images, and Gemini 3.5 Flash-Lite reached 0.363 at about $1.36. Qwen3.8 27B was the most accurate model hosted on Ultralytics Platform, which needs no provider key.

To start, users add a Google API key under Settings > API Keys in the workspace that owns the dataset, or ask the workspace owner to do so. Batch Annotation adds the standard platform processing fee of $1.00 per 1,000 images, separate from what Google bills. Each run saves a dataset version first, so users can roll back.

What it means for annotation teams

For teams labeling detection datasets, the results suggest Gemini 4 Argon is the strongest class-prompted option in the platform, at least on Ultralytics' own test set. The company still advises treating its boxes as a first draft for human-in-the-loop review, and checking a few predictions on your own images before annotating a full dataset. Teams working with PCB or security X-ray images in particular should expect heavy correction.

The launch pricing window also matters for budgeting: costs in the benchmark double after November 30, 2026. Once a dataset is labeled, users can train an Ultralytics YOLO26 model on it in the same workspace.