Gemini 4 Hydrogen ranks third in Ultralytics auto-annotation test

Gemini 4 Hydrogen ranked third among 46 vision-language models in Ultralytics’ auto-annotation benchmark, scoring 0.553 mAP 50 across 36 object detection datasets. The model is available for auto-annotation in Ultralytics Platform, but its token usage gave it a higher estimated cost per image than several competitors.
- Gemini 4 Hydrogen scored 0.553 mAP 50 and ranked third among 46 VLMs.
- Ultralytics estimated its provider cost at $20.32 per 1,000 images.
- Hydrogen led on PlantDoc, SeaDronesSee 2 and IEDXray.
- The model tied for first on Kvasir-SEG and EgoHands.
- Hydrogen is available for auto-annotation in Ultralytics Platform.
Hydrogen places third on Ultralytics’ leaderboard
Ultralytics said Gemini 4 Hydrogen scored 0.553 mAP 50 and 0.350 mAP 50-95 on UL37, its benchmark for object detection auto-annotation. The model ranked behind Gemini 4 Carbon, which scored 0.584, and GPT-6.1 Terra, which scored 0.560. It finished ahead of Claude Fable 5.2 and Gemini 4 Argon.
UL37 tests how closely a model’s predicted boxes match human labels on 360 images. The benchmark includes 10 images from each of 36 real object detection datasets, covering areas such as thyroid ultrasound, thermal drone footage, retail shelves, floor plans and circuit boards. Ultralytics says every image follows the same Platform pipeline and uses the dataset’s own class names, with every dataset counted equally.
Hydrogen’s 0.350 mAP 50-95 score, a measure that also rewards tightly fitting boxes, was second only to Carbon’s 0.371. The model recorded precision of 0.63 and recall of 0.62, with no failed requests and no refusals. Its median processing time was 7.2 seconds per image.
Lower token prices do not translate into lower image costs
Google lists Hydrogen at $1.50 per million input tokens and $7.50 per million output tokens. Those rates are lower than Carbon’s introductory prices, but Ultralytics estimated Hydrogen’s provider cost at $20.32 per 1,000 benchmark images. The estimate excludes Platform’s processing fee, which adds $1.00 per 1,000 images for every model.
For comparison, GPT-6.1 Terra was estimated at $4.36 per 1,000 images, while Gemini 4 Argon was estimated at $7.60. Claude Fable 5.2 was estimated at $37.55. Carbon’s introductory rate produced an estimated cost of $12.93, although Ultralytics said the figure would be about $25.87 under the announced rates starting December 1.
Ultralytics attributed the difference to token usage. A vision-language model bills for the image and prompt it reads, as well as every token it returns, including reasoning tokens. Hydrogen’s token prices are 75% of Carbon’s introductory prices, but its benchmark run used roughly twice as many price-weighted tokens per image as Carbon’s. Against Gemini 3.8 Flash, Hydrogen used about 1.7 times as many price-weighted tokens per image, according to Ultralytics’ calculations.
Strong results on plants, sea rescue and X-rays
Hydrogen recorded the highest score among the 46 tested models on PlantDoc, SeaDronesSee 2 and IEDXray, and tied for the top score on Kvasir-SEG and EgoHands.
On PlantDoc, which contains 29 classes of healthy and diseased crop leaves, Hydrogen scored 0.638. It scored 0.596 on SeaDronesSee 2, a dataset involving swimmers, boats and life-saving equipment in maritime footage. On IEDXray, which covers concealed explosive device components in baggage X-rays, it scored 0.315. Ultralytics described that dataset as difficult, noting that all models scored low.
The model scored 0.805 on Kvasir-SEG, a polyp endoscopy dataset, tying the top result. It also scored 0.995 on EgoHands, tying GPT-6 Astra. Results varied sharply by dataset, however. Hydrogen scored 0.000 on DeepPCB, 0.013 on TBX11K and 0.021 on GroundCUA. Ultralytics advises treating results from individual datasets as signals to test on a user’s own images, rather than as final judgments.
Hydrogen is designed for detection datasets with 1 to 200 classes and returns boxes using a dataset’s own class names. Users review those annotations before training. Ultralytics also says medical image boxes should be treated as drafts for clinical experts to check and approve, not as diagnoses. The results suggest Hydrogen may be useful for selected annotation workloads, but its image-level cost and uneven performance make testing on representative data important.
Ultralytics: Gemini 4 Hydrogen auto-annotation: cheaper per token, pricier per image


