Gemini 4 Carbon tops Ultralytics’ VLM auto-annotation benchmark

Carbon leads across the main accuracy measures

Ultralytics ranked Gemini 4 Carbon first in its November 11, 2026 VLM Leaderboard snapshot. The model scored 0.584 mAP 50 and 0.371 mAP 50-95 on UL37, a benchmark built from 36 object detection datasets.

Carbon also recorded the benchmark’s highest precision, at 0.685, and recall, at 0.642. Precision measures the share of drafted boxes that match the expected objects, while recall measures the share of labeled objects that receive a box. According to Ultralytics, this combination means Carbon’s first annotations leave reviewers with fewer boxes to delete or objects to draw.

UL37 evaluates 360 images, with 10 images from each dataset. The collection covers areas including medical scans, aerial and thermal imagery, retail shelves, sports, documents and circuit boards. Each dataset counts equally, and the images run through the same production pipeline used by Ultralytics Platform users.

Strong results across varied image datasets

Carbon recorded the best score on datasets spanning nine domains, according to Ultralytics. Its results include 0.955 on LLVIP infrared images, 0.942 on Cheng Brain Tumor MRI, 0.882 on Basketball and Hoop, 0.821 on FloorPlanCAD 2, and 0.788 on RUOD Underwater.

It also led on HIT-UAV thermal drone imagery with 0.689, SKU110K shelves with 0.671, PlantDoc with 0.594, Airbus Oil Storage with 0.381 and DDR retinal lesions with 0.133. On some datasets, the margins were extremely narrow. Carbon tied the previous best score on RPC and CeyMo, and led on DDR by 0.001.

The crowded SKU110K retail dataset illustrates the potential value of the result. Its 10 benchmark images contain 1,666 labeled products in one class. Carbon’s 0.671 mAP 50 score surpassed Terra’s previous best of 0.619.

Terra remains the value choice

GPT-6.1 Terra placed second with 0.560 mAP 50. It scored 0.345 mAP 50-95, compared with Carbon’s 0.371. Terra also remains the best model on two datasets in the leaderboard table: TN5000 thyroid ultrasound, at 0.935, and TXL-PBC blood cell microscopy, at 0.889.

The largest difference between the models is cost. Ultralytics estimates provider costs of $12.93 per 1,000 images for Carbon and $4.36 for Terra. Carbon’s median time per image was 7.6 s, compared with 5.1 s for Terra. Carbon had 1% failed requests in the benchmark, while Terra had 0%.

Google charges Carbon an introductory rate of $2 per million input tokens and $10 per million output tokens through November 30, 2026. Those prices become $4 and $20 on December 1, according to the source. At the same token usage, Ultralytics estimates Carbon’s cost will rise from $12.93 to about $26 per 1,000 images.

What the result means for dataset labeling

Ultralytics says Carbon may be suited to crowded retail scenes, thermal and aerial imagery, floor plans and CAD drawings, and datasets where its model-page results show a clear lead. Terra may be the better fit for large general-purpose batches where the budget matters more than a 0.024 mAP 50 difference, as well as data resembling the TN5000 and TXL-PBC datasets.

Carbon became available to all paying developers and enterprises on November 10, 2026, and began rolling out to Google AI Ultra subscribers. The model is available through the API in Google AI Studio and Gemini for business, and through the Gemini app, AI Mode and Antigravity for Ultra subscribers. Ultralytics Platform users can review Carbon’s recorded boxes and expected annotations on the model page before choosing a model.

Batch Annotation adds Ultralytics Platform’s processing fee of $1.00 per 1,000 images, with a $0.01 minimum per run. The provider bills the user’s own key directly, and Ultralytics says it adds no markup to tokens.