Kimi K3.1 Pro Ranks 7th of 29 on Ultralytics VLM Leaderboard for Auto-Annotation

October 24, 2026, 03:12Source: Ultralytics

Moonshot AI's Kimi K3.1 Pro is now available for auto-annotation in Ultralytics Platform, where it scored 0.444 mAP50 on the company's VLM Leaderboard. That is a third higher than Kimi K3 (0.334) and places it 7th of 29 models, behind only closed models from Google, Anthropic and OpenAI.

Key points

Kimi K3.1 Pro scored 0.444 mAP50 on the Ultralytics VLM Leaderboard, up from 0.334 for Kimi K3, and ranks 7th of 29 models.
Every model ranked above it is a closed model from Google, Anthropic or OpenAI.
It improved on Kimi K3 on 25 of 36 datasets, with big gains in sports and road scenes.
Cost was about $22.50 per 1,000 images, with a median of about 22 seconds per image and 6.4% timeouts.
Moonshot lists the model at $2 per million input tokens and $10 per million output tokens.

What Kimi K3.1 Pro brings

Moonshot AI released Kimi K3.1 Pro on October 23, 2026. According to Ultralytics, it is the first model in Moonshot's K3.1 series, which Moonshot calls its most capable 3T-class open-weight model to date. It is a post-trained version of Kimi K3, built with substantially more post-training compute, with a 1-million-token context window and image and video input. The model always reasons before it answers. Moonshot plans to release the full weights by November 4, 2026.

Moonshot also lowered the price. Kimi K3.1 Pro lists at $2 per million input tokens and $10 per million output tokens, down from $3 and $15 for Kimi K3.

How it works in Ultralytics Platform

Kimi K3.1 Pro joins the class-prompted vision language models in the Ultralytics Platform annotation editor. On an object detection dataset with 1 to 200 classes, it returns bounding boxes for the instances of the dataset's classes it finds. Users review and save them like any other prediction.

One image at a time: switch to Smart mode, select Kimi K3.1 Pro under Kimi in the model picker, and click Predict (or press P).
A whole dataset: run Batch Annotation from More actions > Batch Annotation on the dataset page to label every unlabeled image in one run.

Users need an API key from the Kimi API Platform, added in Settings > API Keys of the workspace that owns the dataset. Paid models run on the user's own provider key, so the provider bills each prediction directly. Batch Annotation adds a processing fee of $1.00 per 1,000 images, separate from what Moonshot bills. Each run saves a dataset version first, so it can be rolled back.

Benchmark results

Ultralytics tested the model on the same frozen test as every model on its VLM Leaderboard: 360 images, 10 drawn at random from each of 36 real-world detection datasets. They cover areas such as medical imaging, aerial imagery, PCB inspection, security X-ray, retail, traffic, sports and underwater scenes. Images go through the platform's production annotation pipeline, including a SAM 3.1 step that tightens the boxes.

Six models scored higher: Gemini 4 Argon (0.537), Claude Opus 5.5 (0.527), GPT-6 Astra (0.525), Gemini 3.8 Flash (0.516), GPT-6.1 Sol (0.514) and Claude Sonnet 5.5 (0.472). Kimi K3.1 Pro finished ahead of Claude Haiku 5.5 (0.429), Claude Fable 5.1 (0.427) and every open-weight model Ultralytics runs on its own GPUs, led by Gemma 4 31B (0.390).

Precision rose to 0.560 from 0.439 for Kimi K3, and recall rose to 0.483 from 0.376. The model improved on Kimi K3 on 25 of 36 datasets and in 8 of 12 domains. The biggest gains were in sports (0.68 vs. 0.28), roads and traffic (0.49 vs. 0.26) and documents, UI and CAD (0.41 vs. 0.26). On tennis balls it went from 0.08 to 0.74 mAP50, and on CeyMo road markings from 0.17 to 0.80. In sports it beat Gemini 4 Argon (0.68 vs. 0.60) and outscored Argon on six datasets in all.

Not every area improved. Underwater and security X-ray, one dataset each, fell, retail slipped slightly (0.47 vs. 0.48), and industrial PCB defects stayed near zero for both Kimi models.

Cost, speed and caveats

Ultralytics reports that Kimi K3.1 Pro cost about $22.50 per 1,000 images, close to Kimi K3's $21.50. The lower token price is offset by higher token use per image, so the upgrade delivers about a third more accuracy for roughly the same bill. Five of the six higher-ranked models cost less, including Gemini 4 Argon (about $7.60 per 1,000 images) and Claude Opus 5.5 (about $13.90). Claude Haiku 5.5 came within about 0.015 mAP50 at about $0.37 per 1,000 images.

The model is also slow. Its median request took about 22 seconds per image, against 4.6 seconds for Gemini 4 Argon and 2.7 seconds for Claude Opus 5.5. In the tests, 6.4% of requests did not finish within the platform's 115-second limit, up from 1.7% for Kimi K3. A timed-out image returns no boxes and can be predicted again.

Ultralytics runs all Kimi models at high reasoning effort rather than Moonshot's default maximum, because Kimi K3 exceeded the time limit on about 40% of images at max. Moonshot also offers Kimi K3.1 Pro Highspeed, which it says runs two to three times faster with the same capabilities, but the platform offers the standard version.

What it means for users

For teams that want an open-weight-lineage model for labeling, Kimi K3.1 Pro is a clear step up from Kimi K3 at a similar per-image cost. Based on Ultralytics' own benchmark, though, several closed models are more accurate, and most are faster and cheaper. Ultralytics advises treating the boxes as a first draft for human review and checking a few predictions on your own images before annotating a full dataset, since specialized imagery remains hard for every model.