MIT Researcher Adapts Ai2's Molmo 2 to Track Salmon in Sonar Video

Why fish tracking needs a better approach

Kai Van Brunt, an MIT '25 graduate, works on computer vision for ecology. He carried out the project as a visiting student in MIT's BeeryLab with Sara Beery, an assistant professor of electrical engineering and computer science at MIT. His research focuses on better ways to count salmon from underwater sonar video. According to Ai2, those counts help researchers understand how salmon populations are changing and inform conservation and fisheries management.

Following individual fish through the footage is key to getting the counts right, but doing it by hand takes a lot of time. Computer vision can reduce the amount of video researchers must review, yet tracking systems still make errors. They can lose sight of a fish, "forget" which is which, or switch from one fish to another. Fixing those errors can mean manually correcting predictions across many frames.

Van Brunt asked whether a researcher could instead describe how a prediction was wrong and have the model correct it automatically. That idea became Molmo2Fish.

Building on Molmo 2

Ai2 released Molmo 2 as an open vision-language model for images and video, and made the model, code and training data openly available. According to Ai2, Molmo 2 can follow a particular figure or object as it moves through a video, identify something described in language, and reason about events over time.

Sonar footage looks very different from the videos Molmo 2 originally learned from. Even so, the model already knew how to follow a target through video and use language to pick which one to follow. Van Brunt had noticed that watching fish move made them easier to tell apart than looking at a single frame. He tried several video object detection models, but none improved on the still-image baseline.

Molmo 2 could process a whole clip and output tracks directly, so training on the tracking task did not need a separate detection step. "Molmo 2 was the first model we used which made it straightforward to consider multi-object tracking as a task that a model could learn directly," Van Brunt says. He also said he needed an open model that understood language, spatiotemporal relations and tracking, and that was small enough to fine-tune.

Correcting tracks with plain language

After fine-tuning on salmon videos, Molmo2Fish performed roughly as well as a specialized fish-tracking system before receiving any corrections, with results varying across rivers.

Van Brunt then trained it to respond to user feedback. The model saw examples of an initial prediction, an instruction describing how it should change, and the corrected result. A researcher could say that Molmo2Fish had missed a fish or treated two separate fish as one, then ask for an automatic fix.

This worked especially well when Molmo2Fish was given predictions from a conventional fish-tracking system. The two tended to make different kinds of mistakes, so Molmo2Fish could fix some errors the other system produced. The experiments also showed limits: the model could correct some errors given clear instructions, but struggled to improve its own strongest predictions.

What the open release made possible

Van Brunt did not ultimately use Molmo 2's original training data to build Molmo2Fish. Still, being able to inspect it helped him understand what the model had already learned and decide how to prepare his own data. Access to the code made it easier to tailor Molmo 2 to a very different kind of video.

Van Brunt calls Molmo2Fish a first step toward a model reliable enough for real-world use. He now wants to explore how models that handle both video and language could help scientists find and describe patterns as their questions change. "Broadly, I want to continue exploring learning on video data, and I want to use AI to understand and protect the natural world," he says.