Roboflow Adds Action Recognition for Video-Based Workflow Analysis
Roboflow has added support for action recognition, enabling teams to train and deploy vision models that identify actions across video and return a timeline with start and end times. The company says the capability can turn recorded footage into structured data about what happened, when it happened, and in what order.
- Roboflow now supports training and deployment for action recognition models.
- Supported models are Cosmos 3 Edge and V-JEPA 2.1.
- Models return action timelines with start and end times.
- V-JEPA 2.1 scored higher on all three datasets tested by Roboflow.
- Outputs can support compliance checks, cycle-time analysis, alerts, and automation.
From video clips to action timelines
Action recognition is designed for tasks where timing and sequence matter. Roboflow says teams can use it to check whether a robot completed every step of a program, whether a pit crew followed each stage of a stop, or whether a gymnast performed the elements of a routine in sequence.
The company lists several operations use cases, including standard operating procedure compliance, cycle-time measurement, kitting, packing, changeovers, and robotics. A model can flag a clip when a step is missing, measure how long each step takes, identify retries in a robot program, and label hours of existing recordings. The output can also be sent to another system for analysis, alerts, or automation.
Unlike object detection, which examines what is present in an individual frame, action recognition examines a window of frames to capture motion over time.
Two supported models and three test datasets
Roboflow supports Cosmos 3 Edge and V-JEPA 2.1. Both models process video in 16-second windows, labeling moments in each window before merging results into a timeline for longer clips.
V-JEPA 2.1 is a video encoder from Meta. Roboflow describes it as the smaller and faster-training option, with a confidence score for every action. Cosmos 3 Edge is a video language model from NVIDIA that uses more motion and time context, which can help distinguish steps that look similar in a single frame.
Roboflow trained both models on three public datasets and evaluated them on test videos not used during training. V-JEPA 2.1 recorded higher accuracy on each dataset: 99.7% compared with 96.8% for Cosmos 3 Edge on robot manipulation, 92.8% compared with 72.1% on robot assembly, and 87.3% compared with 80.0% on gear train assembly.
According to Roboflow, Cosmos 3 Edge often placed action boundaries more precisely, which may matter when exact timing is important. The company also cautions that results on a customer’s own data will differ.
Training and deployment workflow
Users create an Action Recognition project, upload videos, and label time ranges in the annotation editor. Roboflow recommends recording training clips with the same camera angle and lighting used in production, while showing the full task. It also recommends labeling a repeated step as one segment when it occurs back to back. On the gear train data, Roboflow says this increased accuracy from 59.8% to 83.4%.
Users can then create a dataset version, choose Cosmos 3 Edge or V-JEPA 2.1, and set a credit cap before training. V-JEPA 2.1 is in Beta on paid plans. Roboflow says the gear train run reached 87.3% accuracy with V-JEPA 2.1 after about 8 hours, while Cosmos 3 Edge reached 80.0% after about 12 hours.
Models can be tested through Roboflow’s SDK, connected to live cameras through a Workflow, or run on a user’s own GPUs with Roboflow Inference. For production, Roboflow recommends fitting the longest action into one 16-second window, lowering the stride for live streams when tighter timestamps are needed, and checking the full sequence rather than isolated steps.
