Mercor cuts AI batch jobs from 12 hours to 2 hours with Vercel AI Gateway

Model capacity became a deadline problem

Mercor connects hundreds of thousands of domain experts with frontier AI labs, helping them generate the human data needed to train and evaluate models. It also runs large batches of automated grading jobs to evaluate models for labs and enterprises. Demand arrives in bursts that can push model traffic to peak capacity for days, while individual evaluation or labeling jobs carry 24-hour deadlines.

Mercor's mix of direct integrations and third-party gateways could not provide enough token-per-minute capacity for its open-model workloads when demand surged. Requests hit provider limits and entered retry queues, and time-sensitive batch jobs stretched beyond 12 hours, consuming more than half of a client's 24-hour deadline.

According to Vercel, AI Gateway added inference capacity and failover routing between providers when requests hit rate limits. Mercor reports that those jobs now finish in 2 to 3 hours. In its first 10 days on Gateway, the team routed 18 million requests and 63 billion tokens without hitting rate limits. Mercor still uses multiple gateways and vendors, valuing Vercel for model access, controls and hands-on support when it needs more capacity.

Budgets and expiry dates for engineers

As model use grew, Mercor needed to see where token spend was going, but its previous setup offered little visibility by team and project. A single loop error in a script could exhaust the available budget.

Hundreds of Mercor engineers now use open models through AI Gateway with per-user keys that carry expiration dates, token limits and spend limits. Security can give experimentation a budget and an end date, then use usage records to see which user, project and workload is driving consumption. Engineers get one API for leading foundation model providers and a growing catalog of open models, so they can try a new model without building another provider integration or managing new credentials.

Business teams build, Passport handles sign-in

A few hundred people across Mercor's business teams now build applications on Vercel. The company's project count has grown from fewer than a dozen to roughly 2,000, and many are temporary environments for a specific client engagement. One production use case gives experts detailed views of their project tasks and progress, built by business teams rather than added to the core platform by engineering.

Passport provides a common authentication layer, handling sign-in to Vercel deployments through an existing identity provider. Mercor's identity team built a product called Audiences that integrates Vercel Passport. Once a team selects the experts who need access, Audiences creates the IdP group, Vercel connection and Passport configuration. Mercor reports provisioning access in seconds for audiences ranging from dozens of experts to tens of thousands.

Offboarding works in reverse. Because access derives from the IdP group, removing the group at the end of an engagement removes access to the project, leaving no stale access behind.

Narrow permissions for agents and what comes next

Mercor also uses agents to answer questions from experts in its marketplace, automate HR workflows and help employees navigate internal systems. Its approach is to give each agent narrow permissions suited to its job, rather than everything available to the person who invoked it. For agents using open models through AI Gateway, Mercor can scope model access, set expiration dates and cap spending, while separately deciding which systems and actions each agent can reach.

Mercor expects more open models to power its internal agents and systems, and more teams outside engineering to build tools for experts. The company says its aim is to make the secure path the fastest path, so teams spend less time on model integrations and application authentication and security work does not grow at the same rate as the building.