Vapi's in-house review bot cuts pull request merge time from 130 to 65 hours

Why Vapi built its own reviewer

Lat Ware, a Member of the Technical Staff at Vapi, began building the bot when pull requests at the company took an average of over 130 hours to merge. Vapi already paid for two AI review tools, but according to the post, neither caught what his early prompts caught on Vapi's codebase.

Building in-house changed the economics. "I was competing with review times measured in days, so I could let the bot spend 20 minutes on a review, which the other products could not do," Ware says in the post.

The bot runs when an eligible pull request opens, including drafts, and authors can tag it for another pass. It skips commits it has already reviewed, so duplicate events do not trigger a second paid review. Each inline finding opens with a bold one-sentence summary and a severity marker, from green for a nitpick to red for a blocker.

A second tool, Pedantic Architecture Review Bot, looks for design problems. Ware built it after a design flaw got past the original bot. It uses almost the same prompt but asks for "possible high-level design pattern problems" and needs a higher effort level. A complexity filter limits when it runs: between Sep 2 and Oct 2, 2026, 763 architecture workflow runs produced 33 reports.

Proving claims and limiting permissions

The first version left so many comments, many about problems that already existed in the code a pull request touched, that Ware named it Pedantic Review Bot. Some engineers complained that its reviews read like careless AI output. He revised the prompt to make comments shorter and easier to act on, and the complaints faded as engineers found the findings accurate. Fixing existing problems in the code being changed became part of the team's maintenance work.

The bot does not report an objection until it has checked it. During analysis, the agent can trace callers, run tests it suspects will fail, or write a small reproduction. Analysis and posting run as separate jobs. The analysis session has its Git credentials stripped and writes findings to a JSON file. A second job validates the file and posts comments without running anything the agent wrote. The bot can comment but cannot approve or push, so engineers decide what to act on.

Between Sep 2 and Oct 2, 2026, the bot reviewed 1,694 pull requests and raised 363 blocker findings across 276 of them. More than half of those blockers involved builds, tests, CI, or deployment, and others concerned runtime behavior and billing. Vapi cites a change that replaced test assertions with Storybook stories the test runner never collects, and a benchmark that reported success without running a single test. In a traffic-splitting change, the architecture bot caught two writes in the wrong order, which meant a release meant to get no traffic briefly got all of it.

What the prompt experiments showed

Ware has experimented with nearly 200 prompts. In one documented test, he ran 101 versions of the opening instruction on a single pull request. They produced 803 raw findings, grouped into 57 distinct issues. An AI judge rated 32 as real, 24 as false and one as unverifiable. No single run found more than 10 of the 32 real issues.

The report first named a more aggressive prompt as the winner, but Ware rejected that because the scoring did not penalize false positives enough. Once they were weighted as he intended, the original, ultra-simple prompt won. It asks the model to aggressively review a pull request for possible problems, post inline comments, make problems easy to solve, add a bold summary and assign a severity rating. Instructions about what to inspect and how to report performed better than reviewer personas, according to the post.

A second test compared budgets on a refactored production service with a known blocker. Three runs at extra-high effort found more than one maximum-effort run for the same cost or less. For code review, three runs of Sonnet 5.5 cost $12.32 and scored 44 severity-weighted points combined, against $12.30 and 35.5 on average for one maximum-effort run. For design review, three extra-high runs cost $6.36 for 68 points, against $11.20 and 51 for one maximum-effort run. The known blocker was hard to find: only eight of 194 code-review runs caught it, and two of 195 design-review runs.

Scope also mattered. Several GPT-6 configurations reported almost nothing because they treated problems in the parent commit as out of scope. After Ware added the sentence "Include errors already present in the parent," GPT-6 Astra's average in the code-review experiment rose from 0.3 to 24.2 points per run. The other models never received that instruction, so the scores are not a fair comparison. In the design experiment, every point GPT-6.1 Sol earned concerned a pre-existing problem.

Limits and what it means for teams

Vapi is open about the limits of its data. Each prompt in the 101-version test ran once, on one pull request, with an AI as judge, and the model comparison also used a single pull request. Some architecture reports were deliberate re-runs on pull requests merged months earlier, so they do not count as catches before merge. The activity report also does not yet track which findings engineers acted on or dismissed.

For teams weighing similar tools, the post points to a few lessons: running several cheaper passes can beat one expensive one, scope instructions can change results sharply, and a useful finding should say whether a pull request introduced a problem or merely inherited it. Vapi's own production result so far is the drop in average merge time from over 130 hours to 65.