Qodo test finds Jev adds one missing test check to AI code review

What Jev added after Qodo

The author evaluated Jev, which she describes as TypeSafe's structured-judgment model, alongside selected automated checks and Qodo code review. The one added contribution came in case c020, part of a Controlled Release Lab that demonstrates guarded rollouts. The gate in that case should stop a rollout when one workspace's queries become slow while aggregate latency stays healthy.

The code implemented that behavior correctly, but its submitted tests covered only healthy behavior, aggregate failure and invalid observations. They missed query-specific isolation. An independent probe slowed selected workspace requests to 900ms while aggregate p95 stayed at 20ms. Two assertions isolate the behavior: expect(result.reasonCodes).toContain('QUERY_LATENCY_HOLD') and expect(result.reasonCodes).not.toContain('LATENCY_HOLD'). The first confirms the query gate held the rollout, and the second rules out an aggregate hold as the explanation. A stronger test also caught a seeded defect that exempted workspace queries.

Qodo's six findings in that case addressed other problems, and adjudication identified none as this required gap. The combined workflow requested it across three passes. Existing blockers left the top-level recommendation unchanged, so the author notes that counting only recommendation changes would miss the contribution.

How the comparison was built

The revised protocol compared four setups: selected checks, checks plus Qodo, checks plus Jev, and all three. Cases paired correct code with adequate tests, correct code with a missing test, and defective code missed by submitted tests. Independent executable checks established the labels and were kept out of reviewer inputs. There were six pilot cases and 24 evaluation cases. The author pinned the model jev-1.13.0 and reused complete Qodo CLI 1.0.3 reviews after identity checks.

The setups did not have equal evidence access. Qodo had repository access, while Jev received fixed source and test evidence. The author says the comparison therefore cannot establish a model ranking under equal conditions. Selected suites, typecheck and build also did not represent all CI.

Qodo identified eight seeded defects and 15 of 16 required gaps, while both Jev workflows requested all 16. The original protocol supplied verification obligations in prompts, and Jev returned choices and confidence without a written reasoning trace. The author stresses that these are requests for known checks and do not establish unprompted discovery or general precision.

Unsupported concerns and cost

Unsupported concerns affected cases c013, c014 and c028. Case c028 required idle-client expiry to free released clients' entries while preserving active concurrency permits. Its submitted test exercised more than 1,024 released clients over at least 55 seconds and retained four occupied permits, and a separate saved 20-requests-per-second probe also passed. Even so, in the combined workflow Jev reported a violation and missing coverage in all three passes, with violation confidence crossing 0.8.

Under the fixed policy, sufficiently confident violations request fixes, while lower-confidence concerns or missing coverage request verification. As a result, Qodo's verification request became a fix request that was unsupported within the named requirement. Fix requests rose from 18 to 21: two reclassified defects Qodo had already identified, plus c028. TypeSafe derives confidence from the answer's probability distribution, and the author cautions that 0.8 does not establish 80% correctness or a safe change.

Across the 24 cases, Jev added one required gap, zero additional confirmed defects, three unsupported concerns and one ambiguous requirement concern, c023. The author says these categories overlap with legitimate findings and should not be combined into a success rate.

The primary collection completed 252 calls across three passes and used approximately 3.88 million reported input tokens, including pilots. For the 24 evaluation cases, combined-workflow medians were 0.62 seconds for ordinary cases and 3.26 seconds summed over five HTTP requests for a query case pass, with a median input of 102,626 tokens for the latter. These figures exclude preparation, adjudication, human work and reused Qodo time. Qodo token usage was unavailable, and developer-time savings were not measured.

What it means for teams

The author concludes that the evidence supports further coverage evaluation, while net benefit remains unknown. All 30 cases were previously seen, Qodo reviews were reused, and one analyst both built and adjudicated the cases. Passes and packets were repeated observations, not independent cases. In the matched 21-case comparison, final recommendations were identical in both versions.

Before letting extra judgments influence approval, the author wants requests bound to named behaviors, traceable evidence and preserved blockers. She calls for fresh cases and measured maintainer effort that count both useful verification and unsupported requests. A recommendation did not execute a fix, approve a change or establish production behavior.