CodeRabbit says GPT-6 Cyber increased security findings by 66%

GPT-6 Cyber found more selected issues

CodeRabbit evaluated GPT-6 Cyber in CodeRabbit Secure by comparing its existing model ensemble with and without the cybersecurity model on the same 33 cases. Every case contained a known security issue that the scan was expected to find.

The ensemble with GPT-6 Cyber identified 22 of the selected issues, compared with 19 without it. That raised the known-issue catch rate from 57.6% to 66.7%, an increase of 9.1 percentage points. CodeRabbit also describes the result as a 15.8% relative increase in selected issues caught.

The cases covered Python, JavaScript, TypeScript, Go, PHP, Rust, Ruby, C, and C#. CodeRabbit scored an issue as caught only when it appeared in the final report with an explanation of how the code could fail and what security consequence could result.

Report volume rose sharply

GPT-6 Cyber increased the number of high- and medium-severity findings in the final reports from 218 to 361, a rise of 65.6%, rounded by CodeRabbit to 66%. Findings at all severity levels increased from 592 to 979, or 65.4%.

The three-issue net gain came from five additional catches and two catches that the configuration without GPT-6 Cyber had made. Nine selected issues remained undetected by both configurations. CodeRabbit says the additional findings still require assessment to determine how many describe valid vulnerabilities.

Reported findings increased in all six categories selected for analysis. Injection and server-side request forgery had the largest percentage increases, while denial of service added the most findings, with 98. Authorization bypass added 77.

The evaluation focused on final developer reports

CodeRabbit counted only findings that reached developers after verification and reporting. A potential issue raised during the investigation did not receive credit unless it appeared in the final report and explained the possible failure.

The investigation used high reasoning effort to trace potential vulnerabilities through code. CodeRabbit did not compare different effort settings in this evaluation, so it cannot isolate the effect of that setting. Its next comparison will change effort at one stage at a time, using the same cases and scoring criteria. It plans to compare retained known issues, complete-scan time, and cost, including verification.

According to CodeRabbit, GPT-6 Cyber is OpenAI’s cybersecurity variant of its GPT-6 models. The company says Cyber models are trained for security research, including finding vulnerabilities and testing whether they can be exploited, with fewer refusals on authorized security tasks. CodeRabbit received access through the OpenAI Daybreak Defense Network, and access to Cyber models is subject to approval.

What the results mean for CodeRabbit users

CodeRabbit says the findings support evaluating GPT-6 Cyber for security-sensitive scans, while retaining the issues already caught by the existing ensemble in the regression set. The company’s integration work will focus on making the additional detections useful to developers before code ships.

That work will need to address the larger report volume through verification and prioritization. CodeRabbit says additional issues worth fixing, preservation of existing catches, and manageable reports are the criteria for assessing the model’s contribution. For teams already dealing with too many alerts, the company says report volume alone is not a reason to change models.

Separately, CodeRabbit reports that teams ran over 13,000 Deep Scans across more than 2,700 organizations from October 23 to November 3. Those product-wide scans identified critical and high-severity findings, with authorization bypass, IDOR, and cross-site scripting among the most common. CodeRabbit distinguishes those results from the benchmark evaluation of GPT-6 Cyber’s contribution to its ensemble.