Artificial intelligence news.
Straight from the original announcements.

Snorkel AI says gated training lifts Qwen3.8-27B on professional tasks

November 19, 2026, 15:32By the AI News Bites Editorial TeamSource: Snorkel AI
Snorkel AI says gated training lifts Qwen3.8-27B on professional tasks
Image: Snorkel AI

Snorkel AI says it raised Qwen3.8-27B's pass@1 performance on OpenAI's GDPval gold set from 31.9% to 40.7% by combining expert-built training data with binary-gated GRPO. The model was trained on 1,105 professional tasks and about 102K training rollouts.

Key points
  • Qwen3.8-27B reached 40.7% pass@1, up from 31.9%.
  • Training used 1,105 professional tasks and about 102K rollouts.
  • Binary-gated GRPO provided fallback learning signals when all attempts failed.
  • The trained model beat human experts in 44.0% of LLM-judged comparisons.
  • Attempts running out of response budget fell from 17.2% to 7.0%.

Training an agent to produce professional deliverables

Snorkel AI trained Qwen3.8-27B on tasks from its GDPval+ series, which contains thousands of professional assignments built by domain experts. The company evaluated the resulting model on OpenAI's GDPval benchmark, which covers work from experienced professionals across nine sectors of the US economy.

The benchmark tasks end with deliverables such as a financial model, a regulatory filing, or a slide deck created from source documents. Each task includes a human-produced reference deliverable and a detailed rubric describing what a complete result should contain. According to Snorkel AI, a single attempt averaged about 50,000 response tokens and could require numerous tool calls.

For evaluation, an attempt passes when it satisfies at least 90% of the rubric items. Snorkel AI says it uses that threshold because LLM judges can be noisy, and may assess the same deliverable differently on separate occasions.

Why binary-gated GRPO performed better

Snorkel AI attributes the gains to binary-gated GRPO, an approach introduced in an earlier post. Standard GRPO uses a binary pass or fail signal. When all sampled attempts fail, however, the group produces no useful distinction between a weak attempt and one that nearly completes the task.

In the company's runs, GRPO sampled eight rollouts for each task. Under standard GRPO, 79% of generated groups contained no passing rollout, meaning four in five groups contributed nothing to training. Yet the median discarded group had a best rollout with a training score of 78%, and 66% of discarded groups included a rollout scoring at least 70%.

Binary-gated GRPO keeps the pass or fail signal when a group contains at least one pass. Only when every rollout fails does it use a fallback based on a broader training score. That score considers the weighted share of rubric items satisfied and whether key results and conclusions agree with the expert reference solution. Snorkel AI says the method raised the share of generated groups carrying a training signal to 95%, compared with 21% under GRPO.

Results across success rates and expert comparisons

After about 102K training rollouts, the binary-gated model reached 40.7% pass@1 on 185 GDPval tasks, averaged over eight attempts per task, up from 31.9% for the base model. Snorkel AI says standard GRPO with a binary signal, trained on the same data and with a similar number of rollouts, stayed close to the base model.

In LLM-judged head-to-head comparisons, the trained model's deliverables beat those of human experts in 44.0% of attempts, compared with 36.4% for the base model. Counting ties, the trained model matched or beat the expert in 47.6% of attempts.

The company reports that binary-gated GRPO outperformed the base model on rubric pass and expert comparison for both pass@1 and pass@8. It was ahead in all nine GDPval sectors on rubric pass and in eight of nine sectors on expert comparison.

What the findings mean for professional AI agents

The results suggest that training agents on difficult professional work may require more information than a simple success label can provide. When successful attempts are rare, binary-gated GRPO uses partial progress to guide learning without replacing the completion criterion when a successful attempt is available.

Snorkel AI also reports a fall in attempts that ran out of response budget, from 17.2% to 7.0%. Under a 64K-token cap, about half the full budget, the trained model already matched the base model's full-budget pass rate. These findings support the company's claim that the method improves both the ability to finish complex tasks and the quality of completed deliverables.

Related news

Paper2Agent turns research papers into interactive AI agents
ResearchNovember 19, 2026, 16:23

Paper2Agent turns research papers into interactive AI agents

Paper2Agent converts a research paper and its codebase into an interactive AI agent that can run the paper’s methods. The framework packages each agent as an MCP server…

Snorkel AI boosts Qwen3.8-27B on difficult professional tasks
ResearchNovember 19, 2026, 15:32

Snorkel AI boosts Qwen3.8-27B on difficult professional tasks

Snorkel AI says it raised Qwen3.8-27B's pass@1 score on OpenAI's GDPval benchmark from 31.9% to 40.7% by combining expert-built training data with a modified…

Mastra launches Gateway for access to more than 500 AI models
Products and toolsNovember 19, 2026

Mastra launches Gateway for access to more than 500 AI models

Mastra has launched Mastra Gateway, a platform that gives developers access to more than 500 language, voice, video and decision models through a single API key and…