Snorkel AI Tests RIFT on 147 Tasks and Finds Rubrics Often Miss Key Criteria

Why the rubric itself needs checking

Rubric-based evaluation is now a common way to score the outputs of language models and agents, and rubrics also provide reward signals in training. Snorkel argues that in both uses the rubric is the measurement instrument, so the validity of a score depends on how well the rubric represents what the task is meant to assess.

Rubric quality, the post says, is rarely examined directly. It is usually inferred from agreement between automated judges and human raters, or from gains in models trained on rubric-derived rewards. Snorkel notes that both measures confound the rubric with the model that applies it, and neither can detect a rubric that is internally consistent and interpretable yet misaligned with the task.

RIFT (RubrIc Failure mode Taxonomy), described in a paper by Aich et al., 2026, addresses this by evaluating the rubric directly. It does not ask a model to assign a quality label. Instead it computes quantitative signals from the rubric's content, using an approach grounded in measurement and education theory.

How RIFT works and how it was validated

RIFT computes four families of quality signals. A panel of simulated experts votes on whether each criterion is essential. Criteria are rated against the task's objectives to measure coverage. Near-duplicate criteria are detected by similarity. Point values are compared with the panel's view of importance. Each signal is reported relative to other tasks in the same benchmark, and a taxonomy of nine failure modes links each deviation to a named defect. The taxonomy was built from expert critiques of 85 rubrics.

To validate the signals, Snorkel injected each failure mode at known severity into 60 expert-authored rubrics. A multinomial logistic classifier over the signal changes identified the injected mode with 75.0% accuracy, against a chance level of 12.5%, under leave-one-task-out cross-validation. An LLM judge given the same rubrics reached 56.7%.

What the five-domain analysis found

The analysis covered 27 to 30 tasks from each of five benchmarks: finance, medicine, software engineering, professional advice and reasoning. Pooled across all 147 tasks, the share of tasks flagged for each mode was:

• Missing criteria: 50%
• Misaligned criteria: 48%
• Ungrounded verification: 37%
• Redundancy: 35%
• Miscalibrated weighting: 29%
• Rigid: 14%
• Subjective language: 10%
• Non-atomic: 7%

Medicine stood out for missing criteria, at 90% of tasks (27 of 30). Misalignment was highest in the judgment-heavy domains: 73% in medicine, 70% in reasoning and 57% in professional advice, versus 11% in finance. Ungrounded verification showed the opposite pattern, appearing in 74% of finance tasks and 57% of software engineering tasks but in only 3% of reasoning tasks.

Snorkel says the results matched two of its three starting expectations: atomicity and language anchoring were largely sound, and coverage, alignment and grounding failures were more frequent. Non-atomic criteria appeared in 11 of 147 tasks and subjective language in 14 of 147. Subjective language was an exception in the reasoning domain, at 33%, where tasks require weighing competing considerations. The post cites a criterion asking whether a response "weighs the effect of following the therapist's advice" without specifying which effects matter.

Limits and what it means for rubric authors

Snorkel flags two limits. With about 30 tasks per domain, the 95% confidence interval on a rate near 50% spans roughly plus or minus 18 points, so comparisons between domains are indicative, and the pooled estimates are more reliable. Hackability is reported separately in the post. Tasks where a mode appeared only as a minor issue are not counted in the percentages.

The company says the aim is to characterize rubric-writing practice in general and help authors improve, and that the signals and taxonomy give authors a shared vocabulary for discussing rubric failures. For teams that rely on rubrics for evaluation or reward signals, the findings suggest that checking whether a rubric covers the task and aligns with it may matter more than polishing wording or splitting bundled criteria.