Snorkel AI applies RIFT to 147 benchmark tasks and finds rubric gaps in coverage
Snorkel AI has applied the RIFT rubric-quality framework to 147 tasks from five public rubric-graded benchmarks and found that missing criteria (50% of tasks) and misaligned criteria (48%) were far more common than bundled requirements (7%) or subjective wording (10%).
- Snorkel AI applied RIFT to 147 tasks from five public rubric-graded benchmarks in finance, medicine, software engineering, professional advice and reasoning.
- Missing criteria appeared in 50% of tasks and misaligned criteria in 48%, while non-atomic criteria (7%) and subjective language (10%) were rarest.
- Missing criteria hit 90% of medical tasks; ungrounded verification peaked at 74% in finance.
- In a corruption test, RIFT signals identified the injected failure mode with 75.0% accuracy versus 56.7% for an LLM judge.
- With about 30 tasks per domain, between-domain comparisons are only indicative, according to Snorkel.
Why the rubric itself needs checking
Rubric-based evaluation is a common way to score language models and agents, and rubrics also supply reward signals in training. Snorkel AI argues that in both cases the rubric is the measurement instrument, so the validity of a score depends on how well the rubric represents what the task is meant to assess.
According to the post, rubric quality is rarely examined directly. It is usually inferred from agreement between automated judges and human raters, or from gains in models trained on rubric-derived rewards. Snorkel says both measures confound the rubric with the model that applies it, and neither can detect a rubric that is internally consistent and interpretable but misaligned with the task.
RIFT (RubrIc Failure mode Taxonomy), described by Aich et al., 2026, targets that gap. It evaluates the rubric on two dimensions from measurement theory: reliability, meaning whether grading is reproducible, and content validity, meaning whether the grading supports the intended interpretation of the score.
How RIFT works
The framework has two parts. The first is a set of quality signals computed from the rubric text. A panel of simulated experts judges whether each criterion is essential, criteria are compared with the task's topics and objectives to measure coverage, and point values are checked to see whether the most important criteria carry the most weight.
The second is a taxonomy of nine failure modes, built from expert critiques of 85 rubrics. Four are tied to reliability: subjective language, ungrounded verification, non-atomic criteria. The rest concern content validity: missing criteria, misaligned, rigid, redundant, hackable and miscalibrated weighting.
Snorkel reports a validation test in which each failure mode was injected at known severity into 60 expert-authored rubrics. A multinomial logistic classifier over the signal changes identified the injected mode with 75.0% accuracy (chance 12.5%) under leave-one-task-out cross-validation. An LLM judge given the same rubrics reached 56.7%.
What the five-domain review found
The review covered 27 to 30 tasks from each of five benchmarks: finance, medicine, software engineering, professional advice and reasoning. Hackability is reported separately, so the table covers eight modes. The pooled rates were:
- Missing criteria: 50%
- Misaligned: 48%
- Ungrounded verification: 37%
- Redundant: 35%
- Miscalibrated weighting: 29%
- Rigid: 14%
- Subjective language: 10%
- Non-atomic: 7%
Medicine stood out for missing criteria, at 90% of tasks (27 of 30). Misalignment was highest in the judgment-heavy domains: 73% in medicine, 70% in reasoning and 57% in professional advice, against 11% in finance. Ungrounded verification showed the opposite pattern, reaching 74% in finance and 57% in software engineering but only 3% in reasoning.
Subjective language was the weak spot only in reasoning, at 33%, where tasks require weighing competing considerations. Snorkel cites a criterion asking whether a response "weighs the effect of following the therapist's advice" without saying which effects matter.
The post states two limits. With about 30 tasks per domain, the 95% confidence interval on a rate near 50% spans roughly plus or minus 18 points, so comparisons between domains are indicative and the pooled figures across all 147 tasks are more reliable. Tasks where a mode appeared only as a minor issue are not counted.
What it means for rubric authors
Snorkel reads the low rates of bundling and subjective wording as a sign that rubric writers are getting better at scoring one criterion at a time. It also says rubrics for verifiable or objective tasks appear to be becoming more objective.
The larger problems lie in whether rubrics cover the prompt, grade the right objective and point to checkable sources. Snorkel says the quality signals and taxonomy make these failures measurable and give authors a shared vocabulary for discussing them, with the aim of helping authors improve rubric-writing practice. Full definitions, decision rules and examples are in the RIFT paper.
Snorkel AI: Checking the Rubrics: RIFT Results on Five Public Benchmarks
