Independent eval research. Open methodology, published datasets, and the tools to run your own studies.
Original, open-source studies
Model comparisons, agent setups, cost, and modalities, measured on real tasks. Read the methodology and results, or rerun a study on your own data.
Each enforcement variant was scored 2 ways. First by whether they adhered to that behavior, and second by the number of tests passed. We found that the agent could have leaky behavior that wasn't surfaced by output-only scoring.
30 tasksdeterministic detector
test_passed on the same 30 tasks
A coding agent rebuilt 27 complex designs through each MCP server, three trials each. Visual similarity scores CLIP embedding distance to the reference screenshot, and faithfulness comes from a claude-sonnet-5 judge.
0–1 CLIP score0.93 practical ceiling
0–1complex designs
225 procedurally generated tasks span arithmetic, symbolic rules, and data transforms, graded by exact match in code with no partial credit. Each model runs every task three times.
225 code-graded taskshigher is better
Run evals like us
Each skill packages best practices from leading AI teams and researchers into a plain SKILL.md file your coding agent can read. Use them to get started running evals on your data.
Point your agent at a skill
Ask it to do what the skill describes, like validating a scorer against human labels or finding hidden failure modes.
- 1.Name the reference tier before computing anything.
- 2.Verify alignment: scorer outputs and reference labels must line up at the item and criterion level.
- 3.Report agreement (κ or α) with uncertainty, not raw accuracy.
- 4.Lead with the most decision-relevant false acceptance before any aggregate, and enumerate the dangerous cells case by case.
- 5.Break errors down by class and severity.
- 6.Test sensitivity and shortcuts: inject known regressions and improvements and confirm the scorer moves; probe whether length, confidence, or polish raise the score independent of quality; probe whether text addressed to the judge moves it; slice agreement by subgroup.