Evaluation

evalgate

Decide whether a metric change is a real regression or noise.

0 downloads/month on PyPI

Install

pip install evalgate-cli

If you use uv, also as an isolated CLI:

uv tool install evalgate-cli

What it does

An eval dropping from 90.0% to 89.4% on 1,000 examples looks like a regression, but at that sample size it is noise. evalgate runs the right statistical test and fails only when the candidate is significantly worse.

In action

A real recorded session of the CLI running.

Read it as text

verdict     worse, but within noise
difference  -0.0300
p-value     0.1900
alpha       0.05

$ evalgate proportions --baseline-score 0.86 --baseline-n 500 --candidate-score 0.83 --candidate-n 500
verdict     worse, but within noise
difference  -0.0300
p-value     0.1900
alpha       0.05

Features

—Two-proportion test over aggregate accuracies.
—McNemar's test for paired per-example results.
—Verdict: improvement, unchanged, noise or regression.
—CI gate with a configurable alpha.

View the code on GitHub →

Other tools

dsdiff

A git-style diff between two datasets, with distribution drift.

splitcheck

Detect rows leaking between train, validation and test splits.

pii-sweep

Scan datasets for PII before sharing them.