pytest-metricguard

Install

pip install git+https://github.com/jmweb-org/pytest-metricguard

What it does

A notebook can report whatever accuracy it wants; nobody reruns it before merging. The version of "the model got worse" that actually matters is the one checked on every pull request, against a number saved from a run everyone agreed was good. pytest-metricguard adds one fixture, metric_guard, that compares a metric against a baseline stored in JSON and fails the test if it dropped more than the tolerance allows.

In action

$ pytest
metric 'roc_auc' regressed beyond tolerance in test_model.py::test_model_quality
  baseline : 0.86
  new      : 0.828
  delta    : -0.03200000000000003 (higher-is-better)
  allowed  : 0.01 (tol=0.01, rel_tol=None)

metric-guard
test                                         metric             baseline        new  verdict
test_model.py::test_model_quality            roc_auc              0.8600     0.8280  worse (FAIL)

1 failed in 0.01s

Features

• —metric_guard.check(name, value, tol=...) inside a normal test, no extra infrastructure.
• —The first run with no baseline records it and warns; it does not fail the test.
• —--update-metric-baselines to accept a new number on purpose, the same idea as snapshot-update.
• —A session summary with baseline, new value and verdict for every metric checked.

Other tools