- LLM
- EVALUATION
- BENCHMARK
LLM Eval Harness
An evaluation harness that compares an LLM against baseline classifiers on a concrete task: reproducible metrics to tell when the LLM is worth it and when it is not. Code at github.com/delcenjo/llm-eval-harness.
Before using an LLM to classify text it is worth asking whether it earns its place. A handful of keyword rules is free, instant and easy to reason about. This harness answers the question with numbers instead of a hunch: on a labelled dataset, does the LLM beat the baseline, and by how much?
- Baseline0.840 acc
- Macro-F10.853
- ComparisonLLM vs rules
The same contract for everyone
One task (routing support messages to intents), two predictors sharing the same predict(text) interface, and an evaluation that scores them in the same table: accuracy, per-class precision/recall, macro-F1 and a confusion matrix. Direct comparison, no format tricks.
The baseline sets a high bar
The keyword rules alone: 0.840 accuracy and 0.853 macro-F1 on the labelled set. That is the number the LLM has to beat clearly to justify its cost and latency. Sometimes it does; sometimes the gap does not pay the bill.
A tool came out of this
The habit of comparing against a baseline with significance tests ended up becoming evalgate, one of my open-source tools: a CI gate that blocks deployment when the evaluation genuinely degrades, not when it is just noise.
Next projects:
Credit Risk Platform
End-to-end MLOps platform for credit-default scoring: reproducible training with a versioned model registry, a FastAPI inference API with input validation, PSI drift monitoring and a Prometheus and Grafana observability stack, all containerised with CI. Code at github.com/delcenjo/credit-risk-platform.
Transformer from scratch
A GPT-style language model implemented from scratch in PyTorch: multi-head attention, causal masking and residual blocks written by hand, plus a byte-pair tokenizer and an ablation study. Code at github.com/delcenjo/transformer-from-scratch.
Built with
