HELM measures models on many scores, not just one

The conversation

HELM, short for Holistic Evaluation of Language Models, is an evaluation framework built by Stanford's Center for Research on Foundation Models, first described in a paper posted to arXiv in November 2022 and credited to Percy Liang, Rishi Bommasani, Tony Lee and more than forty co-authors. The paper introduces the project as a response to what its authors call a lack of transparency around language-model capabilities and limitations, and reports testing 30 language models under the framework at the time of writing. As retrieved on 16 September 2026, the project's own GitHub repository describes HELM as “an open source Python framework... for holistic, reproducible and transparent evaluation of foundation models,” and separately notes that “HELM entered maintenance mode on June 1, 2026,” a status distinct from the actively developed project the 2022 paper describes.

What the documents show

The paper's design runs 42 scenarios against 7 metrics at once, named as “accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency,” rather than collapsing performance into one leaderboard number. The authors' stated rationale is that a single aggregate score can hide trade-offs: optimizing a model for one property, the paper argues, can come at the expense of another, and one number obscures which property was traded away. The GitHub documentation, describing the framework as currently distributed, lists benchmark datasets it standardizes, including “MMLU-Pro, GPQA, IFEval, WildBench,” and states it evaluates “models from various providers... through a unified interface,” naming OpenAI, Anthropic and Google models as examples.

The system boundary

HELM is built as a measurement framework, not a model: it runs existing models against fixed scenarios and reports the results, with no role in training or deploying the systems it evaluates. Its documented boundary sits between what the framework measures and what any single number from it can support; the paper's multi-metric structure exists specifically so a reader cannot substitute one scenario's accuracy figure for a general claim about a model's safety, fairness or efficiency. Any claim drawn from HELM should name the specific scenario and metric it comes from, per the framework's own design, rather than an unqualified HELM score.

Where it fails

The tool's stated limitation is coverage and currency: a 2022 design tied to fixed scenarios and metrics can miss capabilities or harms that emerge later, and the repository's maintenance-mode notice, current as of this retrieval, indicates the framework is no longer under active development as the original paper describes. A builder using any point-in-time benchmark should track whether it is still maintained.

• Which specific scenario and metric is the cited HELM figure drawn from, and does that scenario match your deployment's task?
• Was the model version tested the same version you are evaluating for deployment?
• Is the benchmark framework, as of your retrieval date, still actively maintained and updated with new scenarios?

Taken together, the 2022 paper and the framework's current repository describe a deliberately multi-dimensional measurement tool whose authors designed it to resist single-number summaries, now itself a historical snapshot subject to the same currency question it was built to raise about the models it evaluates.