Benchmark for general-purpose AI chat model

What We Evaluated

We tested systems judged to be of significant public interest, including both top-performing and more accessible examples of both Bare Models and AI Systems. See the FAQ for details on our selection policy.

MLCommons tested systems with the following configurations:

• Bare Models: as a standalone model in the default configuration
• AI Systems: as a whole system possibly configured with options per provider’s instructions

In some cases, a given model is shown both as a Bare Model and as part of an AI System. Please consult the provider’s documentation on the recommended configuration for your use case, and consider the grade for the configuration you employ.

Limitations:

Negative Predictive Power: The tests in this benchmark only have negative predictive power. Performing well on the benchmark does not mean that your model is safe, simply that we have not identified critical safety weaknesses.Significant Variance: There is considerable variance in test outcomes relative to actual behavior, due to selection of prompts from an infinite space of possible prompts and noise from use of automatic evaluation for subjective criteria.

---

How to interpret the results

Systems are graded overall and per-hazard using a 5-point scale of Poor, Fair, Good, Very Good, and Excellent based on the percentage of responses that violate the assessment standard.

In general, grades are relative to the observed safety of two of the “accessible” systems – open weight models with fewer than 15 billion parameters – that perform best on the benchmark, composited to produce a “reference model”.

A grade of “Good” corresponds to a competitive level of safety for a general-purpose chatbot AI system given the present state of the art. Lower and higher grades indicate significantly lower and higher levels of safety.

The following table summarizes the grades: