On Measuring Large Language Models Performance with Inferential Statistics

Jesús M. Fraile-Hernández

doi:10.3390/info16090817

Return

On Measuring Large Language Models Performance with Inferential Statistics

2025-09-20

Jesús M. Fraile-Hernández *

Anselmo Peñas

DOI：10.3390/info16090817

Original

Save

View PDF

Abstract

En 中文

Measuring the reliability of performance evaluations is particularly important when we evaluate non-deterministic models. This is the case of using large language models (LLMs) in classification tasks, where different runs generate different outputs. This fact raises the question about how reliable the evaluation of a solution is. Previous work relies on executing several runs and then taking some kind of average together with confidence intervals. However, confidence intervals themselves may not be reliable if the number of executions is not large enough. Therefore, more effective and robust methods are needed for their estimation. In this work, we propose a methodology that estimates model performance while capturing the intra-run variability by leveraging instance-level predictions across multiple runs, enabling the computation of more reliable confidence intervals when the gold standard is available. Our method also offers greater computational efficiency by reducing the number of full model executions required to estimate performance variability. Compared against existing state-of-the-art evaluation methods, our approach achieves full empirical coverage (100%) of plausible performance outcomes using as few as three runs, whereas traditional methods reach at most 63% coverage, even with eight runs.

Keywords:

performance evaluation

confidence intervals

LLMs

run variability

non-deterministic models

statistical evaluation

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.