Model-Aware Automatic Benchmark Generation with Self-Error Instructions for Data-Driven Models

Kirill Zakharov

doi:10.3390/make7040148

Return

Model-Aware Automatic Benchmark Generation with Self-Error Instructions for Data-Driven Models

2025-11-18

Kirill Zakharov *

Alexander V. Boukhanovsky

DOI：10.3390/make7040148

Original

Save

View PDF

Abstract

En 中文

The growing number of domain-specific machine learning benchmarks has driven methodological progress, yet real-world deployments require a different evaluation approach. Model-aware synthetic benchmarks, designed to emphasize failure modes of existing models, are proposed to address this need. However, evaluating already well-performing models presents a significant challenge, as the limited number of high-quality data points where they exhibit errors makes it difficult to obtain statistically reliable estimates. To address this gap, we proposed a two-step benchmark construction process: (i) using a genetic algorithm to augment the data points where data-driven models exhibit poor prediction quality; (ii) using a generative model to approximate the distribution of these points. We established a general formulation for such benchmark construction, which can be adapted to non-classical machine learning models. Our experimental study demonstrates that our approach enables the accurate evaluation of data-driven models for both regression and classification problems.

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.