.png){kind=icon}
Return
A component-level evaluation framework for diagnosing LLM errors in optimization modelling
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2026.133981.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Large language models (LLMs) are increasingly used to convert natural language descriptions into mathematical optimization formulations. Current evaluations often treat formulations as a whole, relying on coarse metrics such as solution accuracy or runtime, which obscure structural or numerical errors. In this study, we present a comprehensive, component-level evaluation framework for LLM-generated formulations. Beyond the conventional optimality gap, our framework introduces metrics such as precision and recall for decision variables and constraints, constraint and objective root mean squared error (Cons-RMSE and Obj-RMSE), and efficiency indicators based on token usage and latency. We evaluate GPT-5, LLaMA 3.1 Instruct, and DeepSeek Math across optimization problems of varying complexity under six prompting strategies. Results show that GPT-5 consistently outperforms other models, with Chain-of-Thought, Self-Consistency, and Modular Prompting proving most effective. Analysis indicates that solver performance depends primarily on high Constraint Recall and low Cons-RMSE, which together ensure structural correctness and solution reliability. Constraint precision and decision variable metrics play secondary roles, while concise outputs enhance computational efficiency. These findings highlight three principles for NLP-to-optimization modeling: (i) Complete constraint coverage prevents violations, (ii) minimizing Cons-RMSE ensures solver-level accuracy, and (iii) concise outputs improve computational efficiency. The proposed framework establishes a foundation for fine-grained, diagnostic evaluation of LLMs in optimization modeling.
Keywords:
LLM evaluation
optimization modelling
constraint recall
Cons-RMSE
computational efficiency
Journal
IF:
6.5
Papers:
2.5W
Citations:
6.5W
Organization
Cited Papers
No cited papers available