.png){kind=icon}
Return
A Framework for Assessing LLM Consistency in Knowledge Engineering
.png){kind=icon}
PRE
AI
DOI：10.1177/22104968261479512.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Consistency, that is the degree to which a system, process, or its results produce similar outcomes when repeated under identical or different conditions, is a critical concern in knowledge engineering (KE). This is particularly the case given the increasing reliance on large language models (LLMs) in various tasks. This paper introduces CoLLM, a framework designed to assess whether a system or process produces consistent results in LLM-based KE tasks through three tests: (i) the
LLM Repeatability Test
, which evaluates the level of stochasticity or non-determinism of LLMs in existing studies; (ii) the
LLM Update Impact Test
, which examines the effect that LLM updates may have on results; and (iii) the
LLM Replacement Test
, which explores the effect of using alternative LLMs to perform the same study. Through 59 different experiments taken from five separate, recent studies, and leveraging various LLMs and datasets, we investigate the consistency of the results to empirically validate the reliability of the original findings for each study. Our investigation shows that in the majority of cases (81.4%), a consistent behavior with respect to the original studies can be observed, despite some variability across the individual outputs. Additionally, in some cases, changing the choice of LLM can result in a consistent improvement across different metrics. These results demonstrate the viability of the proposed framework in general to assess the consistency of LLM-based KE tasks.
Journal
IF:
2.9
Papers:
609
Citations:
1.6K
Organization
Cited Papers
.png){kind=icon}
ShareSaveShareSave
ShareSaveShareSaveShareSave
View more