.png){kind=icon}
.png){kind=icon}
Return
Improving LLM-based event extraction with annotation guidelines
.png){kind=icon}
OA
AI
M
P
DOI：10.3389/frai.2026.1797435.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Event extraction constitutes a foundational task in information extraction; but reliance on laborious and expensive human annotations severely restricts the availability of training datasets. While recent works have explored Large Language Models (LLMs) as example-driven (or zero-shot) annotators; they are substantially outperformed by supervised techniques on structured extraction tasks; such as event detection and argument extraction; possibly on account of underspecified task instructions. In this work; we investigate to which extent LLMs can benefit from dataset-specific; detailed annotation guidelines that more precisely represent the dataset's underlying (human) annotation procedures. To this end; we propose a guideline-based; three-stage LLM annotation framework for event extraction that incorporates detailed event annotation guidelines and supports multiple LLM annotators to improve robustness. Using the comprehensive and well-documented ACE 2005 English Annotation Guidelines for Events as a reference document; we evaluate four LLMs across three guideline-compliant benchmark datasets. Our findings indicate that; depending on model choice; guideline specificity; and the dataset's relative label accuracy; employing detailed guidelines can considerably boost event extraction performance; gaining up to 6.7 F1 points over commonly used bare-minimum instructions; with particularly remarkable improvements for reasoning-based models. Furthermore; we demonstrate that augmenting existing datasets with LLM-generated argument annotations can improve argument extraction performance under soft-matching evaluation. Overall; our experiments emphasize the importance of annotation guidelines (as well as their specificity) for LLM-based annotations; providing valuable insights on leveraging LLMs as guideline-compliant annotators.
Keywords:
prompt engineering
LLM
information extraction
event extraction
synthetic annotation
Journal
F
IF:
4.7
Papers:
2.2K
Citations:
4.4K
Organization
No organization information available
Cited Papers