.png){kind=icon}
.png){kind=icon}
Return
Complexity-Aware Progressive Data Error Correction with Distilled Language Models and Conformal Reliability Control
.png){kind=icon}
OA
AI
DOI：10.3390/math14101599.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
Reliable tabular data correction is a prerequisite for trustworthy analytics in enterprise information systems. Tabular data in such environments frequently contain formatting errors, semantic conflicts, missing values, and cross-field inconsistencies that degrade downstream analytics and machine learning performance. Rule-based methods efficiently handle structural violations but miss context-dependent errors, whereas large language models (LLMs) offer strong semantic-correction capability at inference costs prohibitive for enterprise-scale deployment. This paper formulates data error correction as a progressive decision process and proposes a complexity-aware framework with three processing stages. The first stage applies deterministic rules for low-complexity structural errors. The second stage employs a task-specialized distilled language model for medium-complexity semantic correction. The third stage performs neural probabilistic-logical reasoning on a factor graph for high-complexity cross-field errors. A learnable routing mechanism assigns each record to the appropriate stage based on a lightweight complexity score. Layer-wise conformal prediction is further introduced to construct calibrated prediction sets with coverage guarantees at each stage, together with a rejection mechanism for low-confidence corrections. The framework is evaluated on one enterprise dataset and two public benchmarks (Hospital and Flights). It improves the record-level complete repair rate by 2.1 to 3.1 percentage points over the strongest baseline (GPT-4o-Direct) and by up to 16.8 points over purely rule-based repair, while reducing average inference latency by approximately 80% relative to direct GPT-4o invocation. Ablation studies confirm the critical role of complexity-aware routing and rule-trigger features, and reliability analyses show that hierarchical conformal calibration maintains tighter coverage than single-level alternatives across varying confidence requirements. These results indicate that complexity-aware progressive routing coupled with hierarchical conformal calibration provides a practical path toward high-throughput, auditable, and reliability-controlled data cleaning suitable for enterprise deployment.
Keywords:
data error correction
complexity-aware routing
knowledge distillation
probabilistic logical reasoning
conformal prediction
reliable data cleaning
Journal
IF:
2.2
Papers:
2.9K
Citations:
3.6W