Appareo: Runtime Reliability Inspection for Deep Learning-Based Image Recognition Systems

Hui Liu

doi:10.1109/tase.2026.3730789

Appareo: Runtime Reliability Inspection for Deep Learning-Based Image Recognition Systems

2026-01-01

0

PRE

AI

DOI：10.1109/tase.2026.3730789

Original

Original request for help

Save

Abstract

En 中文

State-of-the-art (SOTA) deep learning (DL) models may still produce incorrect inferences without signaling uncertainty, limiting their reliability in safety- and quality-critical automation systems. From a software testing perspective, runtime inference reliability assessment for modern convolutional neural networks and Transformer-based image recognition models remains challenging due to the lack of effective test oracles and the limited scalability of existing approaches. This paper proposes Appareo, an unsupervised runtime reliability assessment framework for DL-based image recognition systems. Inspired by metamorphic testing (MT) principles, Appareo constructs semantics-preserving transformed inputs and evaluates inference consistency between original and transformed samples. To strengthen the runtime test oracle, Appareo further incorporates interpretation-consistency verification by comparing critical recognition regions extracted from heatmap-based visual interpretation techniques. Violations of prediction consistency or interpretation consistency are treated as indicators of unreliable inference behavior. We conduct extensive evaluations on large-scale image recognition tasks using convolutional-neural-network-based and Transformer-based models over the ImageNet and Caltech datasets. Experimental results show that Appareo achieves an F1 score of 0.77 for unintended abnormal sample detection and outperforms several SOTA adversarial detectors with an average AUC of 0.80 under diverse adversarial attacks. In addition, we evaluate Appareo in a real-world automated visual inspection scenario under both adversarial perturbations and environmental uncertainties. The results demonstrate that Appareo is an effective and scalable runtime reliability assessment framework for modern DL-based image recognition systems. Note to Practitioners-DL-based image recognition models are widely used in intelligent manufacturing and automated inspection systems, but they may produce unreliable predictions under environmental disturbances or adversarial perturbations. This work proposes an unsupervised runtime reliability assessment framework that verifies inference consistency under semantics-preserving input transformations. The framework jointly analyzes prediction consistency and interpretation consistency to identify potentially unreliable inference behaviors without requiring labeled deployment data or modifications to existing models. Experimental results on large-scale image classification tasks and a real-world automated visual inspection scenario demonstrate that the proposed approach can effectively detect unintended abnormal samples and adversarial perturbations while maintaining practical runtime overhead. The framework provides supplementary reliability signals before inference results are propagated to downstream control modules, supporting intelligent industrial automation and robotic perception systems.

Keywords:

Modeling

Runtime

Signal detection

Inspection

Image recognition

Visualization

Conferences

Forests

Labeling

Testing

Soft testing

reliability assessment

metamorphic testing

image recognition

adversarial detection

Appareo: Runtime Reliability Inspection for Deep Learning-Based Image Recognition Systems

Abstract

Journal

Organization

Cited Papers