HEADoC: Highly Efficient and Accurate Document Classifier Optimized Using Semantic Distances

Muhammad Usman Khan

doi:10.1007/s13748-025-00411-x

HEADoC: Highly Efficient and Accurate Document Classifier Optimized Using Semantic Distances

2025-10-01

0

PRE

AI

M

Muhammad Usman Khan *

M

Muhammad Adnan Tariq

M

Muhammad Shaoor Siddique

DOI：10.1007/s13748-025-00411-x

Original

Original request for help

Save

Abstract

En 中文

Automated document classification is a vital research domain, utilizing approaches that employ textual features (e.g., plain text analysis), visual features (e.g., feature extraction from document images), or hybrid techniques that combine both modalities to capture both textual and visual features to enhance performance. Our work introduces two hybrid models, HEADoCBASE\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{BASE}$$\end{document} with 27.7 million parameters and HEADoCLARGE\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{LARGE}$$\end{document} with 90.58 million parameters. A key innovation is our deep attention mechanism, a streamlined yet efficient method inspired by conventional attention frameworks, which facilitates the smooth integration of the two modalities. During experimentation, we observed that ArcFace loss, a metric learning approach effective for heterogeneous datasets with distinct intra-class characteristics, performed poorly in our task due to the homogeneity of document classes in standard benchmarks. Both models were evaluated on the RVL-CDIP and Tobacco3482 datasets. On Tobacco3482, HEADoCBASE\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{BASE}$$\end{document} achieved 95.98% accuracy, while HEADoCLARGE\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{LARGE}$$\end{document} reached 96.66%. For RVL-CDIP, accuracies were 92.95% and 93.62%, respectively. HEADoCBASE\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{BASE}$$\end{document} not only surpassed numerous state-of-the-art models but also proved itself to be the most compact architecture in comparison, highlighting efficiency without sacrificing performance. Having the most compact sizes amongst their competitors, our models can be trained faster than its rival architectures.

Keywords:

Document classification

Efficient deep learning

Ensemble

Deep attention

Arcface loss

HEADoC: Highly Efficient and Accurate Document Classifier Optimized Using Semantic Distances

Abstract

Journal

Organization

Cited Papers