MFS-DoH: efficient malicious DNS-over-HTTPS traffic detection using multi-stage feature selection

Moulya, D. M.

doi:10.1007/s11416-026-00626-z

Return

MFS-DoH: efficient malicious DNS-over-HTTPS traffic detection using multi-stage feature selection

2026-05-11

PRE

Moulya, D. M. *

Saumya Hegde

DOI：10.1007/s11416-026-00626-z

Original

Original request for help

Save

Abstract

En 中文

DNS over HTTPS (DoH) enhances privacy by encrypting DNS queries, but it also presents challenges for detecting malicious traffic, such as tunneling and command-and-control (C2) channels. This paper presents MFS-DoH, a detection pipeline for malicious DoH traffic that integrates a multi-stage feature selection (MFS) process with a hierarchical classification architecture. The MFS pipeline combines LightGBM-based importance ranking, ANOVA-Lasso regularization, and mutual information ranking to reduce 29 flow-level features to a compact set of five discriminative attributes. The hierarchical detection approach consists of two layers: Layer 1 separates DoH traffic from non-DoH traffic, while Layer 2 distinguishes between benign and malicious DoH traffic. We evaluate the proposed method on three public benchmark datasets covering tunneling-based and DGA-driven malware traffic. Within the combined dataset, the Voting Ensemble with the final five-feature set achieves 99.47% accuracy in Layer 1 and 99.95% accuracy in Layer 2, with statistically comparable performance to the full 29-feature baseline while reducing feature dimensionality by 83%. A targeted padding attack produces evasion rates of 15-22% across classifiers. Feature selection stability analysis confirms that the selected subset is fully reproducible across estimator sizes and random seeds. These results demonstrate that operationally viable detection is achievable with minimal features under in-distribution conditions, while highlighting open challenges in cross-domain generalisation and adversarial robustness.

Keywords:

Domain name system

DNS over HTTPS

Malicious traffic

Malware

Machine learning