arrow
Return

HyperEvade: Countering Anti-Debugging Techniques and Enhancing Transparency in Nested Virtualization using HyperDbg

delete2026-01-01
delete0
PRE
AI
B
Björn Ruytenberg *
M
Mohammad Sina Karvandi
DOI:10.5381/jot.2026.25.1.a8delete
deleteOriginal
deleteOriginal request for help
deleteShare
deleteSave
Abstract

Abstract

En 中文
Modern malware increasingly employs sophisticated anti-debugging and anti-virtualization techniques to evade analysis, particularly targeting artifacts left by virtualization and nested virtualization environments such as VMware Workstation, Hyper-V, and KVM. HyperDbg, an open-source hypervisor-level debugger, introduces advanced mechanisms to mitigate both its own hyper visor footprints and those of the underlying nested virtualization stack. In this paper, we demonstrate the capabilities of adding a transparency layer on top of the HyperDbg debugger to detect, mitigate, and bypass common and advanced anti-debugging methods leveraged against such environments. We refer to it as the HyperEvade project. Although achieving complete transparency remains infeasible, it significantly raises the bar for malware attempting to detect analysis environments, making evasion substantially more difficult. We further highlight the critical importance of these techniques in practical malware analysis workflows, particularly in scenarios involving snapshot restoration for analyzing and debugging internal malware behavior. By reducing observable artifacts, HyperEvade enhances the reliability of snapshot-based analysis and debugging, allowing researchers to stealthily investigate and understand the inner workings of evasive malware without premature detection or execution of anti-analysis payloads.
Keywords:
Anti-Debugging
Anti-Virtualization
Nested Virtualization
Debugging Malware
Binary Analysis

Journal

J
Journal of Object Technology
IF:
1.4
Papers:
20
Citations:
0

Organization

V
vrije universiteit amsterdam
Scholars:
3.1K
Papers: 1.4K
Citations: 0