arrow
Return

HYPERSEC: An Extensible Hypervisor-Assisted Framework for Kernel Rootkit Detection

delete2026-01-01
delete0
PRE
AI
L
Lionel Hemmerlé
G
Guillaume Hiet
F
Frédéric Tronel *
P
Pierre Wilke
J
Jean-Christophe Prévotet
DOI:10.1007/978-3-032-08124-7_25delete
deleteOriginal
deleteShare
deleteSave
Abstract

Abstract

En 中文
Modern Endpoint Detection and Reaction (EDR) systems must remain reliable even when attackers acquire high privileges within the monitored operating system. Ideally, such systems should be protected from compromise, while maintaining deep visibility into the internal behavior of the target system. To meet this challenge, we aim to isolate the monitored system in a virtual machine (VM) and move the EDR logic into the hypervisor. We present HyperSec, a domain-specific language that allows the VM to safely delegate detection logic to the hypervisor by transmitting specialized monitoring programs. This design bridges the semantic gap by enabling the hypervisor to understand OS-level structures, while constraining accepted programs to protect the hypervisor against potential vulnerabilities. We evaluated our system against kernel rootkits that hide processes or elevate privileges and showed that HyperSec enables reliable detection of such threats. Our results indicate that the performance overhead remains acceptable, paving the way for broader adoption of hypervisor-based EDR with custom in-VM insight.
Keywords:
Security
Hypervisor
Intrusion Detection Systems
Rootkit

Journal

I
INFORMATION SECURITY, ISC 2025
IF:
0
Papers:
26
Citations:
1

Organization

C
centre national de la recherche scientifique (cnrs)
Scholars:
24.5W
Papers: 18.2W
Citations: 12.7W
I
inria
Scholars:
53
Papers: 27
Citations: 740
Cited Papers

Cited Papers

The Hidden Threat: Analysis of Linux Rootkit Techniques and Limitations of Current Detection Tools
err
IF0
err2024-10-26
err0
PREAI
errJakob Stühn; Jan-Niclas Hilgert; Martin Lambertz
errShare
errSave
SoK: Introspections on Trust and the Semantic Gap
err2014-05-01
err0
PREAI
errBhushan Jain; Mirza Basim Baig; Dongli Zhang; Donald E. Porter; Radu Sion
errShare
errSave
errShare
errSave
Denali
err2002-01-01
err0
PREAI
errAndrew Whitaker; Marianne Shaw; Steven D. Gribble
errShare
errSave
errShare
errSave
Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection
err2011-05-01
err0
PREAI
errBrendan Dolan-Gavitt; Tim Leek; Michael Zhivich; Jonathon Giffin; Wenke Lee
errShare
errSave
Singularity
err2007-04-01
err0
PREAI
errGalen C. Hunt; James R. Larus
errShare
errSave
Trace of the Times: Rootkit Detection through Temporal Anomalies in Kernel Activity
err2025-12-01
err0
PREAI
errLandauer, Max; Alton, Leonhard; Lindorfer, Martina; Skopik, Florian; Wurzenberger, Markus; Hotwagner, Wolfgang
errShare
errSave
researcher View more