Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics Testing

doi:10.1109/TIFS.2023.3289254

Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics Testing

2023-01-01

2

PRE

AI

DOI：10.1109/TIFS.2023.3289254

Original

Original request for help

Save

Abstract

En 中文

As an efficient and multi-platform scripting language, Lua is gaining increasing popularity in the industry. Unfortunately, Lua's unique advantages also catch cybercriminals' attention. A growing number of IoT malware authors switch to Lua for malicious payload development and then distribute malware in bytecode form. To impede malware code analysis, malware authors obfuscate standard Lua bytecode into a customized bytecode specification. Only the attached interpreter can execute that particular bytecode file. Rapid recovery of Lua obfuscated bytecode is essential for a swift response to new malware threats. However, existing generic code deobfuscation approaches cannot keep up with the pace of emerging threats. In this paper, we present a novel reverse engineering technique, called interpreter semantics testing. Given a customized interpreter used to execute obfuscated Lua bytecode, we construct a set of LuaGadgets that can adapt to the customized interpreter. Each LuaGadget contains a carefully chosen opcode sequence to fulfill an observable calculation-it is designed to test one or two particular opcodes at a time. Next, we mutate unknown opcode values to generate a bunch of test cases and run them using the customized interpreter; we can observe the expected result only when the mutation hits the opcode's right value. We perform test case prioritization to cost-effectively recover the semantics of all obfuscated opcodes. Our approach makes no assumptions about the interpreter's structure and is free from analyzing the numerous execution traces of opcode handlers. We have evaluated our tool, LuaHunt, with Lua malware variants and real-world applications. LuaHunt is able to recover the obfuscated bytecode's semantics within 90 seconds for each test case, and all of our deobfuscation results can pass the correctness testing. The encouraging results demonstrate that LuaHunt is a promising tool to lighten the burden of security analysts.

Keywords:

Malware

Codes

Semantics

Testing

Virtualization

Standards

Source coding

Malware analysis

bytecode obfsucation

deobfuscation

Lua

interpreter semantics testing

Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics Testing

Abstract

Journal

Organization