API Unhooking: How Malware Evades EDR System Surveillance

To understand how malware evades surveillance, one must first comprehend how EDRs monitor activity. When a program runs on a Windows operating system, it frequently needs to interact with the OS to perform tasks like reading a file, allocating memory, or creating a network connection. It does this by calling Windows Application Programming Interfaces (APIs), which are typically exported by core system libraries like kernel32.dll or ntdll.dll.

An EDR system needs to intercept these API calls to inspect them before they are executed by the OS kernel. It achieves this through API Hooking, specifically User-Mode API Hooking.

The Mechanics of Inline Hooking

The most common method used by EDRs is Inline Hooking (also known as detouring). When an EDR agent is installed on a system, it injects its own dynamic-link library (DLL) into every newly created process. This injected EDR DLL then modifies the memory space of the target process.

Specifically, the EDR locates critical API functions within memory (e.g., NtAllocateVirtualMemory or NtCreateThreadEx inside ntdll.dll). It overwrites the very first few instructions of these functions with a JMP (jump) instruction. This jump redirects the execution flow from the original Windows API directly into the EDR's own monitoring code.

When a process (benign or malicious) attempts to call the hooked API, the following sequence occurs:

1. The process calls the API.
2. The inserted JMP instruction immediately diverts execution to the EDR's DLL.
3. The EDR inspects the parameters of the API call (e.g., determining which file is being accessed or what memory is being allocated).
4. If the EDR determines the behavior is malicious, it blocks the action and alerts the security team.
5. If the behavior is benign, the EDR executes the original, overwritten instructions and returns control to the intended OS function, allowing it to complete.

Through this mechanism, the EDR acts as an omnipresent proxy, scrutinizing every critical action a program attempts to take.

If API Hooking is the EDR's method of surveillance, API Unhooking is the malware's method of cutting the camera wires. API Unhooking involves the malware actively detecting the presence of these EDR hooks in memory and removing them, restoring the original API functions to their pristine, unmonitored state. Once the hooks are removed, the malware can call the APIs directly; the OS executes the actions, and the EDR remains completely unaware because the redirecting JMP instructions are gone.

How Malware Detects Hooks

Before a sophisticated malware strain unhooks an API, it often checks if a hook is actually present. It does this by examining the first few bytes of critical API functions in memory.

In a pristine Windows environment, system APIs in ntdll.dll typically start with specific byte sequences, often involving a MOV instruction followed by a SYSCALL instruction (sometimes referred to as the syscall stub). If the malware inspects the start of an API and instead finds a JMP instruction (opcode E9 in x86/x64 architecture), it is a clear indicator that an EDR has hooked that function.

Malware authors employ several techniques to remove EDR hooks, ranging from simple memory patching to complex module reloading.

1. Direct Memory Patching (Restoring the Stub)

The most direct approach involves the malware reading the original bytes of the hooked function and writing them back into memory, overwriting the EDR's JMP instruction.

But where does the malware get the original bytes? Sometimes, malware authors hardcode the original bytes for specific Windows versions directly into their payload. However, this is fragile because Windows updates frequently change the exact byte sequences of APIs.

A more robust method involves the malware locating the EDR's trampoline. When an EDR hooks a function, it saves the original overwritten bytes in a separate memory location (the trampoline) so it can execute them later if the action is deemed benign. Advanced malware can search memory to find this trampoline, extract the original bytes, and write them back over the hook using APIs like WriteProcessMemory.

2. Reloading a Fresh Copy of NTDLL

The most reliable and widely used technique for API Unhooking in modern malware involves loading a clean, unhooked copy of ntdll.dll directly from the hard drive, bypassing the hooked version residing in the process's memory.

When a process starts, the Windows loader automatically maps ntdll.dll into memory, and the EDR immediately hooks it. However, the original ntdll.dll file sitting on the disk (C:\Windows\System32\ntdll.dll) is clean and unhooked.

The malware executes the following steps:

1. It opens the clean ntdll.dll file from the disk.
2. It maps the contents of this clean file into a new section of its own memory space.
3. It identifies the text section (.text) of the clean DLL, which contains the executable code.
4. It copies the clean .text section over the hooked .text section of the already loaded ntdll.dll in memory.

By doing this, the malware effectively overwrites all the EDR's JMP instructions simultaneously, restoring every API to its original, unmonitored state. The EDR is completely blinded, allowing the malware to execute its malicious payload—such as process hollowing or shellcode injection—without triggering any alerts.

3. Direct System Calls (Bypassing the API Entirely)

While not strictly "unhooking," utilizing Direct System Calls is a related evasion technique that achieves the same goal: bypassing EDR hooks in user mode.

Instead of calling the hooked API function in ntdll.dll (which would trigger the EDR), the malware implements the assembly instructions necessary to transition from user mode to kernel mode directly. By executing the SYSCALL instruction itself, the malware bypasses the hooked DLL entirely, communicating directly with the operating system kernel. Because the EDR's hooks are placed in user-mode memory (ntdll.dll), they are entirely circumvented.

The cybersecurity landscape is a continuous arms race. As malware adopts API Unhooking and Direct Syscalls, EDR vendors update their platforms to detect and mitigate these evasion techniques.

Kernel-Mode Telemetry and Callbacks

Because user-mode hooks are inherently vulnerable to tampering, modern EDRs are increasingly relying on kernel-mode telemetry. EDRs install a kernel driver that registers for various system callbacks provided by the Windows operating system, such as ObRegisterCallbacks for process/thread creation and CmRegisterCallback for registry operations.

Since the EDR is monitoring activity from within the kernel, it doesn't matter if the malware unhooks the user-mode APIs or uses Direct Syscalls; the kernel will still notify the EDR driver when the action occurs. Bypassing kernel callbacks requires the malware to load a malicious driver of its own (Bring Your Own Vulnerable Driver - BYOVD), which is significantly more difficult due to Windows Driver Signature Enforcement (DSE).

Call Stack Analysis

To combat Direct System Calls, advanced EDRs analyze the call stack when a critical kernel transition occurs. Normally, a system call originates from ntdll.dll. If an EDR observes a system call originating directly from a region of memory associated with the application's executable (rather than a legitimate system DLL), it strongly indicates the use of Direct Syscalls by malware, and the EDR will flag or block the process.

Detecting Unhooking Attempts

EDRs also actively monitor for attempts to unhook their libraries. They may set memory protection flags to prevent unauthorized modification of the .text section of ntdll.dll. If malware attempts to overwrite the hooks by changing the memory protections (e.g., using VirtualProtect), the EDR intercepts this action and terminates the malicious process. Furthermore, some EDRs periodically scan the memory space of running processes to verify the integrity of their hooks; if a hook is found missing, the process is considered compromised.