Syscalls Execution: Direct Kernel Communication Tactics for Bypassing Security Software

To understand how syscalls bypass security software, one must first understand the normal execution flow of a Windows application. The Windows architecture is divided into two distinct privilege levels: User Mode (Ring 3) and Kernel Mode (Ring 0).

User Mode (Ring 3): This is the restricted environment where standard applications run—your web browser, word processor, and, initially, most malware. Applications in User Mode cannot directly access hardware, memory belonging to other processes, or critical system resources. If they want to do anything substantial, they must ask the Kernel for permission.

Kernel Mode (Ring 0): This is the highly privileged core of the operating system. The Kernel has unrestricted access to all hardware, memory, and system resources.

When a User Mode application wants to allocate memory (a common first step for malware injecting a payload), it does not talk directly to the Kernel. Instead, it follows a structured path through the Windows API:

1. The High-Level Call: The application calls a well-documented Windows API function, such as VirtualAllocEx located in kernel32.dll.
2. The Native API: kernel32.dll does not actually perform the allocation. It acts as a wrapper, translating the request and passing it down to the Native API, specifically the undocumented function NtAllocateVirtualMemory located in ntdll.dll.
3. The Syscall Transition: ntdll.dll is the lowest level of User Mode. Its sole purpose is to transition the request from User Mode into Kernel Mode. It does this by setting up the necessary CPU registers (including an identifying System Call Number, or SSN) and executing a specific assembly instruction (syscall on x64 architecture).
4. Kernel Execution: The syscall instruction causes a context switch. The CPU transitions into Ring 0, hands the request to the Kernel (specifically the Executive Subsystem), which verifies the permissions, allocates the memory, and returns the result back up the chain to the application.

Endpoint Detection and Response (EDR) solutions are designed to monitor this exact execution flow to detect anomalous or malicious behavior. Since ntdll.dll is the final gateway before a request enters the opaque Kernel Mode, it is the prime location for EDRs to place their surveillance.

EDRs utilize a technique called "User-Mode API Hooking." When a process starts, the EDR injects its own dynamic-link library (DLL) into that process's memory space. The EDR then modifies the memory of ntdll.dll. Specifically, it overwrites the very first few assembly instructions of critical functions (like NtAllocateVirtualMemory or NtWriteVirtualMemory) with a "JMP" (jump) instruction.

When malware attempts to call the API, the execution flow hits the ntdll.dll function but is immediately hijacked by the JMP instruction. The flow is redirected into the EDR's injected DLL. The EDR inspects the request: "What process is asking for this memory? Is the payload a known malware signature? Is it trying to inject code into a critical system process like lsass.exe?"

If the EDR deems the request malicious, it blocks the execution, kills the process, and generates an alert. If it deems the request benign, it executes the original assembly instructions it overwrote, allows the syscall to transition to the Kernel, and returns the result to the application. From the application's perspective, the interception is invisible.

Direct Syscall execution is an evasion technique designed to completely bypass the EDR's hooks placed in ntdll.dll. The concept is simple: if the EDR is monitoring the gateway, don't use the gateway.

Instead of calling VirtualAllocEx (which calls NtAllocateVirtualMemory in the hooked ntdll.dll), the malware developer hardcodes the exact assembly language instructions necessary to perform the transition into Kernel Mode directly within their own malicious executable.

To execute a direct syscall, the malware must:

1. Determine the System Call Number (SSN): Every function in the Kernel is identified by a specific number (e.g., NtAllocateVirtualMemory might be 0x18). The malware must know this number.
2. Setup the Registers: The malware uses assembly language to manually load the required arguments into the correct CPU registers and places the SSN into the EAX register.
3. Execute the syscall Instruction: The malware executes the syscall assembly instruction directly from its own memory space.

Because the malware executes the syscall instruction itself, the execution flow never touches the hooked functions within ntdll.dll. The EDR, waiting patiently at the ntdll.dll gateway, sees absolutely nothing. The malware successfully communicates directly with the Kernel, allocates memory, injects its payload, and remains completely undetected by user-mode monitoring.

While powerful, direct syscall execution is not without its significant challenges and drawbacks for malware developers.

The "Hell's Gate" Problem: System Call Numbers (SSNs) are not static. Microsoft frequently changes the SSNs for critical functions between different versions of Windows, and even between minor service pack updates. A direct syscall hardcoded with SSN 0x18 for Windows 10 build 19041 will likely crash the system on Windows 11 because 0x18 might now point to an entirely different Kernel function.

To overcome this, advanced malware dynamically resolves SSNs at runtime. Techniques like "Hell's Gate" or "Halo's Gate" involve the malware reading the legitimate ntdll.dll file directly from the hard drive (bypassing the hooked version in memory), parsing its structure, and dynamically extracting the correct SSN for the specific OS version it is currently running on.

The Mark of the Syscall (Indicators of Compromise): EDR vendors are well aware of direct syscall techniques. While they cannot intercept the user-mode call, they can look for the resulting Indicators of Compromise (IOCs).

• RIP (Instruction Pointer) Anomalies: When a legitimate syscall is executed, the return address (where the Kernel returns execution after finishing) should point to an address within the memory space of ntdll.dll. If the Kernel returns execution to an address originating from a random, unbacked region of memory (where the malware resides), it is a massive red flag. Modern EDRs using Kernel-level telemetry (like ETW-Ti) monitor these return addresses and will flag processes executing syscalls from outside ntdll.dll.

To counter the detection of anomalous return addresses, malware developers evolved their tactics to "Indirect Syscalls."

Indirect syscalls attempt to achieve the best of both worlds: bypassing the EDR hooks while maintaining the appearance of legitimate execution flow. Instead of executing the syscall instruction from within its own malicious memory space, the malware manually sets up the CPU registers with the correct arguments and the SSN. However, for the final step, it does not execute syscall.

Instead, the malware searches through the memory space of ntdll.dll to find an existing, legitimate syscall instruction that has not been hooked by the EDR (or it jumps slightly past the EDR's hook). It then executes a JMP instruction to that specific location within ntdll.dll.

The syscall instruction is executed from the legitimate memory space of ntdll.dll. When the Kernel finishes the request and analyzes the return address, it sees that the request originated from ntdll.dll, which appears completely normal. The EDR's user-mode hooks are bypassed, and the Kernel-level telemetry is fooled into seeing a legitimate execution flow.