Direct Syscalls: Bypassing Antivirus Systems to Communicate with the Operating System

To understand Direct Syscalls, we must first understand how a modern operating system like Windows separates privileges. Windows operates in two distinct modes:

1. User Mode: This is where normal applications run (e.g., your web browser, Microsoft Word, and most malware). Code running in User Mode is heavily restricted. It cannot directly interact with hardware, allocate memory arbitrarily, or access files belonging to other users.
2. Kernel Mode: This is the core of the operating system. Code running here has absolute, unrestricted access to the computer's hardware, memory, and file systems.

When a User Mode application needs to do something privileged—like read a file from the hard drive or allocate memory—it cannot do it directly. It must ask the Kernel to do it on its behalf.

The Windows API (WinAPI) and NTAPI

This request process follows a specific chain of command.

1. The application calls a documented function from the Windows API (WinAPI), typically located in a DLL file like kernel32.dll. (For example, VirtualAlloc to allocate memory).
2. The WinAPI function then passes the request down to the Native API (NTAPI), located in ntdll.dll. (For example, NtAllocateVirtualMemory).
3. The NTAPI function is the final stop in User Mode. It executes a special assembly instruction (specifically, the syscall instruction on 64-bit systems).
4. This syscall instruction is the bridge. It causes the processor to transition from User Mode into Kernel Mode, handing the request over to the actual Windows Kernel to execute the privileged action.

EDR solutions need to know when an application is requesting a malicious action. To do this, they employ a technique called "API Hooking," primarily targeting the Native API (ntdll.dll).

When Windows boots up and an application launches, the EDR injects its own code into the application's memory space. The EDR locates the NTAPI functions inside ntdll.dll (like NtAllocateVirtualMemory or NtWriteVirtualMemory) and subtly alters the first few bytes of those functions.

The EDR replaces these initial bytes with a JMP (jump) instruction that points to the EDR's own inspection engine. Now, when a program (or malware) tries to execute the function:

1. The program calls the NTAPI function in ntdll.dll.
2. Because of the hook, execution instantly jumps to the EDR's inspection engine.
3. The EDR analyzes the request. It asks: "Is this program trying to inject code into a critical process like lsass.exe?"
4. If the request is malicious, the EDR blocks it and raises an alert. If it is benign, the EDR allows the execution to jump back to ntdll.dll and proceed to the kernel via the syscall.

API Hooking relies entirely on the assumption that the application will politely ask ntdll.dll to handle the transition to the Kernel. Direct Syscalls completely break this assumption.

If an attacker knows that the EDR has hooked ntdll.dll in memory, they simply bypass ntdll.dll altogether.

A Direct Syscall involves the malware executing the syscall assembly instruction directly from its own code, rather than calling the function inside the monitored ntdll.dll.

The Mechanics of the Attack

To execute a Direct Syscall, the malware needs two things:

1. The System Service Number (SSN): Every function in the kernel has a unique ID number (the SSN). For example, NtAllocateVirtualMemory might be SSN 0x18. The Kernel needs this number to know which function you are requesting when you execute the syscall instruction.
2. The Assembly Stub: A small block of assembly code that sets up the required CPU registers with the SSN and the function arguments, and then executes the syscall instruction.

Advanced malware authors will statically compile these assembly stubs directly into their malicious executable. When the malware wants to allocate memory, it doesn't call kernel32.dll or ntdll.dll. It simply pushes the correct SSN into the CPU's EAX register and executes its own syscall instruction.

The CPU transitions to Kernel Mode and executes the request. Because the malware completely bypassed the hooked ntdll.dll in User Mode, the EDR's inspection engine is never triggered. The EDR is completely blind to the malicious activity.

As EDRs became aware of Direct Syscalls, they adapted. Some EDRs began monitoring for syscall instructions originating from memory regions outside of ntdll.dll. If an EDR sees a syscall happening from an unknown executable's memory space, it flags it as highly suspicious.

This led to the development of Indirect Syscalls. Instead of executing the syscall instruction within its own memory, the malware prepares the registers with the correct SSN and arguments, but then it jumps into the legitimate ntdll.dll memory space, specifically landing exactly on the syscall instruction located just after the EDR's hook. This makes the syscall appear to originate from the legitimate ntdll.dll, further confusing the EDR.

Furthermore, System Service Numbers (SSNs) change between different versions and build numbers of Windows. A hardcoded SSN that works on Windows 10 Build 19041 might crash on Build 19042. To solve this, advanced malware uses dynamic SSN resolution techniques like Hell's Gate or Halo's Gate. These techniques allow the malware to parse the unhooked version of ntdll.dll directly from the hard drive (bypassing the hooked version in memory) to dynamically discover the correct SSNs at runtime, ensuring the malware executes flawlessly across any Windows version.

Detecting and mitigating Direct Syscalls is one of the most challenging tasks in modern endpoint security, as the attacker is essentially operating at the same privilege level as the security software's user-mode components.

1. Kernel-Level Telemetry (ETW-TI)

Since User Mode hooking (ntdll.dll) can be bypassed, modern EDR solutions must rely more heavily on Kernel-level telemetry. Microsoft introduced Event Tracing for Windows - Threat Intelligence (ETW-TI), a kernel-level logging mechanism accessible only to Early Launch Anti-Malware (ELAM) certified security vendors. ETW-TI provides a stream of events directly from the kernel, detailing process creation, memory allocation, and thread injection, completely independent of any User Mode hooks. Even if an attacker uses Direct Syscalls to allocate memory, the kernel will still report the action via ETW-TI, allowing the EDR to detect the malicious behavior.

2. Call Stack Analysis

When a legitimate application executes a syscall via ntdll.dll, the "Call Stack" (the record of which functions called which functions to get to this point) looks normal, originating from a known API like kernel32.dll. When malware uses a Direct Syscall, the Call Stack often looks highly anomalous, abruptly starting from the malware's own unbacked memory region. Advanced EDRs analyze these call stacks during critical events (like thread injection) to identify discrepancies indicating a syscall bypass.

3. Memory Scanning and YARA Rules

While the behavior might be hidden, the malware executable itself must still reside in memory. Security teams should deploy regular memory scanning using YARA rules specifically designed to detect the distinct assembly stubs associated with known Direct Syscall implementation frameworks (like SysWhispers or Hell's Gate).