Indirect Syscalls: Advanced Techniques for Evading Modern Endpoint Security

To understand Indirect Syscalls, it is necessary to first understand how user-mode applications communicate with the kernel and how EDRs attempt to monitor this communication.

The Role of System Calls (Syscalls)

Modern operating systems, particularly Windows, operate on a strict privilege model.

• User Mode (Ring 3): This is where normal applications (like a web browser, a word processor, or a piece of malware) run. Applications here have restricted access; they cannot directly read from the hard drive, allocate physical memory, or create new processes.
• Kernel Mode (Ring 0): This is the core of the operating system. The kernel has absolute, unrestricted access to the underlying hardware.

When a user-mode application needs to perform a privileged action (e.g., writing a file), it cannot do it directly. It must ask the kernel to do it on its behalf. The mechanism for this request is the System Call (Syscall).

In Windows, the standard path looks like this:

1. The application calls a high-level API in kernel32.dll (e.g., WriteFile).
2. kernel32.dll calls a lower-level API in ntdll.dll (e.g., NtWriteFile).
3. Inside ntdll.dll, the CPU executes a special assembly instruction (syscall on x64 architectures). This instruction forces the CPU to transition from User Mode to Kernel Mode.
4. The kernel receives the request, performs the action (writes the file), and returns the result back to User Mode.

User-Mode API Hooking (The EDR's Defense)

Because almost all malicious activity (process injection, credential dumping, file encryption) requires system calls, EDRs need to monitor them. The most common and stable way for an EDR to do this is by "hooking" the functions inside ntdll.dll in User Mode.

When an application starts, the EDR injects its own DLL into the application's memory space. The EDR then overwrites the first few instructions of critical functions in ntdll.dll (like NtAllocateVirtualMemory or NtWriteVirtualMemory) with a "JMP" (jump) instruction. Now, when malware tries to call NtWriteVirtualMemory, the execution jumps to the EDR's inspection code first. The EDR analyzes the request. If it looks malicious, the EDR blocks it and kills the process. If it looks benign, the EDR allows the execution to continue to the kernel.

The Evolution: Direct Syscalls

To bypass these EDR hooks, malware authors developed Direct Syscalls. Instead of calling the hooked ntdll.dll, the malware brings its own copy of the syscall assembly instructions (the "stub") embedded within its own code. The malware figures out the correct "System Service Number" (SSN)—a unique identifier for each kernel function—and executes the syscall instruction directly from its own memory space.

This completely bypasses the EDR's hooks in ntdll.dll. The EDR in user mode never sees the request, and the malware successfully communicates with the kernel.

Direct Syscalls were highly effective for a time. However, EDR vendors adapted. They realized that they could monitor the transition from User Mode to Kernel Mode using Kernel Callbacks (Event Tracing for Windows - ETW, or specific kernel drivers).

When the kernel receives a syscall, it can check the Call Stack—the trail of breadcrumbs showing where the syscall instruction originated.

• If the syscall originated from within the legitimate, signed ntdll.dll, it is considered normal behavior.
• If the EDR sees that the syscall originated from an unknown, unsigned region of memory (like the malware's executable space), it instantly flags it as highly suspicious and terminates the process.

This is the exact problem that Indirect Syscalls solve.

How Indirect Syscalls Work

The goal of an Indirect Syscall is to execute the syscall instruction so that it appears, to the kernel and the EDR, as if it originated from the legitimate ntdll.dll, while simultaneously ensuring the execution bypasses the EDR's user-mode hooks.

Here is the step-by-step process of an Indirect Syscall:

1. Preparation (Finding the SSN): Just like with Direct Syscalls, the malware first needs to determine the System Service Number (SSN) for the function it wants to call (e.g., NtAllocateVirtualMemory). It usually does this by dynamically parsing the legitimate ntdll.dll loaded in memory, reading the SSN just before the EDR's hook.
2. Locating the Legitimate syscall Instruction: This is the crucial step. The malware searches the memory space of the legitimate, loaded ntdll.dll to find the exact memory address of an actual syscall instruction (followed by a ret instruction) that belongs to Microsoft.
3. Setting up the Registers: The malware prepares the CPU registers with the necessary arguments for the function it wants to call. Crucially, it moves the correct SSN into the EAX register (this is how the kernel knows which function is being requested).
4. The "Jump" (The Indirect Execution): Instead of executing the syscall instruction from its own memory space (which would get caught by call stack analysis), the malware executes a JMP (jump) instruction, pointing the execution flow directly to the address of the legitimate syscall instruction inside ntdll.dll that it found in Step 2.

Why It Bypasses EDRs

Indirect Syscalls effectively bypass both primary layers of EDR defense:

1. Bypassing User-Mode Hooks: The malware never calls the beginning of the function in ntdll.dll where the EDR placed its hook. It jumps directly to the end of the function, right where the syscall instruction resides, entirely skipping the EDR's inspection logic.
2. Defeating Call Stack Analysis: When the kernel receives the request and checks the Call Stack, it looks at the Return Instruction Pointer (RIP). Because the malware jumped to the syscall instruction inside ntdll.dll, the RIP points to a legitimate address within a Microsoft-signed DLL. The kernel and the EDR's kernel driver look at the origin, see ntdll.dll, assume it is legitimate, and allow the execution.

By spoofing the origin of the system call, Indirect Syscalls represent a highly sophisticated, stealthy method for executing malicious code under the radar of advanced security tools.

Indirect Syscalls are not just theoretical concepts; they are actively used by advanced threat actors and incorporated into modern penetration testing frameworks.

1. Cobalt Strike and BBRF

Cobalt Strike, the premier commercial command-and-control (C2) framework (heavily abused by threat actors), constantly updates its evasion capabilities. Modern versions of Cobalt Strike's "Artifact Kit" and various community-developed malleable C2 profiles utilize Indirect Syscalls to inject payloads into remote processes (Process Injection) without triggering EDR alerts. By using Indirect Syscalls to allocate memory and write the payload, the beacon remains undetected during the critical injection phase.

2. Ransomware Operators

Sophisticated ransomware groups, such as LockBit and Conti (and their successors), frequently employ syscall evasion techniques. Before they begin encrypting files, they must often terminate security processes or delete Volume Shadow Copies. Attempting these actions via standard APIs would immediately trigger the EDR. By employing Indirect Syscalls to interact with the system at a low level, the ransomware can neutralize defenses and complete its encryption routine before the EDR realizes an attack is underway.

Detecting and mitigating Indirect Syscalls is one of the most significant challenges in modern endpoint security. Because the technique exploits the fundamental architecture of the operating system, there is no easy "patch." Defense requires advanced, multi-layered EDR capabilities.

1. Advanced Call Stack Analysis (ETWti)

Simply checking if the syscall originated from ntdll.dll is no longer sufficient. EDRs must utilize advanced kernel telemetry, specifically Event Tracing for Windows - Threat Intelligence (ETWti). ETWti allows the EDR to perform deep call stack analysis from the kernel level. Instead of just looking at the immediate origin of the syscall (which the Indirect Syscall spoofed), ETWti allows the EDR to walk the entire call stack backward. If the EDR traces the stack back from ntdll.dll and discovers that the execution flow originated in an unbacked memory region (memory not associated with a file on disk, typical of injected malware), it can identify the Indirect Syscall and terminate the process.

2. Monitoring for Unbacked Memory

Indirect Syscalls are a means to an end; the malware still has to exist in memory to execute the jump. Often, the malicious code executing the Indirect Syscall resides in "unbacked" or dynamically allocated memory (memory allocated via VirtualAlloc with EXECUTE_READWRITE permissions). EDRs can aggressively scan memory for executable regions that are not backed by a legitimate DLL or executable file on the disk. Finding code executing in these regions is a strong indicator of compromise, regardless of how that code attempts to communicate with the kernel.

3. Behavioral and Heuristic Detection

Since relying solely on API hooks is flawed, EDRs must rely heavily on behavioral heuristics. Even if the malware uses Indirect Syscalls to hide its individual actions, the sequence of those actions might still be anomalous. For example, if a seemingly benign process (like notepad.exe) suddenly opens a handle to lsass.exe (the Local Security Authority Subsystem Service), allocates memory inside it, and attempts to read its memory (a classic credential dumping pattern), the EDR should flag this behavior as highly suspicious, regardless of whether the process used Indirect Syscalls to accomplish it. The overall behavior gives the attacker away.