Process Doppelganging: Advanced Malware Memory Tricks to Evade Antivirus

To understand the genius of Process Doppelganging, one must trace the evolution of malware evasion. Early evasion focused on obfuscation and packing—encrypting the malicious executable on disk so the antivirus scanner couldn't read its true contents until the moment it was executed. However, security vendors quickly adapted by developing behavioral heuristics and in-memory scanning, forcing attackers to find entirely new avenues of attack.

This led to the proliferation of Process Hollowing, a pioneering fileless technique. In Process Hollowing, the malware launches a legitimate, trusted Windows application (like svchost.exe or explorer.exe) in a suspended state. Before the legitimate application can actually run, the malware forcibly unmaps (hollows out) the legitimate code from the process's memory space and injects its own malicious payload into the empty shell. Finally, it resumes the process. To the operating system and basic security tools, it appears that svchost.exe is running normally, but the memory space is entirely controlled by the attacker.

While effective for years, modern security products have learned to detect Process Hollowing. Antivirus engines and EDRs now scrutinize API calls like CreateProcess (in suspended mode), NtUnmapViewOfSection (the hollowing), and WriteProcessMemory (the injection). Because Process Hollowing relies on highly suspicious, noisy API calls that are rarely used by legitimate software, it has become increasingly easy for defenders to flag and terminate. Attackers needed a new technique that achieved the same goal—running malicious code disguised as a legitimate process—but utilized entirely legitimate, unmonitored API calls. This necessity birthed Process Doppelganging.

The entire mechanism of Process Doppelganging relies on the exploitation of a legitimate, albeit obscure, feature of the Windows New Technology File System (NTFS) known as Transactional NTFS (TxF). Microsoft introduced TxF to solve a common software engineering problem: ensuring data integrity during complex file operations.

Imagine a software installer updating multiple critical system files simultaneously. If the power fails halfway through the installation, the system might be left in a corrupted, unbootable state. TxF solves this by applying database-style atomic transactions to the file system. A developer can initiate a TxF transaction, make numerous modifications to various files, and then either "commit" the transaction (saving all changes simultaneously to the disk) or "rollback" the transaction (canceling all changes and reverting the files to their original state).

Crucially, TxF operates in complete isolation. When a file is modified within an uncommitted transaction, those modifications are entirely invisible to other applications, the operating system, and, most importantly, antivirus scanners. If an antivirus scanner attempts to read a file that is currently being modified within a transaction, it will only see the original, unmodified version of the file that exists on the hard drive. The modified, uncommitted version exists only within the memory space of the transaction context. This powerful isolation mechanism, designed for data integrity, provides the perfect blind spot for malware to manipulate files without triggering security alerts.

Process Doppelganging executes a meticulously choreographed sequence of API calls to abuse TxF and launch malware. The attack requires four distinct phases: Transact, Load, Rollback, and Animate.

Phase 1: Transact. The attacker first creates a new NTFS transaction using the CreateTransaction API. Within this transaction, the attacker opens a clean, legitimate Windows executable file that already exists on the disk (for example, notepad.exe). Because this happens within the transaction, the attacker essentially creates an isolated, invisible copy of notepad.exe. The attacker then overwrites this invisible copy with their malicious payload. At this point, the malicious code exists within the transaction, but if the antivirus scans the actual notepad.exe file on the disk, it sees nothing wrong.

Phase 2: Load. This is the most critical and complex step. The attacker must create a memory section for the malicious payload to execute. They use the NtCreateSection API, pointing it at the modified, uncommitted file within the transaction. The Windows Memory Manager, unaware of the transaction's temporary nature, maps the malicious payload into a new memory section, fully preparing it for execution as if it were a legitimate file on disk.

Phase 3: Rollback. The attacker simply rolls back the NTFS transaction using the RollbackTransaction API. This instantly erases the modified notepad.exe from the transaction context. There is absolutely no trace of the malicious file written to the physical hard drive. However, because the memory section was already created in Phase 2, the malicious payload remains perfectly preserved in the system's dynamic RAM, waiting to be executed.

Phase 4: Animate. The attacker must now breathe life into the malicious memory section. They create a new process and a new execution thread pointing to the in-memory payload using obscure, low-level Native APIs like NtCreateProcessEx and NtCreateThreadEx. The operating system executes the thread. To the Task Manager and rudimentary security tools, the process appears to be the legitimate notepad.exe (because the memory section was created from that file's path), but it is executing the malicious payload entirely from RAM.

Process Doppelganging is frequently compared to Process Hollowing, as both achieve the ultimate goal of masquerading malware as a legitimate process. However, Doppelganging is significantly more advanced and significantly harder to detect due to the specific APIs it utilizes.

Process Hollowing is highly noisy. It requires creating a legitimate process, suspending it, actively unmapping its memory (which triggers massive red flags for EDRs), allocating new memory, writing the payload, and resuming the thread. Security vendors have heavily instrumented the NtUnmapViewOfSection and WriteProcessMemory APIs to detect this exact behavior.

Process Doppelganging, conversely, never creates a suspended process and never unmaps existing memory. It creates the malicious memory section directly from the isolated TxF transaction and executes it. The APIs used for Doppelganging—CreateTransaction, NtCreateSection, and RollbackTransaction—are entirely legitimate and frequently used by standard Windows processes (like the Windows Update service). Because the malware relies on these benign APIs, it generates far less suspicious telemetry. Furthermore, because the malicious payload is never committed to disk, file-scanning antivirus engines are completely bypassed. By the time the process is executing in memory, the underlying transaction has already been rolled back, leaving zero forensic evidence on the hard drive.

While Process Doppelganging is theoretically brilliant, its practical implementation is incredibly complex, requiring a profound understanding of undocumented Windows internal structures and low-level Native APIs. Consequently, it is not typically found in mass-distributed commodity malware or standard phishing campaigns. Instead, it is the hallmark of highly sophisticated, Advanced Persistent Threat (APT) groups and state-sponsored actors conducting targeted espionage or high-value ransomware operations.

One of the most notable real-world implementations of Process Doppelganging was observed in the SynAck ransomware. The operators of SynAck utilized this technique to bypass enterprise antivirus solutions, allowing the ransomware to silently encrypt critical files across entire corporate networks before the security teams even realized an intrusion had occurred. Other advanced banking trojans and espionage toolkits have also integrated variations of Doppelganging to maintain stealthy persistence on high-value targets, such as financial institutions and government infrastructure.

The impact of this technique is profound because it undermines the core assumptions of traditional security monitoring. When a security analyst sees svchost.exe or explorer.exe running in the Task Manager, they intuitively trust it. Doppelganging shatters that trust, forcing defenders to realize that the name of a process, its location on disk, and even its digital signature do not guarantee that the code executing in memory is actually legitimate.

Detecting Process Doppelganging is a significant challenge, requiring advanced Endpoint Detection and Response (EDR) capabilities that operate at the deepest levels of the Windows operating system. Traditional, signature-based antivirus is entirely ineffective against this threat.

The primary defensive strategy relies on behavioral analysis and deep API monitoring. Advanced EDR solutions hook into the Windows kernel and monitor the specific sequence of Native APIs utilized by the malware. While the individual APIs (CreateTransaction, RollbackTransaction, NtCreateProcessEx) are legitimate, the specific sequence—creating a transaction, modifying an executable within it, creating a memory section, rolling back the transaction, and then executing the process from that specific memory section—is highly anomalous behavior for standard applications. EDRs can flag this specific behavioral chain as a strong indicator of compromise.

Furthermore, memory forensics plays a crucial role in detection. Security analysts and advanced security tools can perform memory scanning to identify discrepancies between the code currently executing in a process's memory space and the code that actually exists in the corresponding executable file on disk. If notepad.exe is running, but the code in memory does not match the binary of the notepad.exe file on the hard drive (because the in-memory code was created from the rolled-back transaction), it is a definitive sign of memory manipulation. Finally, organizations can mitigate the risk by deploying Application Control policies (like Windows Defender Application Control) to strictly limit which processes are allowed to execute, although sophisticated attackers continuously seek ways to bypass these controls using "living off the land" techniques.