Linux Forensics: Digital Evidence Collection from Compromised Servers

Before running a single command on a compromised system, an investigator must understand the "Order of Volatility." Digital evidence exists in various states on a computer. Some evidence is highly volatile and disappears the moment the machine is powered off or rebooted; other evidence is non-volatile and persists on the hard drive.

To prevent destroying critical clues, an investigator must collect evidence starting from the most volatile to the least volatile:

1. CPU Registers and Cache: (Extremely volatile, rarely captured in standard incident response).
2. Routing Tables, ARP Cache, Process Tables, Kernel Statistics: Data held in memory regarding current system operations.
3. System RAM (Memory): Contains running processes, unencrypted passwords, network connections, and loaded malware.
4. Temporary File Systems and Swap Space: Data temporarily written to disk to supplement RAM.
5. Hard Disks (File System and Logs): Non-volatile storage containing files, configuration data, and system logs.
6. Remote Logging and Monitoring Data: Logs shipped to external SIEMs or syslog servers.

Because of this order, the golden rule of live incident response is: Do not reboot the compromised machine. If you reboot, you lose all information about active network connections, running malware processes, and potentially the decryption keys for ransomware residing in RAM.

When approaching a live, compromised Linux server, the investigator must act carefully to minimize altering the state of the machine. The mere act of logging in and running commands changes file timestamps and memory contents. Whenever possible, investigators should use trusted, statically compiled binaries from a clean USB drive (a "forensics toolkit") rather than relying on the server's own potentially compromised binaries (e.g., an attacker might have replaced the ls command to hide their files).

Capturing System Memory (RAM)

Capturing RAM is often the most critical step. Memory forensics can reveal rootkits hiding from the operating system, command-line histories, and active network connections. Tools like LiME (Linux Memory Extractor) or AVML (Acquire Volatile Memory for Linux) are compiled as loadable kernel modules to dump the contents of RAM to a raw file, which can later be analyzed offline using frameworks like Volatility.

Gathering Network and Process Information

If a full memory dump isn't immediately feasible, the investigator must manually collect volatile network and process data using standard tools, redirecting the output to an external drive or secure network location:

• Active Connections: netstat -anp or ss -tupan reveals open ports, established connections, and the specific Process IDs (PIDs) generating the traffic. This helps identify Command and Control (C2) beacons or active data exfiltration.
• Running Processes: ps aux --forest provides a hierarchical view of running processes. Investigators look for anomalous process names, high CPU usage from unknown binaries (like cryptominers), or processes running from unusual directories (e.g., /tmp or /dev/shm).
• Open Files: lsof (List Open Files) is incredibly powerful. Running lsof -p <PID> shows exactly what files, network sockets, and libraries a suspicious process is interacting with.

Once the volatile data is secured, the focus shifts to the hard drive. Depending on the severity of the incident and legal requirements, the server may be powered down to physically duplicate the hard drive.

The standard forensic procedure is to create a bit-for-bit image of the entire storage device using the dd or dcfldd command. dd if=/dev/sda of=/mnt/forensic_drive/server_image.dd bs=4M status=progress

This process captures everything, including deleted files, unallocated space, and file slack. Crucially, the investigator must generate cryptographic hashes (like SHA-256) of the original drive and the resulting image to prove the evidence has not been tampered with—maintaining the Chain of Custody. All subsequent analysis is performed on a copy of this image, never on the original evidence.

With the disk image mounted on a secure forensic workstation, the deep dive begins. The investigator analyzes the file system to reconstruct the attack timeline.

Investigating System Logs

Linux logs, typically found in /var/log, are the breadcrumbs left behind by the operating system and applications.

• /var/log/auth.log (or secure on RHEL/CentOS): The primary target for investigating unauthorized access. It records SSH logins, sudo usage, and failed authentication attempts. A massive spike in failed SSH logins followed by a success indicates a successful brute-force attack.
• /var/log/syslog (or messages): General system activity logs. Useful for spotting unexpected daemon starts or kernel errors caused by unstable exploits or rootkits.
• Web Server Logs (e.g., /var/log/apache2/access.log or /var/log/nginx/access.log): If the server hosted a web application, these logs are critical. Investigators look for SQL injection payloads, Directory Traversal attempts, or the uploading of web shells (e.g., requests to suspicious .php files).

Analyzing User Accounts and Histories

Attackers often create hidden accounts or hijack existing ones to maintain persistence.

• /etc/passwd and /etc/shadow: Reviewing these files helps identify recently added users or users who have had their password requirements inexplicably altered.
• Bash History: Checking the .bash_history file in user home directories (e.g., /home/user/.bash_history or /root/.bash_history) can be a goldmine. While sophisticated attackers will clear their history, mistakes happen. The history file can reveal the exact commands the attacker typed, the tools they downloaded, and their lateral movement attempts.

Hunting for Persistence Mechanisms

To survive reboots, attackers must establish persistence. Investigators systematically hunt for these mechanisms:

• Cron Jobs: Attackers frequently schedule malicious scripts to run periodically. Examining /etc/crontab, the /etc/cron.* directories, and individual user crontabs (/var/spool/cron/crontabs) is essential.
• Systemd Services: Modern Linux systems use systemd. Attackers may create malicious service files (.service) in /etc/systemd/system/ to launch their malware automatically upon boot.
• SSH Keys: Checking the ~/.ssh/authorized_keys file for all users. An attacker will often drop their public SSH key here to guarantee backdoor access without needing a password.

Finding Malware and Anomalous Files

Finding the actual malicious payload requires combing the file system.

• Checking Temporary Directories: Directories like /tmp, /var/tmp, and /dev/shm are world-writable and frequently used by attackers to download and execute initial stage malware or exploit scripts.
• File Timestamps (MAC Times): Linux tracks Modification, Access, and Change (inode) times. Using tools like the Sleuth Kit's fls and mactime, investigators can create a timeline of file activity to see exactly what files were created or modified during the timeframe of the suspected breach.
• Finding SUID/SGID Binaries: The command find / -perm -4000 -type f 2>/dev/null searches for files with the SUID bit set. If an attacker leaves a customized shell binary with SUID root privileges, any standard user executing it instantly gains root access.