Detection Engineering: Building Modern Alert Systems to Identify Cyber Attacks

Detection Engineering treats security alerts as code. Just as a software developer writes code to perform a specific function, a Detection Engineer writes logic (using specific query languages like Splunk SPL, KQL, or Sigma) to identify malicious behavior in massive datasets of log telemetry.

To build effective detections, engineers focus on several core concepts:

1. Behavior over Signatures

Traditional antivirus relies on "signatures"—identifying a specific file hash or a known bad IP address. This is a brittle approach; an attacker can change a file hash by modifying a single byte of code. Detection Engineering focuses on "behaviors." Instead of looking for a specific piece of malware, an engineer writes a rule to detect the actions that malware must take to be successful (e.g., an Office application spawning a PowerShell process that connects to the internet). Behaviors are much harder for an attacker to change without fundamentally altering their attack strategy.

2. High-Fidelity Alerting

The primary goal of Detection Engineering is to maximize the signal-to-noise ratio. A "high-fidelity" alert is one that triggers only when there is a very high probability of malicious activity, minimizing false positives. Achieving high fidelity requires deep knowledge of the organization's baseline "normal" activity. An engineer must tune the rule to filter out legitimate administrative scripts or approved software that might mimic malicious behavior.

3. Threat Intelligence Driven

Detections are not built in a vacuum. They must be driven by actionable Threat Intelligence. Engineers continuously analyze reports on new Advanced Persistent Threat (APT) campaigns, ransomware techniques, and newly published vulnerabilities to understand how adversaries are operating today. They then translate this intelligence into specific detection logic tailored for their own environment.

Creating a robust alert is a structured process, typically following a defined lifecycle akin to the Software Development Life Cycle (SDLC).

Phase 1: Research and Hypothesis Generation

The process begins with identifying a specific threat to detect. The engineer might read a threat intelligence report detailing how a specific ransomware group uses the vssadmin.exe tool to delete Volume Shadow Copies (backups) before encrypting files. The engineer forms a hypothesis: "If I monitor command-line execution logs for vssadmin.exe being run with the arguments delete shadows, I will detect this ransomware behavior."

Phase 2: Log Source Identification

To detect the behavior, the engineer needs the right data. They determine if the necessary telemetry is currently being collected. In the vssadmin example, the engineer needs endpoint logs containing command-line execution arguments (such as Windows Event ID 4688 with command-line auditing enabled, or Sysmon Event ID 1). If the data isn't there, the engineer must work with IT to ingest the correct logs into the SIEM.

Phase 3: Rule Development and Logic Creation

Using the SIEM's query language, the engineer writes the detection logic. The rule must be precise. A poorly written rule might just alert every time vssadmin.exe runs, which would trigger false positives if the IT team uses the tool legitimately. The engineer writes the query to look specifically for the malicious argument structure and filters out known, authorized IT admin accounts or specific server IP addresses.

Phase 4: Testing and Validation (Purple Teaming)

A detection is useless if you don't know it works. The engineer must test the rule against real data. They often collaborate with the Red Team (or use breach and attack simulation tools) to deliberately execute the malicious behavior (e.g., running the vssadmin delete shadows command) in a controlled test environment. They verify that the SIEM ingests the log, the rule logic correctly parses it, and the alert fires exactly as expected.

Phase 5: Deployment and Tuning

Once validated, the rule is deployed to the production environment. However, the job is not done. The engineer closely monitors the rule for the first few weeks. If the rule generates unexpected false positives due to a previously unknown legitimate business process, the engineer must iteratively tune the logic, adding exclusions to suppress the noise and maintain the rule's high fidelity.

A foundational element of modern Detection Engineering is the MITRE ATT&CK (Adversary Tactics, Techniques, and Common Knowledge) framework. It is a globally accessible knowledge base of adversary tactics and techniques based on real-world observations.

Detection Engineers use MITRE ATT&CK as a roadmap. The framework categorizes attacks into Tactics (the adversary's technical goal, e.g., "Initial Access" or "Lateral Movement") and Techniques (how they achieve that goal, e.g., "Phishing" or "Pass the Hash").

By mapping their detection rules to specific MITRE ATT&CK techniques, an organization can visualize its defensive coverage. A security team can look at the ATT&CK matrix and clearly see which techniques they have robust detections for, and which techniques represent dangerous blind spots, allowing them to prioritize their future Detection Engineering efforts strategically.

To illustrate the power of behavioral detection, consider these real-world scenarios.

Scenario 1: Detecting "Living off the Land" (LotL) Attackers frequently use "Living off the Land" techniques, utilizing legitimate administrative tools already present on the system (like PowerShell, WMI, or Certutil) to execute their attacks, hoping to blend in with normal traffic. A traditional AV signature won't block PowerShell because it's a legitimate Microsoft tool. A Detection Engineer combats this by writing rules focused on suspicious command-line arguments. For example, they might write a rule to detect any instance of powershell.exe launching with the -EncodedCommand flag (which adversaries use to hide malicious scripts in Base64 format) combined with the -Hidden or -NonInteractive flags. While administrators sometimes use these flags, their combined presence is highly indicative of malicious intent and warrants immediate investigation.

Scenario 2: Detecting Kerberoasting Kerberoasting is an advanced technique used to extract service account credentials from Active Directory without needing administrative privileges. The attacker requests service tickets and then attempts to crack the passwords offline. A Detection Engineer can identify this by monitoring Windows Event Logs for specific anomalies. They write a rule to look for Event ID 4769 (A Kerberos service ticket was requested) where the encryption type is RC4 (often preferred by attackers because it's easier to crack) and where a single user account requests an unusually high volume of service tickets in a very short timeframe. This behavioral pattern clearly flags the automated nature of a Kerberoasting attack.

Building a mature Detection Engineering program requires a commitment to continuous improvement and collaboration.

Implement Detection as Code (DaC)

Treat detection rules like software source code. Store your detection logic in a version control system (like Git). This provides an audit trail of who changed a rule and why, allows for peer review of detection logic before deployment, and enables automated testing through CI/CD pipelines. This ensures that the detection repository remains organized, reproducible, and resilient.

Contextualize Alerts for the SOC

An alert that simply says "Suspicious PowerShell Activity" is not helpful. Detection Engineers must enrich alerts with context. The alert output should clearly state what triggered it, map the activity to a MITRE ATT&CK technique, provide the relevant parsed data (user, IP, process tree), and—crucially—include a "playbook" or set of instructions guiding the SOC analyst on exactly how to investigate and remediate the threat.

Embrace the "Detection Engineering Funnel"

Accept that it is impossible to detect everything perfectly from day one. Build rules in tiers. Start with broad, lower-fidelity rules that gather data for threat hunting purposes (these don't trigger alerts, but populate dashboards). As the engineering team analyzes the data and understands the baseline, they refine the logic, moving it down the funnel until it becomes a high-fidelity, actionable alert sent directly to the SOC for immediate response.