The vast majority of companies relying on cloud infrastructure are at least partially using environments hosted by one of the big three Cloud Service Providers. This consolidation around a small number of well-known vendors results in most environments sharing many architectural and security characteristics, allowing attackers to use the same automated TTPs in a multitude of attacks. On paper, this sounds like the perfect case for using threat intelligence and simple out-of-the-box alerting to detect and prevent attacks on cloud environments. If environments and attacks are so similar, it’s often intuitive to assume that detections should be similar as well. This theoretical reality becomes dramatically more complicated once we acknowledge the great challenge of creating effective detections in the cloud - noise.
Seemingly endless amounts of cloud threat intelligence data combined with increasingly high volumes of automated cloud attacks and generic alerts have created a situation in which most organizations can’t effectively handle their detection feeds. Instead of enabling better detections, these feeds often lead to alert fatigue and hinder the identification of true malicious activity.
To tackle this problem and take control of cloud detection, we suggest following a methodology dubbed the “Zero Noise” approach. While initially challenging to execute, taking an attacker’s perspective to create tailored baselines, implementing continuous feedback loops for every detection and following “no alert left behind” mentality, enables us to stop looking for needles in haystacks and focus only on high fidelity attacker TTPs.
A Real-World Example
To illustrate the challenge, let’s start by examining a real-world case study we first encountered during incident response efforts. The targeted environment belonged to a financial services company offering cash deposit and withdrawal in tens of thousands of points of sale across multiple countries. Attackers targeting the company repeatedly found ways into their environment, faked transactions from points of sale to transaction management servers and made it out with considerable sums of money before being blocked.
Despite suffering multiple successful attacks, the organization was struggling to prevent recurrences across their large global network, relying on thousands of legacy systems which could take years to replace. Their attempts to detect new compromises in time were continuously running up against the same problem – alerts were either too specific to detect variations in attacker activity, or too wide and noisy to be handled by the SOC.
The Zero Noise Approach
While achieving actual “zero noise” is obviously impossible in large dynamic environments, this methodological approach is designed to focus SOC attention on the most important alerts while continuously enhancing detections to reduce noise – getting as close as possible to zero. The approach is built around three key guidelines: taking an attacker’s perspective to alerts, implementing continuous detection feedback loops, and most importantly - adopting a “no alert left behind mentality”.
1.Creating alerts with an attacker’s perspective.
While some generic out-of-the-box alerts can be extremely useful, they often fall short of detecting specific sophisticated threats to cloud environments, creating lots of noise along the way. To augment and increase the fidelity of these alerts, tailored detections should be created based on environment baselines and specific known threats to critical systems and crown jewels. This may sound obvious, but to be implemented effectively organizations must take the time to map out these assets and perform continuous red teaming to identify visibility blind spots. This priority and red teaming-based approach creates tailored accurate alerts which result in much lower amounts of noise.
2. Implementing detection feedback loops.
One of the main issues leading SOC teams to drown in noise, is the existence of large amounts of unchecked detections. Striving towards “zero noise” requires establishing clear procedures for reviewing and analyzing the relative effectiveness of every detection in the environment. Every rule or detection logic should be periodically analyzed to determine how many times it triggered alerts, how many of these were false positives, and how much time the team had to spend triaging them before reaching a conclusion. Too many organizations leave unnecessarily noisy or completely inaccurate alerts in place due to a lack of these feedback loops. Detections determined to be too costly or irrelevant – must be enhanced or removed altogether.
3. No alert left behind.
The most crucial, and challenging, element of the Zero Noise approach is adopting this mentality. Whenever an alert is triggered and triaged, the SOC must either determine that it was a true positive indicating malicious activity (in which case the alert has done its job and wider incident response actions should be taken) or get to the bottom of the false positive and enhance detections accordingly.
It is often tempting to immediately set aside alerts deemed to be false positives, as potentially malicious other alerts presumably await in the cue. However, this setting aside perpetuates the causes of false positives and creates more noise, alert fatigue, and rash “false positive” judgments in the future. Instead, SOC members must be given the mandate to triage and understand every false positive, resulting (in most cases) in one of three outcomes:
Removing the detection. Some detections will simply prove too noisy to be useful, and despite theoretically being relevant to detecting true malicious activity, they may not be worth the noise and fatigue they cause. These detections should simply be removed.
Improving detection logic. In many cases, simple tweaks to detection logic can go a long go in reducing noise. These may include specific exclusions of known legitimate activities, modification of thresholds, or enhancements of search criteria. When the “no alert left behind” mentality is widely adopted, enhancements become less and less frequent while improving the overall accuracy of all detections.
Changing Organizational practices. Sometimes detections become noisy due to unnecessary IT or dev-ops practices which look a lot like malicious activity. When fully excluding these non-malicious activities from detection rules is not possible, a change of internal practices may be in order. Consolidating administrative tools, using pre-approved accounts or jump servers, and adhering to clear security guidelines in IT and dev-ops operations, can go a long way in reducing alert noise.
Real-World Implementation
Returning to our real-world example, implementing the Zero Noise approach enabled the victim organization to dramatically reduce noise and detect further compromise attempts as soon as they started. In this case, noise was successfully reduced by implementing four key changes:
Adopting an attacker’s perspective, the organization realized that while attackers successfully infiltrated their environment in many different places, attacks always eventually focused on the financial transaction management servers. Collaborating with the financial fraud team – dedicated alerts were established to detect and prevent attacker-specific irregular activities at the server, including inconsistencies in transaction amount and timestamps. These tailored high-priority detections revealed and blocked subsequent attempts at malicious transactions without generating noise and crucially, without revealing their own existence to attackers.
A large group of detections on DMZ servers running the company website were determined too noisy (mainly due to internet scanning false positives) and fully removed from the environment. This simple step saved hours of daily SOC work.
Detections designed to detect irregular access to load balancers from points of sale were significantly enhanced after analysis revealed specific activities responsible for over 90% of their false positives. These activities were specifically excluded from detection rules and incorporated into automated triage playbooks.
The common Windows administrative tool PsExec was repeatedly used by attackers to move laterally in the environment but was unfortunately also being regularly used by IT personnel to manage Windows machines. After conversations with IT leaders revealed that multiple other remote administration tools were already used in the environment, PsExec was banned for internal use. This eliminated noise and created a powerful IOC to detect the attackers targeting this organization.
Conclusion
While the vast amounts of cloud data present significant challenges to reducing alert noise, following the Zero Noise approach enables defenders to regain control over their unruly detection feeds. Prioritizing tailored detections based on an attacker’s perspective, implementing continuous detection feedback loops, and following a no alert left behind mentality are key to reducing noise and freeing the SOC to detect and respond when it matters most.
I further explored this topic at the SANS CTI summit.
Cloud Detection & Response for Dummies
SecOps teams need a comprehensive approach to CDR to cut through the noise generated by cloud environments and detect, investigate, and contain emerging threats in real time – before they impact the business. Here are the essentials to build out your Cloud Detection strategy.
Download the guide
