Anomaly detection is an approach to real-time threat detection that finds unusual patterns in data that don't match normal behavior, making it easier to spot unusual activities that could be a sign of security threats.
Here a few of the types of anomalies that anomaly detection aims to catch:
Unusual file access: A computer or user (human or non-human) suddenly trying to access files they have never accessed before, especially sensitive files. Imagine a factory floor employee attempting to access payroll data.
Suspicious logins: A large number of login attempts, or logins from an unusual location. For example, that same factory floor employee, who usually works day shifts at your production site, is suddenly logging in from Indonesia in the middle of the night.
Out-of-the-ordinary traffic: Unusual exfiltration attempts or unusual file transfer destination or protocols. For example, that “factory floor employee” (who is in reality an attacker on the other side of the world, who gained access through a compromised password) might be performing massive file transfers to an unknown server.
Think of anomaly detection as being like a quality control inspector on an assembly line: It examines each piece of data, looking for problems or inconsistencies that deviate from the expected norm. If it finds something unexpected, it can pursue automated courses of action and alert your team to give you a heads up.
In this post, we’ll look at where anomaly detection fits into your cybersecurity big picture, some common techniques and use cases, as well as some tips on rolling out anomaly detection without adding to your teams’ workload.
Where does anomaly detection fit in your cybersecurity toolkit?
Anomaly detection tools start with preset norms based on average values for an environment like yours: network traffic, API calls, or logs. Using statistics and analytic algorithms, these tools keep a close eye on system behavior and user actions, making sure nothing deviates from those norms. If it picks up something unusual, the tool triggers an alert.
Anomaly detection doesn’t replace other cybersecurity tools and strategies you may already be using. Instead, it adds important capabilities.
Other security tools may work reactively based on signatures and vulnerability lists, and may rely on a perimeter-based, permission-based model—rather than a behavioral model that adapts to your organization. These tools may also be inflexible or require extensive customization to adapt to your needs, rather than learning naturally on their own what’s normal or unusual for your organization.
Anomaly detection tools, on the other hand, are proactive—they’re constantly on guard. Beyond the initial baseline of normal and anomalous activity for your organization, they fine-tune their results over time using AI and machine learning.
One big advantage of anomaly detection is that it subjects all users and activities, in all locations, to equal scrutiny. Even if a user is authorized and already present within your network, anomaly detection tools will raise a red flag if they are behaving in an unusual way.
And because anomaly detection tools can be taught over time which of these events are nothing to worry about, the number of alerts will decrease over time, easing work for your teams.
What are the biggest benefits of anomaly detection in your cloud-native environment?
Anomaly detection is especially valuable when it comes to cloud-based assets and resources. Because while cloud-native resources can scale and change rapidly, they also often behave in regular and predictable ways because they exist to fulfill a very specific purpose. For example, a Kubernetes pod that's part of a microservice might only access data from a specific database, process it, and send it to another endpoint in the cloud environment. If this pod does anything else, it's a red flag.
Traditional security tools might not pick up anomalous behavior of cloud-native resources because the behavior itself isn’t inherently suspicious. But anomaly detection will quickly flag these events as unusual and alert your teams accordingly.
As cloud adoption grows, anomaly detection is finding a more and more significant place in almost every industry from manufacturing, where it can spot tiny defects early in the process, to finance, where it flags fraud, money laundering, and predicts market risks. In cybersecurity, it can detect and prevent a wide range of cyberattacks.
Intrusion attempts
Anomaly detection picks up unusual access patterns that could indicate an intrusion attempt in progress. For example, if a medical assistant suddenly starts accessing patient records for surgeries they are not involved in, or if they attempt to modify patient billing information, these unusual access patterns could lead the anomaly detection tool to flag an insider threat.
Data exfiltration
Anomaly detection can identify a sudden increase in outbound traffic to a previously unknown IP address. For example, if an advanced persistent threat (APT) group has compromised a financial institution’s system, the anomaly detection tool would flag attempts to exfiltrate sensitive customer data, such as credit card numbers or personally identifiable information (PII).
Malware infections
Anomaly detection can spot signs of malware in your cloud environment before it disrupts business operations. For example, if a manufacturer’s inventory management system begins making unusual encrypted connections, attempting lateral movement to connected systems, and displaying abnormal CPU usage patterns typical of malicious code execution, the anomaly detection tool would flag these potential indicators of compromise before it impacts production.
Which of your team members would benefit most from anomaly detection?
Anomaly detection can be particularly valuable for a few key roles on your teams:
Security Operations Center (SOC) analysts: Detects and prioritizes of critical threats; minimizes alert fatigue
Incident responders: Accelerates investigations and helps pinpoint root causes
Threat hunters: Uncovers hidden threats and streamlines operations tasks
Detection engineers: Expands detection coverage and cuts noise from new detection rules
Anomaly detection also gets all teams working on the same page, creating a common language for more effective communication and collaboration. Anomaly detection tools can help open up silos to build a more holistic security picture that lets you innovate and move faster.
What are anomalies and how do we catch them?
To better understand anomaly detection tools, it helps to zoom in on what an anomaly is in cybersecurity terms. In general, there are two main categories:
Outliers: These are unusual data points that fall far outside of the expected range. An example of this might be a login attempt or network traffic from an unusual geographic location, which could potentially indicate a malicious actor attempting to gain access.
Change in events: These are abrupt shifts in data patterns, such as sudden spikes or drops. For example, a significant number of site visitors making simultaneous requests could potentially indicate a DDoS attack.
To catch these very different types of anomaly, anomaly detection solutions typically employ multiple detection approaches, often in combination. There are many approaches, in a wide variety, but they can be categorized roughly (as we’ll see in the next section).
How does anomaly detection work?
Anomaly detection solutions operate in a number of distinct phases. While all steps are outlined here, many anomaly detection solutions actually work straight out of the box, pre-trained by the vendor so they’re ready to go in your environment with minimal tuning.
1. Ingest and label
Anomaly detection begins by collecting and preparing data from logs, sensors, network telemetry, and other sources. Ideally, data is prepared by cleaning and tagging log data to ensure the highest possible quality for training before the next step of training and testing, but this isn’t always possible.
2. Train and test
All AI models require training. If data is labeled, the anomaly detection solution uses a “supervised” training process, using clean, tagged data to learn the difference between normal and anomalous behavior (here, humans “supervise” the AI through the labeling process).
If data is unlabeled, training takes place through an “unsupervised” process where the anomaly detection solution ingests vast quantities of training data and attempts to determine for itself which data points are normal and which are anomalous. Human team members can then tweak the results to achieve greater accuracy.
3. Monitor and detect
At this point, the model is ready to begin monitoring data streams and attempting to detect deviations from expected patterns in real time. Obviously, there will be a higher number of anomalies detected at first—it’s safer to start with broader definitions of anomalies and narrow these down over time.
4. Alert and report
When an anomaly is detected, the solution quickly initiates automated sequences, shutting down access if needed while alerting security and forensics teams so they can quickly begin investigation and mitigation.
5. Iterate and improve
As the anomaly detection solution is used over time, the models will be regularly refined to address limitations and stay updated with evolving threat intelligence.
How can you overcome the key challenges to anomaly detection?
Anomaly detection, like most security solutions, presents a number of challenges. Here are a few that you need to consider.
Challenge #1: Ethical considerations
There’s a wide range of ethical considerations involved in anomaly detection. Because the solution ingests such a large quantity of information, including user and entity behavior patterns and network activity, you need to comply with any pertinent privacy regulations. There are also legitimate employee privacy concerns when monitoring detailed behavior patterns and network activity. Finally, in any situation where AI is making security and business decisions, you need to establish a high degree of transparency.
Challenge #2: Threat innovation and adaptation
Attackers are constantly coming up with new strategies, and these attacks may not match known threat signatures. Zero-day exploits and novel attack methods can get around or trick systems trained on historical data.
Challenge #3: Drift
One further issue complicating anomaly detection is that, as with any use of AI and ML, we need to combat the problem of drift.
When we talk about “drift,” this actually refers to two phenomena:
Concept drift is caused by a change in business rules. If your systems are set up to trigger warnings for weekend logins, but more and more employees begin working on weekends as an important launch date nears, the model needs to be updated so it won’t flag weekend logins in the same way.
Data drift is caused by shifting patterns of data. If the model was trained on work patterns with traditional productivity software, then as employees migrate to more SaaS-based tools, normal network traffic patterns drift away from what the model was trained on, generating more false positives.
Drift can cause both excessive false positives, as models fall out of sync with reality, and false negatives, because changes that occur gradually over time might not trigger alerts. That’s why anomaly detection tools must differentiate between legitimate evolution in patterns and potential threats.
The solution to the last two challenges is actually the same: All anomaly detection systems need periodic updates to adapt the model to new security realities (e.g., new types of threats) and retrain it with the most current data.
Wiz Defend
Making anomaly detection part of your teams’ toolkit doesn’t have to pile on stress or add to their alert fatigue.
Wiz Defend uses anomaly detection to identify unusual behavior in your cloud environment by comparing current cloud events with historical data and alerts you when anything unusual is detected.
This could include unusual process execution patterns, suspicious use of authorized credentials to access unexpected resources, and abnormal communication between systems that typically don't interact.
We combine this anomaly detection with a rules-based approach to identify malicious activity, so alerts are explainable and actionable for your team.
Trusted by over 45% of Fortune 100 companies, Wiz is your cloud security headquarters, giving you all the benefits of anomaly detection alongside all your other cloud security essentials, like CSPM, CWPP, CIEM, and more.
With this consolidated approach to cloud security, Wiz also gives you enhanced context to detect and respond to threats faster.
And with this enhanced context for threat detection and response, Wiz Defend makes it simple to identify unusual behavior in your cloud environment.
With Wiz, you get the industry’s most comprehensive cloud security—proactively reducing risk with preventive controls while providing real-time threat monitoring and response capabilities.
Get a live demo of Wiz Defend today to see its full capabilities in action.
Don't let malicious code compromise your cloud
Learn why CISOs at the fastest growing companies trust Wiz to protect their cloud environments.
