What are the best tools to secure AI? If you search for the answer online, you’ll notice a sizable knowledge gap. While plenty of information exists about SecOps tools powered by AI, not much is available about security tools powering SecOps operations for AI and machine learning (ML) applications. That’s where this article comes in.
In this blog post, we’ll take a deep dive into the MLSecOps tools landscape by reviewing the five foundational areas of MLSecOps, exploring the growing importance of MLSecOps for organizations, and introducing six interesting open-source tools to check out—plus the AI security posture management solution offered by Wiz.
AI Security 101: 5 Tips to Fight AI Security Risk
Don’t let AI risks catch you off guard. Most AI security risks fall into 4 primary categories - Adversarial attacks, model inversion attacks, data poisoning, and model theft. Discover the 5 best strategies to secure them in this essential guide.
Download Guide
An intro to MLSecOps
MLSecOps, or AISecOps, is a new discipline that aims to define SecOps processes and best practices tailored specifically to securing ML and AI pipelines at scale. Combining key elements of SecOps and MLOps, MLSecOps emerged as a necessary augmentation to traditional DevSecOps, addressing the distinctive challenges of AI applications and MLOps.
What is MLSecOps?
Securing AI applications is the goal of MLSecOps, which is a field that first gained a wider audience in 2023.
MLSecOps is still very new (hence, the lack of information online!), but many resources are being championed by the MLSecOps Community. One of these contributions is the definition of five foundational areas of MLSecOps, including objectives that security teams can focus their efforts on:
Supply chain vulnerability: Assess the security of the entire AI supply chain.
Model provenance: Trace the origin, lineage, and evolution of AI models throughout their life cycle.
Governance, risk, and compliance (GRC): Establish policies, procedures, and controls for adherence to internal and external regulatory standards.
Trusted AI: Offer AI systems that are transparent, fair, and accountable to both internal stakeholders and external users.
Adversarial machine learning: Secure AI systems by testing against adversarial attacks, especially those aimed to influence the model’s behavior.
(Keep in mind that this is the first iteration of MLSecOps objectives, and you can expect the list to grow over time.)
The MLSecOps Community’s list addresses two fundamental aspects of AI applications: the reliance on open-source or third-party providers for datasets, libraries, frameworks, models, infrastructure, and solutions; and the non-deterministic nature of AI models, which makes them difficult to understand and protect as well as impossible to fully control. By focusing on these five areas of AI security, businesses can release AI applications that are safe for both the organization and users.
How are organizations embracing MLSecOps?
MLSecOps is a specialized discipline that requires ad-hoc solutions.
While some large organizations have already embraced MLSecOps, most organizations are just beginning—or considering beginning—their journey. Given the technical knowledge and resources necessary to get started with MLSecOps, it makes sense that most businesses are starting from square one.
Still, considering that MLSecOps is expected to become indispensable for all organizations as AI adoption keeps increasing—and considering that new regulations, such as the EU AI Act and the 2023 Executive Order on AI security, are coming into force—security teams need to make protecting AI applications a priority.
The State of AI in the Cloud Report 2024
Did you know that over 70% of organizations are using managed AI services in their cloud environments? That rivals the popularity of managed Kubernetes services, which we see in over 80% of organizations! See what else our research team uncovered about AI in their analysis of 150,000 cloud accounts.
Download Report
The top 6 open-source AI security tools
Choosing the right tools is the best way to bolster your AI security posture. Below you can learn about some of the most interesting open-source AI security tools available. These tools have been selected as widely applicable to a variety of AI models and frameworks, but other specialized open-source tools also exist, such as TensorFlow Model Analysis, and these are worth researching in order to meet your organization’s unique needs.
When evaluating the adoption of these open-source tools, keep in mind that they may have limited maintenance and support. Your safest choice is to rely on third-party security providers—you can learn about Wiz’s AI security posture management at the end of the article.
Gartner® Emerging Tech: Top 4 Security Risks of GenAI
In this report, Gartner offers insights and recommendations for security and product leaders to gain a competitive edge by addressing the key transformation opportunities presented by these risks.
Download Report
1. NB Defense
NB Defense is a JupyterLab extension and CLI tool for AI vulnerability management, offered by Protect AI.
JupyterLab is the most-used AI model development environment for data science teams across the world. By providing vulnerability management directly at the source of model development, NB Defense allows teams to embed security early in the ML life cycle. It also enables non-security personnel to directly introduce reliable security controls in a straightforward, easy-to-use way.
Main functionality: Detecting vulnerabilities early—from secrets and PII data to common vulnerabilities and exposures (CVEs) and third-party licenses—by providing contextual guidance for data scientists within JupyterLab and automated advanced repository scanning for security operators
MLSecOps area of focus: Trusted AI, via DevSecOps and vulnerability analysis
2. Adversarial Robustness Toolbox
The Adversarial Robustness Toolbox (ART) is a Python library for ML defense against adversarial threats, hosted by the Linux AI & Data Foundation.
AI can offer many different avenues for adversarial actors, from extracting users’ data to creating deep fakes or spreading misinformation. ART was created with both developers and researchers in mind: the library supports evaluating a wide variety of models and applications developed on all data types and defending them against the most common adversarial threats to AI.
Main functionality: Defends against evasion, poisoning, inference, and extraction adversarial attacks through a wide catalog of pre-built attacks, estimators, defenses, evaluations, and metrics
MLSecOps area of focus: Adversarial machine learning, via red and blue teaming
Foolbox and CleverHans are two libraries similar to ART that are worth checking out too.
3. Garak
Garak is a Python package for the vulnerability scanning of large language models (LLMs), created by Leon Derczynski.
The new wave of AI adoption has started with the commercialization of LLMs like ChatGPT. LLMs are quickly being adopted by many organizations to unlock business potential, often via third-party integrations. Garak offers the ability to scan all the most popular LLMs, from OpenAI to HuggingFace and LangChain, to ensure they are secure.
Main functionality: Provides pre-defined vulnerability scanners for LLMs to probe for hallucinations, misinformation, harmful language, jailbreaks, vulnerability to various types of prompt injection, and more
MLSecOps area of focus: Adversarial machine learning, via vulnerability analysis of LLMs for red teaming
4. Privacy Meter
Privacy Meter is a Python library to audit the data privacy of ML models, developed by the NUS Data Privacy and Trustworthy Machine Learning Lab.
AI models are trained on large amounts of data. Training data leakage is one of the most common and expensive threats to AI models. Privacy Meter provides a quantitative analysis of the fundamental privacy risks of (almost) any statistical and ML model, collected into ready-to-use reports with extensive information about both the individual and aggregate risks of data records. Privacy scores allow you to easily identify training data records at high risk of being leaked through model parameters or predictions.
Main functionality: Performs state-of-the-art membership inference attacks, customizable via configuration tiles to use a variety of predefined privacy games, algorithms, and signals
MLSecOps area of focus: Trusted AI, via risk assessment—especially as part of the data protection impact assessment process
5. Audit AI
Audit AI is a Python library for ML bias testing, offered by pymetrics.
AI models learn from the patterns provided in training data and may perpetuate biases and discrimination present within it. Audit AI provides ways to measure bias for statistical and ML models via a user-friendly package built on two libraries that data scientists are very familiar with: pandas and sklearn. Data scientists can use the bias results provided by Audit AI to drive changes in the model development pipeline that can mitigate bias.
Main functionality: Provides implementations of bias tests and algorithm auditing techniques for classification and regression tasks, such as Fisher’s z-test and the Bayes factor
MLSecOps area of focus: Trusted AI, via manual testing and auditing
6. ai-exploits
ai-exploits is a collection of exploits and scanning templates for real-world vulnerabilities, maintained by Protect AI.
Security teams can extend the AI expertise provided by internal SMEs by testing AI applications against the exploits collected in ai-exploits. Built on research performed by Protect AI and vulnerabilities discovered on the Huntr Bug Bounty Platform, this collection of real-world exploits aims to help you protect your current AI systems. ai-exploits also helps you vet third-party providers.
Main functionality: Scans for a variety of vulnerabilities via pre-built tools. Each tool is composed of modules to exploit the vulnerability and templates to automatically scan against the vulnerability. Currently only supporting H2O, MLflow, and Ray for remote code execution, local file inclusion, arbitrary file writes, cross-site request forgery, and server-side request forgery.
MLSecOps area of focus: Supply chain vulnerability and adversarial machine learning, via vulnerability scanning and red teaming
Boost your AI security with Wiz
For your cloud-native AI applications, you can rely on Wiz’s AI security posture management (AI-SPM) to secure your AI applications.
AI-SPM offers full-stack visibility into your AI pipelines by automatically discovering and documenting AI services and technologies to produce an AI-BOM, protecting you against shadow AI. AI-SPM also enforces secure configuration baselines with built-in rules that can detect misconfigurations in AI services as well as in your IaC, and it allows you to proactively discover and remove critical attack paths related to AI models and training data with accurate risk prioritization.
With Wiz, you can rely on state-of-the-art managed infrastructure and services for your AI security, which can help you set up a strong security layer for your AI pipelines—right now. Learn more by visiting the Wiz for AI webpage. If you prefer a live demo, we would love to connect with you.
Develop AI Applications Securely
Learn why CISOs at the fastest growing companies choose Wiz to secure their organization's AI infrastructure.
