Wiz Research has conducted a series of investigations into leading AI-as-a-service providers in recent months. In the course of that work, we have discovered critical vulnerabilities that could have led to the leakage of millions of private AI models and apps. The first installment in that series was done in partnership with the Hugging Face team. Now, in this second installment, we will detail a vulnerability in the Replicate AI platform. 

What we uncovered with Replicate proves our AI hypothesis to be true: challenges of tenant separation carry across AI-as-a-service providers and warrant close inspection given the explosive growth and near ubiquity of these services. Exploitation of this vulnerability would have allowed unauthorized access to the AI prompts and results of all Replicate’s platform customers.  

Wiz Research responsibly disclosed this vulnerability to Replicate in January 2024, and it was immediately clear how seriously they take security matters. We cannot overstate how well Replicate communicated with our team and how swiftly they acted to fix the issue, which was promptly mitigated. No customer data was compromised, and no action is required by customers. 

Introduction

Replicate.com is a platform that enables sharing and interacting with AI models; the company tagline is “deploy custom models at scale, all with one line of code.” Users can browse existing models on the hub, upload their own and finetune models for their specific use cases. Replicate also provides their customers with the ability to host private models and provides an inference infrastructure with a convenient API.  

As we have written previously, malicious AI models represent a major risk to AI systems, and especially for AI-as-a-service providers. Since AI models are often packaged in formats that allow arbitrary code execution, attackers could leverage these models to perform cross-tenant attacks. 

Remote code execution via a malicious model

To enable easy inference of AI models on their platform, Replicate uses their own format for containerizing models named Cog. Cog helps users containerize their AI models, include the correct dependencies and libraries, and bundle a RESTful HTTP API server for easy inference. At the end of the process, a container image is produced. After containerizing a model using Cog, users can upload the resulting image to Replicate’s platform and start interacting with it. 

We created our own malicious Cog container and uploaded it to Replicate’s platform. Then, using Replicate’s interface, we interacted with the container to achieve remote code execution on Replicate’s infrastructure. 

We suspect this code-execution technique is a pattern, where companies and organizations run AI models from untrusted sources, even though these models are code that could potentially be malicious. We used the same technique in our earlier AI security research with Hugging Face and found it was possible to upload a malicious AI model to their managed AI inference service and then facilitate lateral movement within their internal infrastructure.  

From RCE to lateral movement 

After gaining remote code execution as root within our own container on Replicate’s infrastructure, we started investigating the environment. We quickly noticed that we were running within a pod inside a Kubernetes cluster hosted on Google Cloud Platform. The pod we were running in was not a privileged pod, so we could not easily escape to the node. Furthermore, our pod also did not have a Kubernetes service account attached to it, so we could not communicate with the Kubernetes API server. 

We started investigating the network. By using netstat we noticed that there was an already-established TCP connection handled by a process that was running in a different PID namespace. This made us conclude that we shared our network namespace with another container, but not the PID namespace.  

This phenomenon is quite common. In Kubernetes environments, different containers within the same pod share the same network. As development teams often aren't aware of this, it could impair the containers' isolation security and allow an attacker who has gained access to one container to launch network attacks on other containers within the same pod. We've seen this pattern before, and we even created a specific exercise around this issue at our latest Kubernetes CTF (K8s LAN Party). 

As we were running with root privileges and were capable of CAP_NET_RAW and CAP_NET_ADMIN, we could use tcpdump to examine the contents of this TCP connection. Looking at content of the stream we realized it was a plaintext Redis protocol (a popular, open-source, in-memory data store). By performing a reverse DNS lookup on the IP address, its hostname confirmed that it was indeed a Redis instance. 

At this point we were trying to understand what the purpose of this Redis server is. By inspecting the plain-text stream we understood that this Redis instance operates a queue of some sort, and we were wondering – does it serve only our account? Or do other Replicate customers share the same queue?  

After further exploration of the environment, we realized that it’s likely that this Redis instance serves multiple customers, which made it an interesting target for a cross-tenant data access attack. If we manage to access or modify information inside this queue, we should be able to influence the predictions of other customers using the platform – maybe we can question their private models or even interfere with their requests. 

We tried to initiate a new connection to this Redis server using the redis-cli, but the server required credentials for authentication. We could not find these credentials anywhere within our pod. However, we did have an already-authenticated, plaintext, active session within our network namespace, and we had all the network capabilities. What if we injected arbitrary packets to the existing authenticated connection of our neighboring container? 

Injecting Redis commands to centralized Redis

We used rshijack (a utility for TCP injection) to inject arbitrary packets to the existing TCP connection to the Redis server of our neighboring container in order to bypass the authentication. It worked! We performed further reconnaissance on the Redis server and concluded that it’s being used as a queue to manage customer requests and their responses, and that it serves multiple customers (as opposed to just us). 

For the purpose of proving cross-tenant data access to other customers, our plan was to employ our TCP injection to modify an item within the queue that belonged to another account of ours. However, since the queue was managed in Redis Streams, which is append-only, modifying an existing item in the queue proved to be quite challenging.  

Interfering with Customer Prompts An item that represented a request from a customer in the Redis queue contained some interesting fields: 

• Identifiers regarding the model being used. 

• The input the user supplied (the prompt). 

• Some metadata regarding the user that requested the prediction. 

• A webhook field. Whenever there’s an update regarding the prediction, the address in the webhook field is notified. 

Given that we couldn’t modify existing items in the queue, we ended up using our TCP injection to inject a Lua script that did the following: 

• Found the item we want to modify in the queue. 

• Popped it from the queue. 

• Modified its webhook field to be the address of rogue API server under our control. 

• Pushed the modified item back to the queue. 

We were concerned that the server processing the items in the queue would not be able to communicate with our worker pod, so we decided to host our rogue API server on the internet and tunnel its traffic back inside our worker pod.

After injecting our Lua script to the TCP connection, our rogue API server received an HTTP request with the prediction input, and another with the prediction output. We also proved that our rogue server could modify the output prediction. 

Impact

The exploitation of this vulnerability would have posed significant risks to both the Replicate platform and its users. An attacker could have queried the private AI models of customers, potentially exposing proprietary knowledge or sensitive data involved in the model training process. Additionally, intercepting prompts could have exposed sensitive data, including personally identifiable information (PII). 

The ability to alter prompts and responses poses a severe threat to the functionality of AI applications. It gives attackers a way to manipulate AI behavior and compromise the decision-making processes of these models. Such actions directly threaten the accuracy and reliability of AI-driven outputs, undermining the integrity of automated decisions and potentially having far-reaching consequences for users dependent on the compromised models. 

Takeaways

The Wiz research team is highly focused on isolation vulnerabilities in cloud environments. When we saw the rise of AI-as-a-service companies, we were concerned about the potential implications of a malicious actor leveraging them to gain privileged cross-tenant access, since AI models in practice are actually code. 

Malicious models represent a major risk to AI systems, especially for AI-as-a-service providers because attackers may leverage these models to perform cross-tenant attacks. The potential impact is devastating, as attackers may be able to access the millions of private AI models and apps stored within AI-as-a-service providers. 

This research also highlights the value of collaboration between security researchers and platform developers. Collaboration of this type aids in gaining a deeper understanding of the risks associated with the platform, and ultimately enhances its security posture.  

PEACH framework

As the issue we uncovered in this research is ultimately an isolation issue, we think this is a great opportunity to mention PEACH. PEACH is a step-by-step framework we developed for modeling and improving SaaS and PaaS tenant isolation. Whether you're a cloud customer or a cloud application developer, PEACH makes it easier to understand how multi-tenant applications should be secured in the cloud. 

Responsible disclosure

We responsibly disclosed this issue to Replicate in January 2024. Replicate promptly investigated and addressed it by implementing new mitigations. They have also published a very informative post about this same issue.

Throughout the disclosure process, Replicate not only demonstrated a deep commitment to maintaining the security of its products, but also upheld the highest standards of communication and transparency. In our disclosure work with various third-party organizations, the Wiz Research team has rarely encountered such professionalism and expertise. Kudos to the entire Replicate team for setting such a high bar! We commend you for your cooperation and responsiveness. You represent an example to everyone.