Managing secrets and credentials is a difficult challenge in complex, internet-facing cloud environments.
Over the course of our recent research into cloud service provider vulnerabilities, we discovered three overlooked misconfiguration patterns that expose cloud environments to CI/CD supply-chain attack risks. These misconfigurations enable malicious actors to perform lateral movement and manipulate source code in order to tamper with organizations’ artifacts and spread malicious code to internal services or external customers.
In this blog post, we will offer best practices for mitigating CI/CD supply-chain attack risks and reinforce the need for constant monitoring of CI/CD environments for forgotten secrets and network misconfigurations.
Why are CI/CD pipelines valuable targets for attackers?
A CI/CD pipeline is a set of practices and tools that help organizations quickly and efficiently develop, build, and deploy high-quality software to their users while minimizing the risk of errors and bugs.
The CI/CD pipeline typically includes the following steps:
Code is written by developers and committed to a version control system, such as Git.
The code is automatically built and tested using continuous integration (CI) tools, like Jenkins and CircleCI.
The built code is automatically packaged and deployed to a staging environment for testing and validation.
If the tests and validation are successful, the code is deployed to production or stored in the artifact repository for services to pull and use.
CI/CD pipelines are popular targets for attackers because they contain numerous centralized components that impact organizations’ software supply chains. One compromised technology in the pipeline can therefore jeopardize various environments and subscriptions. For example, an attacker can compromise a container registry repository in a cloud environment and then spread malicious container images across the company’s internal services, or externally to customers via its software.
Since CI/CD pipelines are used to automate the building, testing, and deployment processes, they often have access to sensitive information like secrets and credentials that can be leveraged by attackers to conduct lateral movement. Moreover, the automation and complexity of CI/CD pipelines complicate attack detection and prevention, reinforcing their attractiveness to adversaries.
Common CI/CD misconfigurations and how to avoid them
Over the course of our research on cloud environments, we identified prevalent CI/CD misconfigurations that tend to be overlooked by security practitioners and compiled them into a list with recommended remediation steps.
Given attackers often exploit multiple security flaws to compromise a CI/CD component, merely addressing one of the following issues can significantly reduce your organization’s attack surface.
1. Overprivileged access to container registries
Container registries are commonly deployed in containerized cloud environments to store container images that are used in build and production phases. These private images can be pulled with container registry credentials that are often located in the file system of containerized instances.
If an attacker successfully compromises such an instance, they can obtain the container registry credentials from the file system and access all the images residing in the container registry.
With read-only permissions, the attacker can analyze the company’s applications, research for vulnerabilities, and hunt for sensitive secrets and proprietary code stored in the registry's images. With write privileges, the container registry credentials can enable the adversary to overwrite existing production images or spread malicious code to external users or other internal services utilizing the images.
Example attack flow:
An attacker compromises an internet-facing server within a containerized environment
They find container registry credentials in the file system
They leverage the credentials to fetch additional private images from the registry
They infect the images with malicious code
SOLUTION: Least privilege access to container registries
Ensure your container registry solutions enforce proper access controls and scoping. While most cloud service providers provide registry permissions management via IAM identities, SaaS providers typically offer it through the service’s web interface or API.
Assign read-only or write-only privileges to roles, and restrict those with pull and push permissions solely to the repositories and images they need. For instance, developers do not require write permissions as they should not be directly pushing images to registries or containerized services running on container orchestrations. As for roles with push permissions, they should only be involved in the last phase of the CI/CD pipeline to push the final images to the registry.
Additionally, avoid granting the listing resources permission to applications and automations that use container registries in order to prevent an attacker from conducting reconnaissance and learning which repositories can be accessed.
Case Study: Pod service account access to container registries
In order to pull private images, a Kubernetes node must have access to the container registry in which they are stored. The credentials to this container registry are found in a secret resource. The secret’s name is mentioned in each pod configuration’s imagePullSecrets field.
In the event an adversary compromises a pod with access to the container registry secret, they can exploit these credentials to retrieve all the accessible images from the registry. These in turn may contain proprietary code or additional secrets. In the event of container registry credentials with write permissions, the malicious actor could overwrite production images and spread malicious code internally or externally.
Example attack flow:
An attacker compromises a pod in an internet-facing application
They find a Kubernetes service account token in the file system
They use the token to fetch the container registry secret from the cluster
They leverage the credentials to fetch additional private images from the registry
They infect the images with malicious code
SOLUTION: Restricted pod service account access and privileges
The most effective way to prevent this supply-chain risk is to limit pods' service account privileges and curb access to container registry secrets.
2. Forgotten secrets in the build process
Forgotten secrets in cloud environments pose a serious security risk. These secrets, such as cloud access keys, passwords, CI/CD credentials, and API access tokens, are often left behind in production artifacts after the build and deployment processes.
During our research, we found sensitive secrets in various overlooked locations, including Linux bash history files and container image base layers. Some secrets may appear in an image layer but will be hidden in the final squashed file system. These forgotten secrets can allow an attacker to conduct lateral movement and then execute remote code to tamper with images’ build processes.
Example attack flow:
An attacker compromises an internet-facing server
They find CI/CD credentials in the
.bash_historyfile that were from the build process of the OS imageThe secret grants access to an internal storage bucket containing production-ready artifacts used in the build process
The attacker modifies the artifacts to spread malicious code in the next builds
Example attack flow 2:
An attacker fetches publicly available container images from the company's public container registry
They find sensitive secrets in the containers' history files
The images feature all the historical commands used to build the images as well as a key to an internal storage bucket
The storage bucket contains production-ready artifacts used in the images’ build process
The attacker modifies the artifacts in the bucket to spread malicious code in the images’ builds
SOLUTION: Comprehensive scanning for potentially forgotten secrets
To prevent attackers from obtaining sensitive secrets, your organization should implement regular scanning of your environments and production artifacts for secrets in the final phase of each artifact build. This can be done using relevant products or open-source tools such as Trufflehog.
In addition, you should scan all image layers and metadata files as they may be accessible to potential attackers. This can protect sensitive information from being accidentally exposed and subsequently exploited. For example, secrets could be hidden in:
• AWS SSM Logs — /var/log/amazon/ssm/*
• Bash history —~/.bash_history or /root/.bash_history
• Linux Journal files — /var/log/journal/*
By taking these steps, you can safeguard your artifacts and preserve your CI/CD pipelines’ security.
3. Lateral movement paths between production and internal build environments
The connection between an organization's internet-facing applications and its internal CI/CD components can provide adversaries with the ability to carry out lateral movement.
In the Hell's Keychain attack, the lack of adequate network controls allowed researchers to access an internal CI/CD repository. This misconfiguration is not unique to IBM Cloud, as it has been observed across multiple cloud providers and their customers. In our view, this issue is often overlooked in industry discussions and cloud security solutions.
Example attack flow:
An attacker compromises an internet-facing server
They find sensitive secrets in the
.bash_historyfile that were from the build process of the OS imageThe secret grants access to an internal storage bucket
The attacker tries to connect to the storage bucket
They fail due to the restrictive resource policy, which only allows connections from the CI/CD virtual private network
SOLUTION: Isolated critical assets
Implementing strong network policies and enforcing cloud segmentation can protect your organization from this type of CI/CD supply-chain attack. It is also important to monitor cloud access keys in internet-facing environments and verify they do not grant access to CI/CD environments. Furthermore, cloud identities and storage buckets should be restricted to the CI/CD virtual network.
In cases of SaaS CI/CD solutions like GitHub where network access is impossible to curb, it is critical that no service credential or access token be stored in internet-facing applications. If an attacker managed to compromise such an application, they would be able to bypass all network restrictions and directly access the service holding the production code and artifacts.
Finally, you should consider adopting security measures such as two-factor authentication and role-based access controls to safeguard your SaaS CI/CD solutions.
Summary
CI/CD pipelines are an essential part of the software development process, enabling organizations to rapidly and consistently deliver high-quality software. These pipelines, however, are an attractive target to malicious attackers: one compromised component can jeopardize multiple environments and subscriptions.
In this blog post, we have described common CI/CD pipeline misconfigurations and provided tips on how to remediate them in order to prevent supply-chain attacks. These misconfigurations include overprivileged container registry access, forgotten secrets, and a link between the production and CI/CD environment. Implementing just one of the best practices above can significantly reduce your organization’s attack surface and protect your software supply chain.
