In the past few months, there has been a lot of activity among security researchers identifying AWS IAM Roles that were intended to be accessed only from Github Actions associated with specific repos, but were misconfigured so that anyone could assume them and gain access to the victim’s AWS account. Wiz has already detected this issue for customers for a long time, but during the past few weeks, a few things happened to make this misconfiguration less likely to happen again, which is worth looking at for lessons about how other security issues can be fixed.
The misconfiguration
In order to avoid using long-lived AWS IAM user access keys which have security risks, Github released functionality in 2021 for integrating with AWS via an IAM role. Unfortunately, some customers were not implementing this integration correctly. The reasons for these misconfigurations are interesting, as you can’t place the blame directly on the customers or any of the other characters in this story.
Let us begin by showing what the correct integration looks like (which is described in the AWS docs here). This screenshot shows what an IAM role trust policy should look like when an OIDC (OpenID Connect) integration with Github is used:
The misconfiguration these researchers identified was that the sub condition was missing. That condition restricts access to a specific Github repository, which meant that any Github repository could create a Github Action that accessed this AWS IAM role. This meant an attacker could gain access to the AWS accounts of those that had misconfigured this.
Over the course of 2023, four sets of researchers wrote tools to identify this problem, informed vulnerable organizations of issues they identified, and published blog posts with at least one conference presentation about this issue in what seems to have been a twin film phenomena for at least some of them where they were doing this research in parallel before the others had published about it.
Many readers of those blog posts might have assumed that developers were just being lazy and removing this condition in order to reuse the role across multiple Github repos or committing some other user error. But the post from Datadog identified an issue in Terraform that could result in this problem. The issue is that if the JSON in terraform has two elements in a dictionary with the same name, one of the elements will be silently ignored. This would be a mistake for the user to write such a policy, but this outcome is normal for many JSON processing libraries to drop the duplicate element. Terraform uses the Go language’s “encoding/json” package, and that is how that package works.
Is this mistake really that common? I did some google searching on how to setup a Github Action to AWS IAM Role integration and one of the top hits had this mistake in it! This meant that many had likely copy/pasted the incorrect terraform code and ended up with this misconfiguration.
Recent actions taken
At this point, I decided to take action. I should pause here to comment that I don’t know how many others had already started taking action as well, and perhaps things were already on track to be fixed without me. For example, an issue had already been filed against terraform (albeit years ago) as noted by the Datadog researcher in their blog. But the only perspective I have is my own, and it feels like I had an impact. I’ve joked that I sometimes wonder if my reaching out to others about problems like these is like how my dog barks at the delivery person who drops off a package and walks away… and my dog feels proud that his bark had an impact. However, I didn’t want to let a bystander effect allow for this misconfiguration to continue, and I’m lucky that I have the flexibility and encouragement at my job to hunt down things like this, for the betterment of cloud security for all.
Step 1: I commented on the terraform issue to explain the security consequences of it (which had not yet been noted) so I could use this as a reference for other discussions later.
Step 2: I tracked down the author of the popular tutorial containing the mistake and explained the necessary change, which they immediately fixed. 🎉
Step 3: I created a honeypot based around this misconfiguration with a public Github repo to see whether attackers were already automatically scanning for and abusing this problem. As of this publication, attackers have not yet abused my honeypot.
In creating the honeypot, I noticed that the AWS console wizard for creating this integration did not have a way to select a repository to limit access to. OIDC integrations work differently with different identity providers, of which Github is just one of many, so the general wizard did not have a Github specific wizard flow. To implement this integration correctly using the web console, you would need to go through the wizard flow, and then manually make an edit to the resulting policy. This brings me to the next step.
Step 4: I reached out to AWS security to ensure they were aware of the issue (which they were), and to recommend an improvement to the web console wizard flow to make a special flow when a Github integration is being created (which they were working on already as well). That change has now been deployed! 🎉
They also sent emails on August 18 to all customers whose roles contained this misconfiguration. 🎉
AWS already has had Access Analyzer Policy Validator checks, so when you manually edit an IAM Role trust policy in the web console, there have been warnings for missing the Github repo conditions.
Step 5: I reached out to Hashicorp’s security team to ensure they too were aware of the specific issue with terraform and how it can lead to security problems. The issue in terraform is not straightforward to resolve because the JSON processing there works the same as nearly every other tool. Hashicorp has released a security bulletin about this, initiated efforts to have terraform emit a warning when it encounters this issue, and released semgrep rules to detect it.
Where we are now
On September 6, 2023, AWS sent out emails again to all customers with this misconfiguration, but this time informed them that effective immediately no new role trust policies can be created with this misconfiguration! For roles that already have this trust policy, they will continue to work and can be changed, but on November 1, they will be subject to enforcement of this change so that they must include the sub condition key.
This issue can no longer be created except by those with the existing misconfiguration. This story shows the security community at its best: folks from different companies all making different efforts to try to eradicate a security issue by informing the organizations with the misconfiguration and working to reduce the likelihood of it happening again by applying ratcheting pressure on a variety of points. It is useful to consider this example — a community effort that successfully addressed a security issue — as a template for how to confront problems in the future.
