Attackers with cloud knowledge will look for cloud credentials on any resources they gain access to.  On AWS, this may mean looking for IAM role credentials, but there are many ways they might obtain those credentials beyond making requests to 169.254.169.254.  

 The Wiz Sensor needs to detect those methods, so this blog post will discuss how various AWS services give applications IAM role credentials, along with other arcane knowledge of AWS credentials.  

AWS SDK credential providers 

The AWS SDK defines a number of ways in which credentials can be obtained.  

1. IAM user access keys are one way AWS privileges are obtained.  These access keys can unfortunately still be found in many places, such as source code.  Credentials that are explicitly set in code will take precedence by the SDK.   

2. The next place for the SDK to look will be environmental variables.  AWS Lambda, which was released in 2014 for serverless execution, puts the IAM role session credentials in the environment variables by using the variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN (docs).  Prior to the release of IAM roles in 2012, EC2 user-data (http://169.254.169.254/latest/user-data) was sometimes used to store credentials, and these were often set as environment variables before use. 

3. The AWS SDK will then look in the files ~/.aws/credentials and ~/.aws/config for credentials.  Older file paths that the SDK may use are /etc/boto.cfg and ~/.boto.   

4. Now we get to the Instance Metadata Service (IMDS), but it’s not just a single IP address that some think it is.   

IAM roles were first created for EC2 instances back in 2012, and involved an HTTP GET request to http://169.254.169.254/latest/meta-data/iam/security-credentials/ to learn the name of the IAM role, and a subsequent request to the get the session credentials.  In 2019, AWS released IMDSv2 which made a few changes to this (a PUT request, challenge response, Time-To-Live, and HTTP header), to improve it.  That magic host can also be accessed by IPv6 via the address [fd00:ec2::254]. LightSail supports this same mechanism for obtaining role credentials.   

Container services on AWS, such as ECS and EKS, use the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI along with the IP address 169.254.170.2 to perform an HTTP GET to a URL such as http://169.254.170.2/v2/credentials/00000000-0000-0000-0000-000000000000. That same technique also works for SageMaker Notebooks, App Runner, CodeBuild, and Batch.  Nick Jones documented much of this and more in his blog post AWS Access Keys - A Reference. 

That URL can also be found through the environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI which is used by CloudShell and IoT Greengrass 2.0 to reach a service running on localhost and also requires setting the HTTP header Authorization to the value of the environment variable AWS_CONTAINER_AUTHORIZATION_TOKEN. 

EKS Pod Identities works similarly, but uses the IP address 169.254.170.23 (or [fd00:ec2::23] for IPv6).  This also uses the variable AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE (by default set to /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token) which sets an HTTP header Authorization to the value of that file. This is shown in the AWS SDK here.  Prior research by Wiz on this feature can be found here.  

IRSA (IAM Roles for Service Accounts) was released before EKS Pod Identities, and uses OIDC instead. The container has the environment variables AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN set, which are used to make an anonymous call to sts:AssumeRoleWithWebIdentity.  By default the token file will be at /var/run/secrets/eks.amazonaws.com/serviceaccount/token.   

Default Host Management Configuration 

The above notes on the SDK describe the ways to obtain the “normal” IAM role of an EC2 or other resources, but did you know an EC2 can have two IAM roles assigned to it? 

AWS Systems Manager (SSM) includes functionality for an agent to run on your compute resources that can then manage them by doing things like downloading and executing scripts. Customers may find it useful for the agent to have access to an S3 bucket or be given other AWS privileges. 

In order to avoid forcing the IAM role associated with all EC2s at a company to be modified for the privileges someone wants to assign to this agent, AWS released Default Host Management Configuration (DHMC).  This feature must be enabled for a region to use it.  By default, this uses an IAM role named AWSSystemsManagerDefaultEC2InstanceManagementRole and has the AWS Managed IAM policy AmazonSSMManagedEC2InstanceDefaultPolicy, but the role name can be changed and more privileges can be added. 

This feature uses http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance and a few additional steps described here in research by Aidan Steele.  The additional steps involved in getting the creds involve generating a key pair, which is stored in /var/lib/amazon/ssm/Vault/Store/EC2RegistrationKey.  The key pair within that file can then be used without the need to access the metadata service (ie. access to that file is sufficient for obtaining credentials as if the EC2 was requesting them). 

The SSM agent will then store these credentials in /var/lib/amazon/ssm/credentials or /root/.aws/credentials. 

Systems Manager hybrid activation 

Systems Manager hybrid activation is also based on the SSM agent, and is used for managing compute resources within an on-prem environment or other non-AWS resources.  This same technique is also used by ECS Anywhere and is part of IoT Greengrass. The agent is activated using an activation code and activation id, and then creates the files /var/lib/amazon/ssm/Vault/Store/RegistrationKey and /var/lib/amazon/ssm/Vault/Store/InstanceFingerprint which are then used to obtain credentials.  Similarly to DHMC, access to these files is sufficient for obtaining the credentials. 

Internet of Things 

The API iot:AssumeRoleWithCertificate was the first mechanism AWS released that was specifically made for non-AWS environments.  This uses an X.509 certificate, and although the trust policy involved references that privilege, the mechanism uses an account specific endpoint prefix and some other features as described here. This certificate file can be placed anywhere and has no discerning attributes that identify it as being used for this use case so there is not a simple way for defenders or attackers to find these certificates and understand this is their use case. 

IAM Roles Anywhere 

Released in 2022, this service is the most recent way, at the time of this writing, that AWS has provided for non-AWS resources to access IAM Roles.  It also uses X.509 certs and initially only worked with those files being stored on disk.  In September 2023, AWS added PKCS#11 capabilities, so that cryptographic modules could be used to store the certificates instead. If that is used, then there will be a credential_process that can be called to obtain the credentials. 

Cognito 

Cognito is a service for customer identity and access management.  It offers an API called GetCredentialsForIdentity which is passed an identity ID, which is just a region and GUID value, and will return AWS session credentials. 

Datasync 

Datasync is an AWS service for keeping two storage locations in-sync, such as ensuring that an S3 bucket has the same files as a local directory on a server.  Although it does not appear to obtain IAM role credentials, it is able to access the objects of an S3 bucket through an undocumented mechanism.  Within /usr/local/aws-storage-gateway/var/ it will use the files cert.pem and keypair.pem to authenticate to AWS (some discussion here), and the datasync agent will then use those to potentially sync an S3 bucket and a local directory.  

Conclusion  

There are many ways that compute services on AWS obtain their credentials and there are many features and services that have special credentials.  This can result in a single EC2 having multiple IAM principals accessible from it.  In order to detect attackers, we need to know the various ways they might attempt to obtain these credentials.  This article has shown how this is not a simple problem and requires defenders to have just as much, if not more, expertise as attackers in credential access.