*Update (15/01/23): On January 13th, CircleCI published their incident report and provided more details on the attack, including IOCs. We've added an Appendix with hunting queries based on the new details and updated the existing queries' date range to scan for events starting on the known initial compromise date, December 16th.
In a statement on Wednesday, January 4th, 2023, CircleCI reported that a security breach of their development platform resulted in the exposure of thousands of organizations’ sensitive data and secrets. CircleCI has strongly urged its users to take immediate action to protect their information by rotating any secrets they have stored on the platform, such as API tokens or credentials in environment variables. CircleCI users are also encouraged to review their internal logs for evidence of unauthorized access between December 21st, 2022 and January 4th, 2023.
The risk to organizations, however, transcends compromised secrets. Within that window of time between late December and early January, an adversary could have established a foothold within an environment via cloud APIs, enabling them to obtain persistence and therefore withstand a secret rotation.
In this blog, we will present some highly privileged cloud APIs in each one of the major CSPs (AWS, Azure, GCP) that attackers might abuse in order to gain persistence in cloud environments after accessing the environment using compromised keys. Additionally, we will provide hunting queries that will help security and IR teams investigate whether any suspicious activity linked to persistence has occurred using CSP-native interactive query services (AWS Athena, Azure Log Analytics, GCP Logs Explorer). Teams should hunt for signs of persistence regardless of secret rotation.
From compromised secret to ongoing attack
Even when secrets have been rotated or removed in the cloud, persistence (TA0003) can still be a threat to organizations if an attacker has gained initial access (TA0001) to the environment and established a foothold to maintain access. For example, a malicious actor may use compromised credentials or keys to spin up new instances, configure them to communicate with a C2 server, and access data in cloud storage. An attacker might also leverage cloud-native features and services like serverless functions to establish persistence in the cloud environment.
In order to continuously evaluate the security of their cloud environments, organizations should have a comprehensive strategy that goes beyond secret rotation/removal and includes network segmentation and monitoring. This is critical given there have already been numerous documented cases of APT groups using persistence techniques in cloud environments. For instance, APT29 (a.k.a. “Cozy Bear”) has reportedly compromised cloud infrastructure such as VMs to establish persistence, host C2 servers, and created a backdoor for service principals by generating new credentials for them. Furthermore, APT10 (a.k.a. “Stone Panda”) has also allegedly obtained persistence in cloud environments and subsequently performed data exfiltration.
Hunting for signs of persistence in AWS
Adversaries that have compromised secrets related to AWS access keys and secret access keys associated with users may execute any of the following API calls. These calls may allow them to persist in your cloud environments even after your keys have been revoked/rotated, which is why it is crucial to monitor activity to gauge whether they are hiding in your cloud account:
CreateUserAPI: Attackers might create new AWS users in the account and use them to gain persistence in the environment.CreateAccessKeyAPI: Attackers might produce new access keys for existing users and use them to impersonate users’ identities.CreateLoginProfileAPI: Attackers may create new login profiles (Console Login) for existing users without such profiles (e.g. possessing only access keys)UpdateLoginProfileAPI: Attackers could update an existing login profile associated with users and modify their Console Login password.ImportKeyPairAPI: Attackers may import the public key from a key-pair they created and associate it with a new EC2 instance, allowing them to persist via the cloud workload.RunInstancesAPI: Attackers might generate new EC2 instances and run malicious code on them (e.g. reverse shells), enabling them to persist via the cloud workload.ModifyInstanceAttributeAPI: Attackers could inject a malicious startup script into an existing EC2 instance via the `UserData` attribute that will run during each boot, allowing them to persist via the cloud workload.CreateFunctionAPI: Attackers might create new Lambda functions, inject malicious code into them, and invoke them, thereby persisting via the serverless functions.UpdateFunctionCodeAPI: Attackers may update existing Lambda functions’ code to include a malicious payload, enabling them to persist via the serverless function.SendCommandAPI: Attackers could abuse this API to run malicious commands on an EC2 instance and install backdoors on it, allowing them to persist via the cloud workload.StartSessionAPI: Attackers might connect via IAM permissions to an EC2 instance and install backdoors on it, allowing them to persist through the cloud workload.
In order to determine whether there is any sign of persistence, run the following SQL query in AWS Athena with any suspicious access key IDs:
Query #1 — Suspicious persistence activity by AWS principals
SELECT *
FROM "cloudtrail_logs_aws_cloudtrail_logs_********"
WHERE
eventtime >= '2022-12-16T00:00:00Z' AND eventtime < '2023-01-05T00:00:00Z'
useridentity.accessKeyId in (<list-of-access-key-ids>)
AND eventname in ('CreateUser','CreateAccessKey','CreateLoginProfile','UpdateLoginProfile','ImportKeyPair','ModifyInstanceAttribute','CreateFunction20150331','UpdateFunctionCode20150331v2','RunInstances','StartSession','SendCommand')
AND errorcode is nullRemediation
In the event the query has generated results, make sure to investigate each one. If you are either unfamiliar with a finding or deem it suspicious, follow the remediation steps below:
Remove—rather than disable—any new users or access keys that appear suspicious.
Disable or change the password of any updated login profile; enforce strict authentication methods like MFA for existing login profiles.
Remove any EC2 instance or new key associated with a suspected user.
Delete any new EC2 instance created by a suspected user.
Revert any modifications to an existing EC2 instance’s attributes (e.g.
UserData) by a suspected user, and search for any potential backdoors on it. If possible, back up any important data and delete the instance.Restore any modified source code of existing serverless functions by a suspected user and erase any new function.
Search for any potential backdoor residing on an EC2 instance when
SendCommandorStartSessionAPIs were initiated by a suspected user. If possible, back up important data and terminate the instance.
Moreover, adversaries may try to gain persistence by enumerating roles in your account and assuming them constantly. You can execute the following SQL query in AWS Athena with your preferred threshold to search for multiple failed attempts at assuming distinct role ARNs:
Query #2 — Abuse of unsuccessful AssumeRole operation
SELECT useridentity.principalId, useridentity.accessKeyId, useridentity.userName, COUNT(distinct resources) as numRoles
FROM "cloudtrail_logs_aws_cloudtrail_logs_********"
WHERE
eventtime >= '2022-12-16T00:00:00Z' AND eventtime < '2023-01-05T00:00:00Z'
AND useridentity.accessKeyId = '<suspected-access-key-id>'
AND eventname = 'AssumeRole'
AND errorcode is not null
GROUP BY useridentity.principalId, useridentity.accessKeyId, useridentity.userName
HAVING COUNT(distinct resources) > 5 Should the query return results, this would likely indicate suspicious behavior on the part of the chosen principals. You would then need to run a query with your suspected access key IDs to determine whether any role has been successfully assumed by the principal:
Query #3 — Successful AssumeRole operation by a highly suspicious principal
SELECT distinct resources
FROM "cloudtrail_logs_aws_cloudtrail_logs_********"
WHERE
eventtime >= '2022-12-16T00:00:00Z' AND eventtime < '2023-01-05T00:00:00Z'
AND useridentity.accessKeyId = '<suspected-access-key-id>'
AND eventname = 'AssumeRole'
AND errorcode is nullRemediation
In the event you have noticed any unfamiliar logs, revoke any session permissions from the role by following these steps.
Hunting for signs of persistence in Azure
Adversaries that have compromised secrets related to AAD service principals may execute any of the following commands to persist in your cloud environment even after the application’s credentials have been revoked or rotated. Make sure to monitor your audit and activity logs for a hidden adversary inside your cloud tenant:
Update application – certificates and secrets management (audit logs): Attackers might persist by generating new secrets or certificates for a compromised AAD application in the tenant.
Add user (audit logs): Attackers may persist in the environment by creating new AAD users in the tenant.
Reset user password (audit logs): Attackers could reset passwords for existing AAD users in the tenant and establish persistence by impersonating them.
Create or update an Azure Automation runbook (activity logs): Attackers might update or create an Automation runbook and launch a malicious script (e.g. PowerShell) to generate highly privileged AAD identities that serve as backdoors to the cloud environment.
Generate a URI for an Azure Automation webhook (activity logs): Attackers could invoke the Automation runbook via a webhook and continuously access the environment even if the runbook’s highly privileged AAD identities are removed.
Create or update virtual machine (activity logs): Attackers may persist in the cloud workload by creating or updating VMs and injecting a malicious startup script into them via the `UserData` attribute that will run with each boot.
Run command on virtual machine (activity logs): Attackers could persist in the workload by abusing this API to run malicious commands on a VM and install backdoors on it.
Update function code (activity logs): Attackers might update existing serverless functions’ code to insert a malicious payload and persist via the functions.
In order to determine whether there are any signs of persistence, run this KQL query in Azure Log Analytics and include any suspicious service principal object IDs:
Query #4 — Suspicious persistence activity in ARM by AAD service principals
AzureActivity
| where TimeGenerated >= startofday(datetime(2022-12-16)) and TimeGenerated < startofday(datetime('2023-01-05'))
| where Caller in ('<list-of-AAD-SP-Obj-IDs>')
| where OperationNameValue in ("MICROSOFT.AUTOMATION/AUTOMATIONACCOUNTS/RUNBOOKS/PUBLISH/ACTION","MICROSOFT.AUTOMATION/AUTOMATIONACCOUNTS/WEBHOOKS/WRITE","MICROSOFT.COMPUTE/VIRTUALMACHINES/WRITE","MICROSOFT.COMPUTE/VIRTUALMACHINES/RUNCOMMAND/ACTION")
or OperationNameValue matches regex @'MICROSOFT.WEB/SITES/HOSTRUNTIME/VFS/.*/WRITE'
| where ActivityStatusValue == "Success"Remediation
In the event the query has produced results, make sure to examine each one. If you are either unfamiliar with a finding or deem it suspicious, follow the remediation steps below:
Remove any suspicious Automation runbook that was created in an existing or new Automation account.
Remove any suspicious webhook that was created in an existing or new Automation runbook.
Restore any modified source code of existing serverless functions by a suspected service principal and erase any new function.
Delete any VM that was created by a suspicious service principal.
Remove any suspicious startup script injected into a VM by a service principal, and search for any potential backdoor on the VM. If possible, back up important data and delete the machine.
To gauge potential persistence in AAD, run this KQL query in Azure Log Analytics with audit logs and include any suspicious service principal client IDs:
Query #5 — Suspicious persistence activity in AAD by AAD service principals
AuditLogs
| where TimeGenerated >= startofday(datetime(2022-12-16)) and TimeGenerated <
startofday(datetime(2023-01-05))
| where InitiatedBy.app.appId in ('<list-of-AAD-SP-IDs>')
| where ActivityDisplayName in ("Update application – Certificates and secrets
management","Add user","Reset user password")
| where Result == "success"Remediation
If the query has generated results, investigate them. If you are either unfamiliar with the findings or deem them suspicious, follow the remediation steps below:
Remove any suspicious secrets or certificates that were created for existing or new AAD service principals. Additionally, remove any new unfamiliar AAD Applications associated to the
Delete any unknown, new user that was created by a service principal.
If a particular user’s password has been reset, change it and enforce strict authentication methods like MFA.
Hunting for signs of persistence in GCP
Adversaries that have compromised secrets pertaining to IAM service accounts can execute any of the following commands to persist in your cloud environment even after the service accounts’ private keys have been revoked or rotated. Monitor the service accounts’ activity for a potential adversary hidden inside your cloud tenant:
CreateServiceAccountKey: Attackers might create new private keys for existing service accounts in the environment and persist by impersonating them.CreateFunction:Attackers could create serverless functions, inject malicious code into them, and then invoke them in order to persist via the serverless functions.UpdateFunction: Attackers may update existing serverless functions’ code to insert a malicious payload and persist through the function.SSH to VMs (including internal VMs via IAP): Attackers might ‘SSH’ into existing VMs in projects via the
compute sshAPI and persist through the workload.Run startup scripts on VMs: Attackers may inject a malicious startup script into an existing VM that runs with each boot, allowing them to persist via the workload.
Create new compute instances: Attackers could obtain persistence through the workload by creating new compute instances and running malicious code on them.
To screen for persistence in GCP, execute this Logging query in Logs Explorer with any suspicious service account details:
Query #6 — Suspicious persistence activity by GCP service accounts
timestamp >= "2022-12-16T00:00:00Z" AND timestamp < "2023-01-05T00:00:00Z" AND
((resource.type="gce_instance" AND ((jsonPayload.message: "Updating keys for user
<Service-Account-Name>.") OR
(protoPayload.authorizationInfo.permission="compute.instances.setMetadata" AND
protoPayload.metadata.instanceMetadataDelta.addedMetadataKeys="startup-script" AND
protoPayload.authenticationInfo.principalEmail = "<Service-Account-Email>"))) OR
(resource.type="cloud_function" AND protoPayload.authenticationInfo.principalEmail =
"<Service-Account-Email>" AND
(protoPayload.methodName="google.cloud.functions.v1.CloudFunctionsService.UpdateFunction" OR
protoPayload.methodName="google.cloud.functions.v1.CloudFunctionsService.CreateFunction")) OR
(resource.type="service_account" AND
protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey" AND
protoPayload.authenticationInfo.principalEmail = "<Service-Account-Email>")) AND
severity!=ERRORRemediation
If the query has generated findings, inspect them. If you are either unfamiliar with them or view them as suspicious, implement the remediation steps below:
Remove any suspicious private keys that were created for existing or new service accounts.
Restore any modified source code of existing serverless functions by a suspicious service account and erase any new function.
Delete any VM that was created by a suspicious service account.
Log in to a VM with a service account that has initiated suspicious SSH connections, remove its home directory, and search for any potential backdoor on it. If possible, back up important data and delete the machine.
Remove any suspicious startup script injected into a VM by a service account, and search for any potential backdoor on the VM. If possible, back up important data and delete the machine.
Appendix — Queries for cloud activity from malicious IPs involved in the CircleCI incident
On January 13th, CircleCI released indicators of compromise (IOCs) associated with threat actors that targeted their environment. The following queries can assist teams in determining if their cloud environment has been accessed by any known malicious IP address.
Query #1 — Cloud APIs from known malicious IPs in AWS
SELECT *
FROM "cloudtrail_logs_aws_cloudtrail_logs_*********"
WHERE
eventtime >= '2022-12-16T00:00:00Z' AND eventtime < '2023-01-05T00:00:00Z'
AND sourceipaddress in
('178.249.214.10','89.36.78.75','89.36.78.109','89.36.78.135',
'178.249.214.25','72.18.132.58','188.68.229.52','111.90.149.55') Query #2 — Cloud APIs from known malicious IPs in ARM
AzureActivity
| where TimeGenerated >= startofday(datetime(2022-12-16)) and TimeGenerated <
startofday(datetime('2023-01-05'))
| where CallerIpAddress in
("178.249.214.10","89.36.78.75","89.36.78.109","89.36.78.135","178.249.214.25","72.18.132.58","188.68.229.52","111.90.149.55") Query #3 — AAD service principal sign-in activity from known malicious IPs
AADServicePrincipalSignInLogs
| where TimeGenerated >= startofday(datetime(2022-12-16)) and TimeGenerated <
startofday(datetime(2023-01-05))
| where IPAddress in
("178.249.214.10","89.36.78.75","89.36.78.109","89.36.78.135","178.249.214.25","72.18.132.58","188.68.229.52","111.90.149.55") Query #4 — Cloud APIs from known malicious IPs in GCP
timestamp >= "2022-12-16T00:00:00Z" AND timestamp < "2023-01-05T00:00:00Z" AND
protoPayload.requestMetadata.callerIp = ("178.249.214.10" OR "89.36.78.75" OR
"89.36.78.109" OR "89.36.78.135" OR "178.249.214.25" OR "72.18.132.58" OR "188.68.229.52" OR "111.90.149.55") Remediation
If any of the queries generated results, investigate each event separately. Validate that the old keys/secrets associated with the cloud identities have either been rotated or removed. Additionally, look for any signs of persistence mentioned above, as well as indicators of data exfiltration.
