Takeaways:
Everyone that uses VPC Endpoints should enable network activity events for
VpceAccessDeniederrors.For those that have not yet applied restrictive VPC Endpoint Policies, Network Activity events can ensure existing functionality will not be disrupted.
In some cases, network activity events will be a better selector for relevant Data Events than configuring CloudTrail Data Events or Resource-based Data Events like S3 Access Logs.
AWS recently released a new logging capability for VPC Endpoints.
VPC Endpoints are used for private connections between your VPC and supported AWS services, without requiring internet access or using public IPs. They can enhance security by keeping traffic within the AWS network.
VPC Endpoint Policies are resource-based policies that control access to services through a VPC endpoint. This allows fine-grained permissions on which actions can be performed through the endpoint, by which principals, and against which resources.
CloudTrail network activity logs for VPC Endpoints is a new, opt-in class of CloudTrail event. It gives visibility into all AWS API activity that passes through your VPC Endpoints, for supported services.
Network activity events for VPC Endpoints
Network activity events closely resemble CloudTrail management and data events. They include both management (for example, S3 ListBucket) and data plane (for example, S3 GetObject) API calls that traverse your VPC Endpoints. You can find examples of these new events at the bottom of this article.
At launch, only five AWS services were supported: CloudTrail, EC2, KMS, S3, and Secrets Manager. Notably, S3 had been added since the initial preview. There are currently over 200 services that support VPC Endpoint Policies. Hopefully, we’ll see expanded support for network activity events continue.
Sample use cases
These new logs address two general needs.
1. Safely develop and manage VPC Endpoint Policies
VPC Endpoint Policies are a core element of identity-based access restrictions for a data perimeter through their control over which identities have access to services. However, it can be challenging to determine whether a proposed policy might inadvertently disrupt business operations.
Before, you would need to identify all relevant Principals and Resources to your VPC Endpoint, and then ensure you enabled and correlated all relevant Management and Data Plane event activity. Even if you did that, you would not have visibility into the (hopefully unlikely) use case of external principals inside your accounts interacting with external resources. These new logs let you gain centralized visibility into how existing and proposed policies might impact access.
You can use this visibility to evaluate whether a planned change will impact ongoing traffic. Additionally, the option to track and monitor denied requests allows you to detect and diagnose when legitimate traffic is unintentionally blocked.
2. Detect data exfiltration
VPC Endpoint network activity events provide a new source of data on how identities in your VPC are accessing supported AWS services.
While VPC Endpoint Policies can be effective in restricting access to services, this new visibility can support detective controls. That applies both in the case of permissive policies, or in the case of policies successfully preventing attacker activity from transiting your VPC Endpoints.
When attackers use both their own identities and their own resources, for example to exfiltrate data from your VPC to an S3 bucket they control, network activity events offer an opportunity for detection. This can provide a compensating control to gaps in your VPC Endpoint Policies.
Network Activity events for an external principal that was denied inside the VPC will only include a minimal amount of information from what is normally seen in CloudTrail logs. They will not include a user identity ARN or user agent field. Investigation will be limited to sourceIPAddress, accountId, and an opaque principalId. It is currently possible to lookup the full ARN from a principalId.
These logs can also be useful to understand linkages between services in the VPC and your other AWS resources, as well as to detect data exfiltration through unexpected connectivity, or by heuristics on events.
Network activity events for denials offer a chance to catch attackers attempting to use unapproved identities or access unapproved resources. You may also discover legitimate traffic has been blocked, so you can adjust the policies to avoid further impacting business traffic.
Recommendations for Network Activity Events
You should review the needs of your specific environment to determine whether, and to what extent, to enable network activity events. In all cases, if you have any VPC Endpoints in your environment, we recommend that you minimally enable logging for VpceAccessDenied events. As the eventSource field does not support wildcards, you need to configure this for each event source explicitly. The AWS documentation provides an example of how to do that for S3 here with examples for other services working similarly.
These events should be rare in a well-managed environment, which means you gain important visibility without outsized expense. CloudTrail network activity events are the only way to log certain ways in which these events might occur.
From a cost perspective, network activity events operate under a similar pricing structure as Data Events ($0.10 per 100,000 events delivered). Because network activity includes management events, which are otherwise free for the first trail, they can end up more expensive than relying on Management and Data events directly. You should evaluate whether you have a need for network activity events more broadly than the access denied events.
You should assess the relative cost and coverage of enabling additional network activity events, versus using Data Events. Generally, the determining criteria should be:
whether you have a known set of resources allowed for the VPC Endpoint
whether you have other reasons to enable Data Events for those resources
whether those resources are accessed frequently from outside the VPC
The goal would be to avoid the duplication of cost logging from data plane events via both Data Events and network activity events. Network activity events will show the most value when you don’t know the relevant resource relationships, or when those related resources may have irrelevant usage that would add to the cost or complexity of configuring Data Events.
Finally, if you’re pursuing a stringent data perimeter, you should consider network activity events, as these are the sole visibility (within AWS event logging options) into attackers bringing their own identities and resources and using those for exfiltration.
How can Wiz help?
Wiz Defend automatically ingests these logs if you have enabled the collection of them in your CloudTrail, and has detection rules for suspicious activity within them.
Conclusion
The goals of a data perimeter strategy are worth working towards and AWS is working to help customers transition toward this strategy. CloudTrail network activity events for VPC Endpoints fill an important gap for companies building out a data perimeter in AWS. If you are using VPC Endpoints, enabling these events will provide enhanced visibility into both approved and denied requests. This visibility can help you tune policies and detect bad actors. The additional cost of these new logs is an important consideration however, and the billing impact should be estimated before adoption.
Appendix: Sample Events
Copies of full redacted events are available in a GitHub gist.
One important pattern to look for is an external principal being used from within your VPC, which will have the userIdentity field limited to the following:
"userIdentity": {
"type": "AWSAccount",
"principalId": "AIDAEXAMPLE",
"accountId": "012345678901"
}Events will have:
"eventType":"AwsVpceEvent",
"eventCategory":"NetworkActivity" Denials based on VPC Endpoint Policy will have:
"errorCode":"VpceAccessDenied",
"errorMessage":"The request was denied due to a VPC endpoint policy"
