Back in 2019, Amazon’s retail business publicly mentioned they had over 10,000 AWS accounts — and they certainly aren’t the only business with a high account volume.
This matters because a core building block of an effective cloud security strategy is how new accounts are created. Organizations must develop and maintain a consistent process for this account creation to ensure baseline security controls are implemented everywhere. To address this need, many companies deploy a Landing Zone, such as AWS Control Tower. But they may not be aware of what additional steps can be taken as part of a broader account vending strategy, which can have benefits for both developers and security teams.
The Landing Zone is sufficient for perhaps dozens of accounts, but consideration needs to be given to the account vending strategy as a business moves beyond that. In this blog post, we will discuss how Landing Zones focus on what exists inside AWS accounts, while an account vending strategy focuses on what exists outside of the account.
Why do companies end up with so many accounts?
Security isolation is a big reason companies begin creating large numbers of accounts. Another reason is to attribute ownership for cost and security fixes. It is much easier to find the owner of an expensive EC2 or a public S3 bucket when a small team or individual explicitly owns the account containing that resource. The other way companies try to solve for this is with tags, but I believe companies will benefit more from encouraging account proliferation than implementing and enforcing a tagging strategy.
The Landing Zone
Once you have multiple AWS accounts, and are considering the possibility of more, you will want to ensure you have the same security controls and functionality between them. This is where the concept of a Landing Zone begins. AWS created Control Tower as a service to deliver this capability, and has also provided alternatives such as Landing Zone Accelerator. Meanwhile, some companies opt for custom solutions.
The initial feature set of a Landing Zone looks like this:
CloudTrail and possibly other AWS security services or logging sources enabled.
Service Control Policies (SCPs) used as guardrails for all accounts.
Integration with an identity provider for SSO.
Enable additional security settings, such as default EBS encryption.
Deployment of vendor solutions, such as a cloud security vendor.
Shared networking between accounts, such as a Shared VPC, VPC peering, or a Transit Gateway.
Ability to create or add accounts such that the new account will automatically be configured this way.
This initial stage is about getting the accounts to “look the same”, but even with only a handful of accounts, we see a need for differentiation. Companies need to have different owners for accounts in order to segment access and receive alerts about issues in those separate accounts.
This differentiation continues further and may involve:
Defining different cost alarms between accounts, such that a developer’s personal sandbox account should be costing less than $100/mo.
Defining different guardrails for accounts. Some accounts will have exceptions. For example, vendors that still require IMDSv1 or IAM users and therefore can’t be prevented by SCP. Some accounts will have different requirements, such as production environments that are more restrictive than sandbox environments.
Defining different networking patterns such that production accounts may share one network while dev environments may share a different network.
Some of these elements can be addressed through placement within an AWS Organization OU structure (such as SCPs), but this requires building into the automation to avoid manual steps.
You need to ensure that you can incorporate accounts that were not created through the preferred flow. These may be accounts from an acquisition or discovered shadow IT accounts. Sometimes the use case for an account will also change and so the system needs to be flexible for that need as well. As you start having many accounts, the need to delete accounts will also arise.
Moving beyond the Landing Zone
As you move beyond dozens of accounts, you must consider how they are organized. Some good talks related to this topic include the re:Invent 2022 talk from Netflix titled Reimagining multi-account deployments for security and speed and the 2023 CloudSecNext talk from Hashicorp titled Failing to Scale: Bumps in the Road While Scaling Cloud Access.
Both of these talks discuss the strategies involved when the number of accounts grows to a large amount. The Hashicorp talk, for example, discusses the number of accounts growing up to and beyond 20,000. AWS has also created the whitepaper Organizing Your AWS Environment Using Multiple Accounts.
As accounts are automatically created, the new owners of these accounts will need to do a lot of work to set up their ability to use the accounts. This is where an account vending strategy begins: to perform the additional setup and integrations.
An account vending strategy should automatically:
Create a Github repo related to the account and include a skeleton terraform or other Infrastructure as Code (IaC) template.
Setup a Github Action or similar functionality that automatically scans the IaC for issues and deploys the changes to the AWS account.
Setup alerts (ex. Slack) to the account owners for security alerts or other issues.
By automatically making it easy to use IaC, this becomes encouraged (as opposed to ClickOps in the web browser). You can also encourage segmentation of dev/staging/prod by automatically creating 3 AWS accounts for every request for a new account.
In addition to generating the skeleton of this infrastructure, you should also consider other common infrastructure needs. This may mean developing a library of modules for common infrastructure resources, such as easily deploying an EC2 that is pre-configured with all the preferred settings. This concept is generally known as developing “paved roads” - predefined infrastructure templates that make it easier for teams to deploy secure and compliant cloud resources quickly. Applications will need to send logs and metrics somewhere, and those will need to be monitored, so you may need to additionally set up those for each AWS account.
As the company’s infrastructure capabilities mature, the account segmentation and account organization begin defining other rules for the company that tend to exist as assumptions or beliefs held by the security team. But these should be enforced through tooling. For example:
Production and Development environment should not communicate with each other or have trust relationships between each other (such as an S3 bucket in a Production account allowing access from a Development account). Similarly, there may be expectations that different business units be siloed from each other.
Development accounts should not contain PII.
In combining all these things together, the account vending solution should involve gathering information from the requestor about the purpose of the requested account, the owners, etc. It will then use that information to automatically setup not only the AWS account, but all the related systems.
A key understanding of an account vending strategy is that it delivers the ecosystem of requirements, not just an AWS account or the things that live inside AWS. To put this another way, a Landing Zone is usually about what is in the AWS account, but an account vending strategy is more focused on what needs to be done outside of the account.
