What is Kubernetes multi-tenancy?
Sharing Kubernetes clusters across multiple teams and customers is a common practice known as multi-tenancy. It enables organizations to maximize the use of their hardware and infrastructure investments by allowing different teams or customers to share the same cluster resources while maintaining isolation from each other through namespaces. It's a popular approach, especially in larger organizations and cloud environments, due to its cost-effectiveness and efficient use of resources.
As the number of Kubernetes users in an organization grows, the need for multi-tenant solutions increases as well. There are two ways to handle multi-tenancy securely: by provisioning dedicated clusters for each team, or by using control plane isolation to manage shared access to the same cluster. In the latter case, Kubernetes offers three mechanisms to provide control plane isolation (an added layer of security that keeps control planes separate for each tenant): namespaces, RBAC, and quotas. Namespace isolation is a driving factor.
Namespace isolation is key in achieving control plane isolation for several reasons. First, it enhances security by segregating the workloads of different tenants, preventing one tenant from accessing or affecting another's resources. Namespace isolation also enables effective resource management, as resources can be allocated and limited for each namespace. This ensures independent management and scheduling of resources for each tenant. Finally, namespace isolation provides each tenant with the autonomy to design their own policies, configurations, and settings.
The challenge in relying on namespaces for isolation
Namespace isolation enables organizations to use multi-tenant clusters in various scenarios. For example: two teams may share the same cluster with access to workloads that have varying degrees of sensitivity; or companies might run SaaS services that allocate containers or pods for every customer.
However, there is no native mechanism to monitor the logical crossings of namespaces. There is also no way to detect the attack paths or vectors of potential violations. With no native way to monitor namespace boundaries, it becomes difficult to enforce security and ensure isolation among different tenants. This can lead to potential security risks, such as one tenant inadvertently or maliciously accessing another tenant's resources.
Using NamespaceHound
This is where NamespaceHound comes in. Cluster operators can use NamespaceHound to assess the risk of cross-tenant violations in their environment.
NamespaceHound is a tool for detecting the risk of potential namespace crossing violations in multi-tenant clusters. Given a cluster, NamespaceHound will run an analysis to determine all possible ways to cross security boundaries between namespaces. In addition to revealing cross-namespace issues, it inspects the cluster configuration for anonymous access opportunities. If given a specific namespace (via the -n namespace parameter), it will focus on this namespace plus anonymous access to find all possible ways to reach or interfere with resources in another namespace.
Another instance where NamespaceHound is useful is in helping red-teamers and security researchers to find lateral movement paths once they are past the point of initial access into the cluster. Our 2023 Kubernetes Security Report revealed that with successful initial access, the opportunities for lateral movement and privilege escalation are abundant.
NamespaceHound is a Python tool that you can run from a CLI. The available options are as follows:
>python3 nshound.py -h
usage: nshound.py [-h] [--kubeconfig KUBECONFIG] [-n NAMESPACE | -c] [-o {table,json,csv,html}] [-v]
options:
-h, --help show this help message and exit
--kubeconfig KUBECONFIG
.kubeconfig file containing the cluster access credentials
-n NAMESPACE, --namespace NAMESPACE
look for escape paths from this namespace only
-c, --controlplane show issues from kube-system
-o {table,json,csv,html}, --output {table,json,csv,html}
output format - json, csv or table
-v, --verbose increase output verbosity Security and Privacy
To function properly, NamespaceHound requires read permissions on all resource types. Of course, principals mapped to admin or cluster-admin roles, and system:masters groups will work as well.
NamespaceHound does not save any data about the target cluster locally. It does not build graphs and does not save object material. Upon every run, NamespaceHound establishes a new connection with the cluster and performs API server queries in the same capacity.
Sample namespace crossings
NamespaceHound can detect a wide scope of namespace crossings. Two examples are included below. (For the complete list of 22 namespace crossings, see the Github repository.)
| Name | Severity | Confidence | Description | Method |
|---|---|---|---|---|
| RBAC_SECRETS_STEALING | High | High | A service account from this namespace has access to secrets in another namespace. | Querying RBAC API |
| RBAC_WORKLOAD_CREATION | High | High | A service account from this namespace can create workloads in another namespace. | Querying RBAC API |
Extending the PEACH framework to Kubernetes
How can cluster operators ensure the effectiveness of namespace security boundaries? The strength of a security boundary is limited by the weakest link in its design or implementation. Wiz designed a framework, PEACH, for modeling and improving SaaS and PaaS tenant isolation by managing the attack surface exposed by user interfaces. The framework previously targeted general cloud environments, but it has been extended to apply to Kubernetes multi-tenancy.
Determining security boundaries in a K8s cluster
In a multi-tenant cluster, tenants are typically exposed via a means of data plane access, such as load balancer services or application gateways. The interface functionality, however, is not different from other cloud and application interfaces. You can employ similar security techniques for exposed Kubernetes services.
Hardware Separation
In some clusters, namespace-level separation can be augmented with node affinity. The scheduling algorithm will follow the designed affinity model to only schedule workloads belonging to certain tenants to their assigned K8s worker node. (Of course, this will impose a performance hit in terms of the number of extra nodes needed to support the load.)
Containerization
Instances belonging to different tenants run in separate containers running on shared hardware (or nested within a shared virtual machine) via operating-system-level virtualization.
Data Segmentation
Control plane configuration is stored in ETCD and is therefore shared among tenants, even if it is encrypted. Data plane resources are typically bound to a namespace; however, certain types of resources are global and require per-resource review. K8s RBAC is granular enough to ensure service account access per resource name.
Network Segmentation
Instances belonging to different tenants run on separate (virtual or physical) machines in the same network, which is segmented into single tenant VPNs. The use of network policies between namespaces is encouraged.
Identity Segmentation
Identity segmentation is present by default. K8s service accounts are bound to a namespace.
Measuring hardening factors in a K8s cluster
When measuring the effectiveness of namespace security boundaries, we can use the PEACH framework’s 5 hardening (or enforcing) factors:
Privilege Hardening
Kubernetes RBAC (role-based access control) is a method of regulating access to cluster resources based on the roles of individual principals within the cluster. RBAC should be considered a strong security boundary only if the following guidelines are observed:
There are two types of roles in K8s RBAC: namespace-level
Roleand cluster-levelClusterRole. Tenant principals must be assigned only Roles that are not global admins.It is important to follow the principle of least privileges when assigning roles to principals. Assign only necessary verb access to specific resources and never assign * to verbs or resources.
Be careful when assigning privileges that allow host access to tenant principals (for example,
/pod:create). Unless a pod is sandboxed, the host is a shared medium between the namespaces and can be used to access the data and secrets of other namespaces.Use Admission Controller to enforce out-of-line behaviors, such as running privileged pods, starting pods running as root, or starting pods with the default service account.
Encryption hardening
Some Kubernetes resources are globally scoped (Webhooks, for example), and thus may be shared across tenants. Sensitive data or workflows should not use global resources. If they do, it's best to employ additional authentication via additional RBAC dedicated ClusterRoles.
PersistentVolume and PersistentVolumeClaim are K8s objects frequently used for managing custom storage access in the cluster. Official Kubernetes documentation on multi-tenancy suggests configuring a separate StorageClass for each tenant and using it to strengthen isolation. If a StorageClass is shared, you should set a reclaim policy of "Delete" to ensure that a PersistentVolume cannot be reused across different namespaces.
If you are using external secret storage, be sure to isolate the storage access per tenant.
Authentication Hardening
Consider the need for tenant communication with an API server. If not needed, opt out of automounting service account tokens into the pods. Do this either on the pod level (good) or kubelet level (better) by using the automountServiceAccountToken: false option.
Connectivity Hardening
By default, any pod within the cluster can communicate with any other pod. Cross-tenant and tenant egress connectivity should be blocked by Kubernetes Network Policies. It is advisable to use namespace labels for Network Policies as a more stable labeling mechanism over pod labels.
NamespaceHound functionality in Wiz
In addition to running NamespaceHound from a CLI, you can also access its functionality from the Forensics section of Kubernetes namespace objects in the Wiz portal. The Forensics tab shows the potential impact of an object being compromised. This is particularly suited to show NamespaceHound-type results:
From this page, you can view the blast radius of a particular namespace and explore specific namespace crossing opportunities.
Hopefully this overview of NamespaceHound has given you some useful insight into security practices for multi-tenant Kubernetes clusters. For more information on Kubernetes security strategies, read about our key takeaways from the 2023 Kubernetes Security report.
