Amazon EKS and Security Groups for Pods

Consideration and configuration details to enable Security groups for pods in Kubernetes cluster

Overview

In this story I want to focus on a recently released feature called Security Groups for pods.

Pods are the smallest deployable units of computing that you can create and manage in Kubernetes. A pod is a group of one or more containers, with shared storage/network resources, and a specification for how to run the containers. On the other side we have AWS Security groups (SG). A security group acts as a virtual firewall for your instances to control inbound and outbound traffic. In our case, pod is also considered as an instance. Normally, when you launch an instance in a VPC, you can assign up to five security groups to the instance. Security groups act at the instance level, not the subnet level. Therefore, each instance in a subnet in your VPC can be assigned to a different set of security groups. With this new feature for EKS, we are now in a position to attach SGs to pods which are running inside Kubernetes cluster.

Context + Details

Containerised applications running in Kubernetes frequently require access to other services running within the cluster as well as external AWS services, such as Amazon RDS or Amazon Elasticache Redis. On AWS, controlling network level access between services is often accomplished via EC2 security groups. Before today, you could only assign security groups at the node level, and every pod on a node shared the same security groups.

Security groups for pods make it easy to achieve network security compliance by running applications with varying network security requirements on shared compute resources. Network security rules that span pod to pod and pod to external AWS service traffic can be defined in a single place with EC2 security groups, and applied to applications with Kubernetes native APIs.

Until Security Groups for pods feature, we had following mechanisms to configure access to/from pods;

• NetworkPolicies (Calico)
• Service resource
• Ingress resource
• VirtualService and Gateway (Istio)
• Load balancer controller

There might be some other ways to allow ingress/egress rules that I have missed or never used before. This is already a good selection of tools and resources so I don’t fully understand why you would need SGs for pods. But, we have it :).

Not a solution

Allowing for SGs to be associated with pods is meant to solve one problem which whitelisting. You can whitelist a particular SG as an ingress rule in another SG in order to access resources such as RDS or ElastiCache. However, the problem really sits in the design or architecture of the system. VPC that runs your EKS shouldn’t be the place where you have all your RDS clusters or Redis clusters, this simply isn’t great. Although you are using Kubernetes to share resources such as memory or CPU, you shouldn’t share the same virtual network for all applications’ dependencies.

Therefore, you still need to have multiple VPCs and so make use of VPC peering and/or Transit Gateway. So, it doesn’t solve major connectivity problems that I find huge limitations in first place when working with containers.

Setup details

Going back to feature implementation, here are the details of my setup;

• EKS 1.18.9, version platform 2
• AWS VPC CNI version 1.7.5
• Worker Nodes AMI ID: ami-0584b5127af4da5b0

All EKS worker nodes are running in private subnets and route out through NAT Gateway.

Major Considerations and requirements

List of important aspects around SGs for pods

• Amazon EKS cluster with version 1.17 with platform version eks.3 or later.
• Traffic flow to and from pods with associated security groups are not subjected to Calico network policy enforcement and are limited to Amazon EC2 security group enforcement only. This will likely be fixed in the future release of Calico.
• Source NAT is disabled for outbound traffic from pods with assigned SGs so that outbound SG rules are applied. So pods with assigned SGs must be launched on nodes that are deployed in a private subnet configured with a NAT gateway or instance. Pods with assigned SGs deployed to public subnets are not able to access the internet.
• Kubernetes services of type NodePort and LoadBalancer using instance targets with an externalTrafficPolicy set to Local are not supported with pods that you assign SGs to.
• If you’re also using pod security policies to restrict access to pod mutation, then the eks-vpc-resource-controller and vpc-resource-controller Kubernetes service accounts must be specified in the Kubernetes ClusterRoleBinding for the the Role that your psp is assigned to. Example subjects block configuration

subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
  - kind: ServiceAccount
    name: vpc-resource-controller
  - kind: ServiceAccount
    name: eks-vpc-resource-controller

Implementation

Cluster IAM permissions

IAM policies associated with IAM role attached to EKS cluster need to have the following managed policy included: arn:aws:iam::aws:policy/AmazonEKSVPCResourceController

Using AWS CLI:

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController --role-name <EKS IAM Cluster Role>

Using Terraform:

resource "aws_iam_role_policy_attachment" "policyResourceController" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.cluster.name
}

VPC CNI Plugin

Official code for can be found in github repo. In this section I want to point out three important configurations which are highlighted in the code snipped below.

"apiVersion": "apps/v1"
"kind": "DaemonSet"
"metadata":
  "labels":
    "k8s-app": "aws-node"
  "name": "aws-node"
  "namespace": "kube-system"
"spec":
  "selector":
    "matchLabels":
      "k8s-app": "aws-node"
  "template":
    "metadata":
      "labels":
        "k8s-app": "aws-node"
    "spec":
      "containers":
      - "env":
        - "name": "ADDITIONAL_ENI_TAGS"
          "value": "{}"
        - "name": "AWS_VPC_CNI_NODE_PORT_SUPPORT"
          "value": "true"
        - "name": "AWS_VPC_ENI_MTU"
          "value": "9001"
        - "name": "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER"
          "value": "false"
        - "name": "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG"
          "value": "false"
        - "name": "AWS_VPC_K8S_CNI_EXTERNALSNAT"
          "value": "true"
        - "name": "AWS_VPC_K8S_CNI_LOGLEVEL"
          "value": "DEBUG"
        - "name": "AWS_VPC_K8S_CNI_LOG_FILE"
          "value": "/host/var/log/aws-routed-eni/ipamd.log"
        - "name": "AWS_VPC_K8S_CNI_RANDOMIZESNAT"
          "value": "prng"
        - "name": "AWS_VPC_K8S_CNI_VETHPREFIX"
          "value": "eni"
        - "name": "AWS_VPC_K8S_PLUGIN_LOG_FILE"
          "value": "/var/log/aws-routed-eni/plugin.log"
        - "name": "AWS_VPC_K8S_PLUGIN_LOG_LEVEL"
          "value": "DEBUG"
        - "name": "DISABLE_INTROSPECTION"
          "value": "false"
        - "name": "DISABLE_METRICS"
          "value": "false"
        - "name": "ENABLE_POD_ENI"
          "value": "true"
        - "name": "MY_NODE_NAME"
          "valueFrom":
            "fieldRef":
              "fieldPath": "spec.nodeName"
        - "name": "WARM_ENI_TARGET"
          "value": "1"
        "image": "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni:v1.7.5"

1. You require at least version 1.7.1 of CNI plugin
2. AWS_VPC_K8S_CNI_EXTERNALSNAT Important to specify this parameter as it indicates whether an external NAT gateway should be used to provide SNAT of secondary ENI IP addresses. If set to true, the SNAT iptables rule and off-VPC IP rule are not applied, and these rules are removed if they have already been applied.
3. Setting ENABLE_POD_ENI to true will add the vpc.amazonaws.com/has-trunk-attached label to the node if it is possible to attach an additional ENI. This is an essential configuration required to enable SGs for pods.

The above yaml snippet works fine, however if you need an option to do it with kubectl then run the following:

kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true

Important to note that I have came across two issues during this process. First problem was related to the upgrade of VPC CNI plugin. When I trying upgrading the plugin to latest version 1.7.5, aws-node pods got stuck in terminating state. Stuck pods have to be force deleted. Second issue or maybe intended behaviour was that vpc.amazonaws.com/has-trunk-attached label was set to false across all nodes. In order for nodes to have that label set to true, I had to rotate all nodes; effectively bringing up new nodes. In bigger clusters this can be time consuming task.

You can see which of your nodes have aws-k8s-trunk-eni set to true with the following command:

kubectl get nodes -o wide -l vpc.amazonaws.com/has-trunk-attached=true

Optionally, if are you using liveness or readiness probes, you need to disable TCP early demux, so that the kubelet can connect to pods on branch network interfaces via TCP. To disable TCP early demux:

"initContainers":
- "env":
  - "name": "DISABLE_TCP_EARLY_DEMUX"
    "value": "true"
  "image": "602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni-init:v1.7.5"

You can find full yaml configuration in my github eks repo here.

Examples

PodSelector

This example illustrates usage of PodSelector for SecurityGroupPolicy which will match against pods that have app label set to backend

apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: sg-service-account-policy
  namespace: default
spec:
  podSelector:
    matchLabels: 
      app: backend
  securityGroups:
    groupIds:
      - sg-123456789

Example deployment yaml which will spin up a single pod and will get a correct security group attached:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-deployment
  namespace: default
  labels:
    app: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: test-container
        image: image-id
        ports:
        - containerPort: 80

serviceAccountSelector

This example illustrates usage of serviceAccountSelector for SecurityGroupPolicy which will match service accounts that have app label set to backend

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sg-service-account
  namespace: default
  labels:
    app: backend
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: sg-service-account-policy
  namespace: default
spec:
  serviceAccountSelector: 
    matchLabels: 
      app: backend
  securityGroups:
    groupIds: 
      - sg-123456789

and finally pod definition will look as follows:

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
spec:
  serviceAccountName: sg-service-account
  containers:
  - name: test-container
    image: image-id

Important notes:

• The security groups referenced in the SecurityGroupPolicy resource must exist. If they don’t exist, then, when you deploy a pod that matches the selector, your pod remains stuck in the creation process. If you describe the pod, you’ll see an error message similar to the following one: An error occurred (InvalidSecurityGroupID.NotFound) when calling the CreateNetworkInterface operation: The securityGroup ID 'sg-123456788' does not exist.
• The security group must allow inbound communication from the cluster security group (for kubelet) over any ports you've configured probes for.
• The security group must allow outbound communication to the cluster security group (for CoreDNS) over TCP and UDP port 53. The cluster security group must also allow inbound TCP and UDP port 53 communication from all security groups associated to pods.

Conclusion

This new feature is definitely a step forward and will help many engineers in developing their containerised apps. But we all sit in engineering world and there are many things to consider when it comes to running a secure Kubernetes cluster. Every company has their own security and compliance policies, some of which are tightly coupled to security groups. I did find it very easy to configure my clusters to use SGs for pods and I don’t believe any real engineer will struggle with it. However, this is yet another Kubernetes resource which further expands and effectively complicates various configurations.

I hope this article will help people move forward quicker with their development tasks. Enjoy your Kubernetes.

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber