AWS EKS and kube-bench

Story details how to run kube-bench job against EKS cluster

Overview

Kube-bench is an open source project written in Go. This application checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark. Entire project is available in github. Tests which are executed are configured with YAML files, and this makes kube-bench easy to update as test specifications evolve.

Security should be a a critical component of configuring and maintaining Kubernetes clusters and applications in any company. Amazon EKS provides secure, managed Kubernetes clusters by default, but you still need to ensure that you configure the nodes and applications you run as part of the cluster to ensure a secure implementation.

In this story, I will provide an easy way to run kube-bench tests against EKS cluster and I will show how to assess the Amazon EKS cluster nodes you have created against the CIS EKS Kubernetes benchmark.

CIS Kubernetes Benchmark

Since Amazon EKS provides a managed control plane, not all of the recommendations from the CIS Kubernetes Benchmark are applicable as customers are not responsible for configuring or managing the control plane.

CIS Amazon EKS Benchmark v1.0.0 provides guidance for node security configurations for Kubernetes and aligns with CIS Kubernetes Benchmark v1.5.1.

As noted in the project there is not a one-to-one mapping between releases of Kubernetes and releases of the CIS benchmark. See CIS Kubernetes Benchmark support to see which releases of Kubernetes are covered by different releases of the benchmark.

As mentioned already, it is impossible to inspect the master nodes of managed clusters, e.g. GKE, EKS and AKS, using kube-bench as one does not have access to such nodes, although it is still possible to use kube-bench to check worker node configuration in these environments and this is exactly what we are going to do.

Implementation

Note that in my work I am using EKS 1.18 and if you are using older version findings might be different. However, at the end of the day you want to find out about findings so the same method will work for all EKS clusters.

We are not going to install it directly on the nodes as it is simply waste of time. We will deploy it using yaml templates by utilising kubernetes jobs.

Standard mode

Execution against worker nodes only:

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      containers:
        - name: kube-bench
          image: aquasec/kube-bench:latest
          command: ["kube-bench", "node", "--benchmark", "eks-1.0"]
          volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
        - name: etc-systemd
          hostPath:
            path: "/etc/systemd"
        - name: etc-kubernetes
          hostPath:
            path: "/etc/kubernetes"

To apply simply save the file and run:

kubectl apply -f job-eks.yaml

Debug mode

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-debug
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      containers:
        - name: kube-bench
          image: aquasec/kube-bench:latest
          command: ["kube-bench", "-v", "3", "--logtostderr", "--benchmark", "eks-1.0"]
          volumeMounts:
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
        - name: etc-systemd
          hostPath:
            path: "/etc/systemd"
        - name: etc-kubernetes
          hostPath:
            path: "/etc/kubernetes"

To apply simply save the file and run:

kubectl apply -f job-debug-eks.yaml

Viewing results

Like with any other kubernetes job, it will create a pod and the status of that pod will be completed (after successful execution of tests).

In order to view the results, you can easily view the pod’s logs.

Cleanup

kubectl delete -f job-eks.yaml
# or
kubectl delete -f job-debug-eks.yaml

Conclusion

In this story I have presented the simplest way to run kube-bench tests against EKS worker nodes. Hope this helps someone quickly find various security issues.

As a bonus, if you are using AWS Security Hub you can send kube-bench findings directly to your console. See the article -> https://aws.amazon.com/about-aws/whats-new/2020/12/aws-security-hub-adds-open-source-tool-integration-with-kube-bench-and-cloud-custodian/

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber