Amazon EKS add-ons implemented with Terraform

5 min readJun 24, 2021

True story behind AWS EKS (Kubernetes) add-ons; VPC CNI plugin, Kube-proxy and CoreDNS

Introduction

Reading this story, you should probably know by now what is AWS EKS. EKS is a managed Kubernetes service were Amazon Web Services is responsible for the entire control-plane. Amazon EKS is a fully managed service that makes it easy for you to run Kubernetes on AWS without needing to be an expert in managing Kubernetes clusters.

The feature we are going to concentrate on today is EKS add-ons. AWS guys are claiming that this is a major step forward to provide fully managed Kubernetes clusters. EKS add-ons allows you to configure, deploy, and update the operational software, or add-ons, that provide key functionality to support your Kubernetes applications. These add-ons include critical tools for cluster networking like the Amazon VPC CNI, as well as operational software for observability, management, scaling, and security. At the time of writing this story, following add-ons are supported:

Amazon VPC CNI plugin and kube-proxy, Amazon EKS now allows you to enable add-ons when you create a new cluster or at any time after the cluster is running. EKS will start the add-on software on the cluster and allow you to deploy new versions of the add-on with a single command. Every add-on includes the latest security patches and bug fixes, and is validated by AWS to work with Amazon EKS. This reduces the amount of work you need to do in order to start, manage, and upgrade production-ready Kubernetes clusters, which helps to keep your clusters stable and secure.

Intro: even more

AWS mentions in their documents that we can update specific Amazon EKS managed configuration fields through the Amazon EKS API. Additionally, we can modify configuration fields not managed by Amazon EKS directly within the Kubernetes cluster once the add-on starts. This includes creating or modifying custom resource definitions or ConfigMaps to define specific configuration fields for an add-on where applicable. This is made possible using the Kubernetes server side apply feature and the fields managed and documented by Amazon EKS. For more information, see Kubernetes 1.18 Feature Server-side Apply Beta 2 in the Kubernetes documentation.

Important thing to note here is that AWS mentioned that “These changes are not overridden by Amazon EKS once they are made.” I actually disagree, if you modify anything deployed through EKS Add-on including ConfigMaps, EKS will overwrite your changes over time. I have came across this issue couple of times and I will talk about those problem later on so keep reading :).

Implementation

In this section I will only cover implementations for kube-proxy and coreDNS. If you want to implement VPC CNI plugin, please for the following guide -> https://aws.amazon.com/blogs/containers/introducing-amazon-eks-add-ons/

Considerations

To configure add-ons for the cluster your IAM user must have administrative privileges within the cluster. For more information, see Cluster authentication.
Amazon EKS add-ons are only available with Amazon EKS clusters running Kubernetes version 1.18 and later.
Amazon EKS add-ons run on the nodes that you provision or configure for your cluster. Node types include Amazon EC2 instances and Fargate.

Kube-proxy managed add-on using Terraform

Terraform resource configuration

resource "aws_eks_addon" "kube_proxy" {
  addon_name        = "kube-proxy"
  addon_version     = "v1.20.4-eksbuild.2"
  resolve_conflicts = "OVERWRITE"  
  tags = merge(
    var.tags,
    {
      "eks_addon" = "kube-proxy"
    }
  )
}

Issues:

After implementing kube-proxy with Terraform I immediately hit issues such as:

│ Error: unexpected EKS add-on (eks-test-eu:kube-proxy) state returned during creation: creation not successful (CREATE_FAILED): Errors:
│ Error 1: Code: ConfigurationConflict / Message: Apply failed with 1 conflict: conflict with "before-first-apply" using v1: .data.config

and

Error: unexpected EKS add-on (eks-test-eu:kube-proxy) state returned during creation: creation not successful (CREATE_FAILED): Errors:
│ Error 1: Code: AccessDenied / Message: clusterrolebindings.rbac.authorization.k8s.io "eks:kube-proxy" is forbidden: user "eks:addon-manager" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
│ {APIGroups:["discovery.k8s.io"], Resources:["endpointslices"], Verbs:["get"]}

Note that this happened to me in both Ireland (eu-west-1) and China Beijing (cn-north-1) regions.

Fix:

Add missing permissions to eks:addon-manager cluster role:

kubectl edit clusterrole eks:addon-manager

and make sure you have the following permissions added:

apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
  - get

CoreDNS managed add-on using Terraform

Terraform resource configuration

resource "aws_eks_addon" "core_dns" {
  addon_name        = "coredns"
  addon_version     = "v1.8.3-eksbuild.1"
  resolve_conflicts = "OVERWRITE"
  tags = merge(
    var.tags,
    {
      "eks_addon" = "coredns"
    }
  )
}

Issues:

Error: unexpected EKS add-on (eks-test-eu:coredns) state returned during creation: timeout while waiting for state to become 'ACTIVE, CREATE_FAILED' (last state: 'CREATING', timeout: 20m0s)
│ [WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration

Fix:

The deployment takes long time, around 20 mins so just be patient. It will fail first time around but CoreDNS service will work as expected inside the cluster.

The above error has shown however second terraform apply worked as expected so no major issues here.

Issues

Some issues related specifically to certain EKS add-ons have been already been mentioned. However, there are much more complex issues that can’t be solved yet. The issues that I am referring to are related to the fact that you can’t modify any of the settings inside add-ons.

If you decide to edit for example a configmap in CoreDNS setting to support custom proxy then EKS add-on deployment will overwrite that change rather quickly.

If you interested, there is a lot more info from various developers facing problems with it https://github.com/aws/containers-roadmap/issues/1159

In general, if you are trying to make any changes to coredns or kube-proxy or even VPC CNI then don’t use EKS add-ons at this early stage. You simply won’t be able to customise these Kubernetes operators/applications.

Conclusions

I have to say that from the start of the deployment of the first add-on, there were issues. Like with many AWS products at launch, I don’t believe this is production ready. However, it works well if you fix all the issues and don’t have to make any customisations. That said, you do require good Kubernetes RBAC knowledge to spot and quickly fix permission issues.

If you are interested in the entire terraform setup for EKS, you can find it on my GitHub -> https://github.com/marcincuber/eks/tree/master/terraform-aws

Enjoy Kubernetes!!!

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Hi guys, I am Marcin and I am Technical Lead specialising in the field of DevOps. I am also a certified AWS Solutions…

github.com

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber