Amazon EKS add-ons: Custom and advanced configuration with Terraform
Configuring advanced configuration for EKS addons using Terraform and AWS provider.
Introduction
EKS addons have been available to us for some time and I have talked about them in my previous post here. They evolved over time and we saw in October 2022 that Amazon EKS add-ons had the ability to preserve configurations. This was an initial step towards enabling us to safely modify the configuration of Kubernetes applications.
We finally reached the point where we can configure values for each individual addon. We can now provide configuration directly through the Amazon EKS add-ons API, to install and configure their operational software during cluster creation in a single step. This enables me to leverage my existing Infrastructure as Code (IaC) Terraform to configure Amazon EKS vended operational software and meet performance, compliance, or additional requirements not handled by default configuration settings.
Support for advanced configuration through the Amazon EKS add-ons API is available for the following add-ons:
Feature Overview
A new configurationValues parameter has been added to the Amazon EKS add-ons API. At the moment it accepts configuration as a JSON blob, with support for YAML coming soon. The JSON blob needs to conform to the underlying add-on configuration JSON schema, which is now available through the new Amazon EKS DescribeAddonConfiguration API.
Terraform implementation
In my Terraform implementation, I am only going to test the new functionality with Core DNS component. It is the one that was always problematic for me as I needed to add additional configuration.
Important! Values that you add to your JSON configuration file need to be supported by the helm chart. Mainly because Helm is used under the hood of addons. In the case of core-dns, you can see all available values in https://github.com/coredns/helm/blob/master/charts/coredns/values.yaml
core-dns.json file content
{
"replicaCount": 3,
"resources":
{
"limits":
{
"cpu": "100m",
"memory": "200Mi"
},
"requests":
{
"cpu": "100m",
"memory": "200Mi"
}
}
}locals {
core_dns_config = file("${path.module}/configs/core-dns.json")
}
resource "aws_eks_addon" "core_dns" {
count = var.eks_addon_version_core_dns != null ? 1 : 0
cluster_name = aws_eks_cluster.cluster.name
addon_name = "coredns"
addon_version = var.eks_addon_version_core_dns
resolve_conflicts = "OVERWRITE"
configuration_values = local.core_dns_config
preserve = true
tags = {
"eks_addon" = "coredns"
}
}The above terraform configuration works fine and it was easy and fast to deploy.
As seen above I am using locals to source the content of the file. Interestingly, the above works as the following operation happen during terraform plan:
# aws_eks_addon.core_dns[0] will be updated in-place
~ resource "aws_eks_addon" "core_dns" {
+ configuration_values = jsonencode(
{
+ replicaCount = 3
+ resources = {
+ limits = {
+ cpu = "100m"
+ memory = "200Mi"
}
+ requests = {
+ cpu = "100m"
+ memory = "200Mi"
}
}
}
)
id = "test-eu-dev:coredns"
tags = {
"eks_addon" = "coredns"
}
# (9 unchanged attributes hidden)
}So you can see that contents of the file are inserted into jsonencode(). This configuration works as expected.
To my surprise when I tried doing the same inside locals that is:
locals {
core_dns_config = jsonencode(file("${path.module}/configs/core-dns.json"))
}It didn’t work. The above local configuration doesn’t work. So make sure that you don’t perform jsonencode inside your locals. For some reason, it behaves differently from the function that is run inside eks_addon resource. Perhaps I should be passing a decoded json so that double encoding is not happening.
topologySpreadConstraints CoreDNS
I have tried adding a topology spread argument for CoreDNS because Helm chart does support it.
"topologySpreadConstraints":
[
{
"maxSkew": 1,
"topologyKey": "topology.kubernetes.io/zone",
"whenUnsatisfiable": "ScheduleAnyway",
"labelSelector":
{
"matchLabels":
{
"eks.amazonaws.com/component": "coredns"
}
}
}
]However, it looks like EKS addons are limited to certain values that you can use. In this case I received following error:
aws_eks_addon.core_dns[0]: Modifying... [id=test-eu-dev:coredns]
╷
│ Error: error updating EKS Add-On (test-eu-dev:coredns): InvalidParameterException: ConfigurationValue provided in request is not supported [$.topologySpreadConstraints: is not defined in the schema and the schema does not allow additional properties]
│ {
│ RespMetadata: {
│ StatusCode: 400,
│ RequestID: "5268587a-61b6-49bb-9b08-cc015d57068e"
│ },
│ AddonName: "coredns",
│ ClusterName: "test-eu-dev",
│ Message_: "ConfigurationValue provided in request is not supported [$.topologySpreadConstraints: is not defined in the schema and the schema does not allow additional properties]"
│ }
│
│ with aws_eks_addon.core_dns[0],
│ on eks-addons.tf line 20, in resource "aws_eks_addon" "core_dns":
│ 20: resource "aws_eks_addon" "core_dns" {Conclusion
The long-awaited feature is finally here. Sadly, for now, we can only use JSON but Yaml support is coming soon. Stay tuned!
If you are running Amazon EKS and you haven’t tried add-ons yet, now is the time to do it as it finally supports everything you need! You can now discover additional settings that are underpinned by the Helm charts that are used to manage add-ons in EKS clusters. This provides for more granular configurations, a more flexible user experience, and better fits new and existing DevOps automation.
Enjoy Kubernetes!!!
Sponsor Me
Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.
Thanks for reading everybody. Marcin Cuber
Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.
Thanks for reading everybody. Marcin Cuber
