Amazon EKS Upgrade Journey From 1.30 to 1.31- say hello to another cutee “Elli”

8 min read1 hour ago

We are now welcoming “Elli” release. Process and considerations while upgrading EKS control-plane to version 1.31.

Overview- A Glimpse into the Future with Kubernetes 1.31 and the Elli Theme

Amazon EKS (Elastic Kubernetes Service) continues to evolve, bringing users the cutting-edge features of Kubernetes while maintaining the robustness and flexibility required for running production workloads. With each upgrade, users gain access to improvements, optimisations, and new features, and the release of EKS 1.31 is no exception. As Amazon EKS keeps pace with upstream Kubernetes versions, the upgrade process involves not just new functionality but also potential breaking changes that must be handled with care.

In this story, we embark on the journey of upgrading Amazon EKS from version 1.30 to 1.31, building on the experience from upgrading to 1.30 and introducing the key features of Kubernetes 1.31. A significant part of this journey includes embracing the Kubernetes 1.31 release theme — Elli, named after the ancient Norse mythological figure Elli, who personifies old age and endurance. This theme aligns with the stability, maturity, and resilience that the Kubernetes project has achieved over the past 10 years.

Previous Stories and Upgrades

If you are looking at

upgrading EKS from 1.29 to 1.30 check out this story
upgrading EKS from 1.28 to 1.29 check out this story
upgrading EKS from 1.27 to 1.28 check out this story
upgrading EKS from 1.26 to 1.27 check out this story
upgrading EKS from 1.25 to 1.26 check out this story
upgrading EKS from 1.24 to 1.25 check out this story

From 1.30 to 1.31: Why Upgrade?

The decision to upgrade from EKS 1.30 to 1.31 is driven by several factors. Kubernetes 1.31 introduces new features, deprecates old APIs, and addresses security vulnerabilities and performance optimisations. The release aims to improve the user experience, both in terms of operational ease and resource management. Additionally, running a version that is supported by Amazon EKS ensures continued support, access to patches, and compatibility with EKS-managed add-ons.

Notable Changes from EKS 1.29 to 1.30 Recap

In the recent upgrade to EKS 1.30, several key changes were introduced, including the default graduation of sidecar containers and improvements in ephemeral volumes. These changes allowed users to better manage their workloads and improve storage performance, signalling Kubernetes’ maturity in managing production-grade deployments. But upgrading to 1.30 was also a necessary stepping stone to 1.31, as it laid the groundwork for future enhancements, including those tied to security, observability, and workload management.

What’s New in Kubernetes 1.31? (Elli)

Kubernetes 1.31, with its Elli release theme, brings a combination of incremental improvements and significant changes across several areas, from security to storage to usability. Following 11 enhancements have been promoted to Stable:

- Important Kubelet flag Removal

The kubelet flag --keep-terminated-pod-volumes deprecated since 2017 has been removed as part of the v1.31 release. This change impacts how terminated pod volumes are handled by the kubelet. If you are using this flag in your node configurations, you must update your bootstrap scripts and launch templates to remove it before upgrading.

- Workload Management & Performance

Memory QoS (Quality of Service): improvements in memory QoS for better workload management. This feature enhances how Kubernetes handles memory-intensive applications, allowing for more predictable memory allocation and preventing over-provisioning. For example, you can define QoS guarantees for containers like this:

resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "256Mi"

Improved Node Resource Management: Kubernetes 1.31 enhances how node resources are allocated to containers. A more efficient scheduler better distributes resources like CPU and memory, reducing resource fragmentation and ensuring more efficient utilisation of cluster nodes.

- Graduations

AppArmor Support has now reached General Availability (GA), allowing easier application of AppArmor profiles using the securityContext field rather than annotations. This simplifies management of security contexts within pods.
Persistent Volume Last Transition Time has also reached GA. This feature logs the last time a persistent volume (PV) transitioned between phases, which can help with monitoring and debugging storage issues.
Kube-Proxy Enhancements for Ingress Reliability graduated to stable, introducing better handling of terminating nodes through connection draining and improved health checks

- Deprecations and Removals

With every new version, certain APIs are deprecated and eventually removed. In Kubernetes 1.31, the following are deprecated:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-mutating-webhook

- Future Deprecations to expect

Upgrade your EKS with terraform

To upgrade Amazon Elastic Kubernetes Service (EKS) from version 1.30 to 1.31 using Terraform, you need to follow a structured approach. The upgrade process involves upgrading the EKS control plane, worker nodes, and associated components like AWS load balancers, CoreDNS, and kube-proxy.

Upgrade Prerequisites

Terraform Version: Ensure you have the latest version of Terraform installed that supports the AWS provider.
AWS CLI: Update the AWS CLI if needed, as it will be used during the process.
Backup: Always backup critical configurations, Kubernetes resources, and ensure that worker nodes can be recreated if needed.

Review the EKS Version Support and Release Notes

Check Compatibility: Ensure that all components in your Kubernetes cluster are compatible with the new version (1.31). Review the EKS release notes to see the changes and breaking issues between versions 1.30 and 1.31.
Upgrade Plan: Ensure all third-party services and Kubernetes add-ons are updated to work with the new version.

Just like with all my upgrades. I use Terraform as it is fast, efficient and simplifies my life. I used following providers for the upgrade:

This time upgrade of the control plane takes around ~7 minutes. I would say this is super fast and zero issues afterwards have been experienced by me. I don’t even think I noticed any unavailable from API server itself which did happen in previous upgrades. AWS are doing a great job at reducing the time it takes to upgrade EKS control plane. This upgrade was by a minute faster than the previous one which shows good improvements.

I immediately upgraded cluster-critical worker nodes (three of them) which took around ~11 minutes to join the upgraded EKS cluster. This step is dependent on how many worker nodes you have and how many pods need to be drained from old nodes. So times may different if you have more nodes to rotate.

In general full upgrade process controlplane + worker nodes took around ~18 mins. Really good time I would say.

I personally use Terraform to deploy and upgrade my EKS clusters. Here is an example of the EKS cluster resource.

resource "aws_eks_cluster" "cluster" {
  name                      = local.name_prefix
  role_arn                  = aws_iam_role.cluster.arn
  version                   = "1.31"  # Update this from 1.30 to 1.31
  ...
}

For the worker nodes I have used official AMI: ami-0c2536a6df765d448. This AMI could be London region specific which is eu-west-2. I didn’t notice any issues after rotating all nodes. Nodes are running following version: v1.31.0-eks-a737599.

Templates I use for creating EKS clusters using Terraform can be found in my Github repository reachable under https://github.com/marcincuber/eks

Upgrading Managed EKS Add-ons

In this case the change is trivial and works fine, simply update the version of the add-on. In my case, from this release I utilise kube-proxy, coreDNS and ebs-csi-driver.

Terraform resources for add-ons

resource "aws_eks_addon" "kube_proxy" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "kube-proxy"
  addon_version     = "v1.31.0-eksbuild.5"
  resolve_conflicts = "OVERWRITE"
}

resource "aws_eks_addon" "core_dns" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "coredns"
  addon_version     = "v1.11.3-eksbuild.1"
  resolve_conflicts = "OVERWRITE"
}

resource "aws_eks_addon" "aws_ebs_csi_driver" {
  cluster_name      = aws_eks_cluster.cluster[0].name
  addon_name        = "aws-ebs-csi-driver"
  addon_version     = "v1.35.0-eksbuild.1"
  resolve_conflicts = "OVERWRITE"
}

After upgrading EKS control-plane

Remember to upgrade core deployments and daemon sets that are recommended for EKS 1.31.

CoreDNS — v1.11.3-eksbuild.1
Kube-proxy — 1.31.0-eksbuild.5
VPC CNI — 1.18.3-eksbuild.3
aws-ebs-csi-driver- v1.35.0-eksbuild.1

The above is just a recommendation from AWS. You should look at upgrading all your components to match the Kubernetes 1.31 version. They could include:

Load Balancer Controller
calico-node
Cluster Autoscaler or Karpenter
External Secrets Operator
Kube State Metrics
Metrics Server
csi-secrets-store
calico-typha and calico-typha-horizontal-autoscaler
Reloader
Keda (event driven autoscaler)
nvidia device plugin (used while utilising GPUs)

Validate the Upgrade yourself

Check Control Plane Version: Use the AWS CLI or AWS Management Console to verify the control plane version.

aws eks describe-cluster --name example-eks-cluster --query cluster.version

Check Node Group Versions: Ensure the worker nodes are running the correct version:

kubectl get nodes

Check Add-on Versions: Verify that CoreDNS and kube-proxy have been upgraded:

kubectl get pods -n kube-system -o wide | grep coredns
kubectl get daemonset -n kube-system kube-proxy -o wide

Post-Upgrade Checks

Ensure all your workloads are running correctly after the upgrade.
Test applications and services deployed in the cluster.
Check logs for any errors and resolve issues as needed.

Final Result

> $ kubectl version                                                                                                                                                                 [±6b962d2 ✓]
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.0-eks-a737599

I like to stay up to date with my CLIs so make sure you upgrade your kubectl to match your Kubernetes cluster version.

Summary and Conclusions

Even quicker upgrade of the EKS cluster than ever before. In 8mins the task to upgrade the controlplane was completed. I use Terraform to run my clusters and node upgrades so the github actions pipeline makes my life super easy.

Yet again, no significant issues. Hope you will have the same easy job to perform. All workloads worked just fine. I didn’t have to modify anything really.

If you are interested in the entire terraform setup for EKS, you can find it on my GitHub -> https://github.com/marcincuber/eks

Hope this article nicely aggregates all the important information around upgrading EKS to version 1.31 and it will help people speed up their task.

Long story short, you hate and/or you love Kubernetes but you still use it ;).

Please note that my notes relay on official AWS and Kubernetes sources.

Enjoy Kubernetes!!!

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Hi guys, I am Marcin and I am Technical Lead specialising in the field of DevOps. I am also a certified AWS Solutions…

github.com

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber