Amazon EKS- migrating Karpenter resources from alpha to beta API version

Detailed migration journey from v1alpha5 to API version v1beta1 of karpenter.sh

Introduction

Karpenter is a Kubernetes node lifecycle manager created by AWS. It is responsible for maintaining, running, scaling, and minimising cluster node configurations. It is a major project that was started to provide a better alternative to cluster-autoscaler which Karpenter greatly achieved in quick succession.

The API changes are being rolled out as part of Karpenter version 0.32.X release. Existing Karpenter resources need to be upgraded to the version, which can be found in the upgrade guide, however, it is not as simple as I thought. I hope this story will provide you with details that are missing elsewhere and will speed up your upgrade process. Existing alpha APIs are now deprecated and remain available in this single version. Starting with the release v0.33.0, Karpenter will only support its v1beta1 APIs. Like many open-source Kubernetes projects, Karpenter APIs also follow a maturity progression of alpha → beta → stable. The graduation from alpha to beta required significant changes to the APIs, which are highlighted in this post.

What is changing

With 0.32.X release, Karpenter deprecates the Provisioner, AWSNodeTemplate and Machine APIs, while introducing NodePool, EC2NodeClass, and NodeClaim. This simply means that all resources need to be re-written, re-deployed and replaced.

API group and kind naming

The v1beta1 version introduces the following new APIs while deprecating the existing ones:

• karpenter.sh/Provisioner -> karpenter.sh/NodePool
• karpenter.sh/Machine -> karpenter.sh/NodeClaim
• karpenter.k8s.aws/AWSNodeTemplate -> karpenter.k8s.aws/EC2NodeClass

Each of these naming changes comes with schema changes that need to be considered as you update to the latest version of Karpenter. Let’s look at each change and what the new API definition looks like. In-depth read can be found in official concepts doc.

Helm chart changes

Not only resources changed but also values inside Karpenter Helm chart had been modified.

Goal

This story will show the steps I performed to first upgrade to new version of the helm chart, followed by IAM Role Policy changes and lastly v1beta1 resource replacement.

I am using Flux2 Gitops operator to deploy my Kubernetes manifests and Terraform for AWS resources such as IAM Roles and policies.

Helm Chart Upgrade

<v0.32.x- before upgrade

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: karpenter
  namespace: karpenter
spec:
  releaseName: karpenter
  chart:
    spec:
      chart: karpenter
      version: v0.31.1
      sourceRef:
        kind: HelmRepository
        name: karpenter
        namespace: flux-system
  interval: 1h0m0s
  install:
    remediation:
      retries: 3
  values:
    replicas: 3
    serviceAccount:
      name: karpenter
    logLevel: debug
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/karpenter-controller-dev
    settings:
      aws:
        clusterName: dev
        defaultInstanceProfile: "node-karpenter-dev"
        enablePodENI: true
      interruptionQueue: "karpenter-spot-interruption-dev"
    nodeSelector:
      kubernetes.io/os: linux
      workload: system-critical

Importantly, note that `defaultInstanceProfile` has been removed from helm chart and IAM Role profile is no longer specified. With the new version of Karpenter, each NodeClass resource will contain role argument which will be set to IAM Role name (not the profile name). Be careful with this during migration.

You will also see below that the entire aws settings block has been deprecated as well.

≥v0.32.x- after upgrade

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: karpenter
  namespace: karpenter
spec:
  releaseName: karpenter
  chart:
    spec:
      chart: karpenter
      version: v0.32.1
      sourceRef:
        kind: HelmRepository
        name: karpenter
        namespace: flux-system
  interval: 1h0m0s
  install:
    remediation:
      retries: 3
  values:
    replicas: 3
    serviceAccount:
      name: karpenter
    logLevel: debug
    controller:
      resources:
        requests:
          cpu: 1
          memory: 1Gi
        limits:
          cpu: 1
          memory: 1Gi
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/karpenter-controller-dev
    settings:
      clusterName: dev
      interruptionQueue: "karpenter-spot-interruption-dev"
      featureGates:
        drift: false
    nodeSelector:
      kubernetes.io/os: linux
      workload: system-critical

CRDs upgrade

I have a seperate helm release to handle CRDs upgrade so I simply increamented the version to the latest available. This deployed all beta apiversion resources that I needed for the migration.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: karpenter-crds
  namespace: karpenter
spec:
  releaseName: karpenter-crds
  interval: 10m
  chart:
    spec:
      chart: karpenter-crd
      version: "v0.32.1"
      sourceRef:
        kind: HelmRepository
        name: karpenter-crds
        namespace: flux-system

IAM Role and policy

It is important to mention that there are in fact two IAM Roles that need creating.

1. karpenter-node
2. karpenter-controller

For me it was very unclear that you need two role from the offical docs. Here are the details for both:

1. Karpenter Node IAM role

resource "aws_iam_role" "eks_node_karpenter" {
  name = "node-karpenter-dev"

  assume_role_policy = data.aws_iam_policy_document.eks_node_karpenter_assume_role_policy.json

  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  ]
}

resource "aws_iam_instance_profile" "eks_node_karpenter" {
  name = "node-karpenter-dev"
  role = aws_iam_role.eks_node_karpenter.name
}

data "aws_iam_policy_document" "eks_node_karpenter_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

2. Karpenter Controller IAM role

resource "aws_iam_role" "karpenter_controller" {
  name        = "karpenter-controller-dev"
  description = "Allow karpenter-controller EC2 read and write operations."

  assume_role_policy = templatefile("policies/oidc_assume_role_policy.json", {
    OIDC_ARN  = aws_iam_openid_connect_provider.cluster.arn,
    OIDC_URL  = replace(aws_iam_openid_connect_provider.cluster.url, "https://", ""),
    NAMESPACE = "karpenter",
    SA_NAME   = "karpenter"
  })

  force_detach_policies = true

  tags = {
    "ServiceAccountName"      = "karpenter"
    "ServiceAccountNamespace" = "karpenter"
  }
}

resource "aws_iam_role_policy" "karpenter_controller" {
  name = "KarpenterControllerPolicy"
  role = aws_iam_role.karpenter_controller.id

  policy = data.aws_iam_policy_document.karpenter_controller.json
}

The policy for this role can be found in my Github as it is rather long:

eks/terraform/oidc-iam-policies.tf at main · marcincuber/eks

IAM Role migration- details

The Karpenter controller uses an AWS Identity and Access Management (AWS IAM) role to grant the permissions to launch and operate Amazon Elastic Compute Cloud (Amazon EC2) instances in your AWS account. As part of the upgrade, I created a new policy with permissions by adding the following:

1. Add to the ec2:RunInstances, ec2:CreateFleet, and ec2:CreateLaunchTemplate permissions scoped down to the tag-based constraint karpenter.sh/nodepool instead of the previous tag key karpenter.sh/provisioner-name.
2. Grant permissions to the actions iam:CreateInstanceProfile, iam:AddRoleToInstanceProfile, iam:RemoveRoleFromInstanceProfile, iam:DeleteInstanceProfile, and iam:GetInstanceProfile. All of these permissions (with the exception of the GetInstanceProfile permission) are constrained by tag-based policy that ensures that the controller only has permission to operate on instance profiles that it was responsible for creating. These are needed to support the Karpenter-managed instance profiles.

The full policy written in Terraform can be found in the link which is provided in the section above as well.

Migrating provisioners and node templates to NodePools and NodeClasses

API migration

To transition from the alpha to the new v1beta1 APIs, you should first install the new v1beta1 Custom Resource Definitions (CRDs). This is what we achieved by upgrading helm chart to the newest version. Subsequently, we need to generate the beta equivalent of each alpha API for both Provisioners and AWSNodeTemplates. It’s worth noting that the migration from Machine to NodeClaim is managed by Karpenter as we transition CustomResources from Provisioners to NodePools.

Karpenter team provides a tool karpenter-convert, which is a command line utility designed to streamline the creation of NodePool and EC2NodeClass objects. In the following, you’ll find the steps to effectively utilize this tool:

1. Install the command line utility: go install github.com/aws/karpenter/tools/karpenter-convert/cmd/karpenter-convert@latest
2. Migrate each provisioner into a NodePool: karpenter-convert -f provisioner.yaml > nodepool.yaml
3. Migrate each AWSNodeTemplate into an EC2NodeClass: karpenter-convert -f nodetemplate.yaml > nodeclass.yaml
4. For each EC2NodeClass generated by the tool you need to manually specify the AWS role. The tool leaves a placeholder $KARPENTER_NODE_ROLE, which you need to replace with your actual role name. This is the name of the IAM Role created above; Karpenter Node IAM role (node-karpenter-dev).

Note that for my migration I haven’t used the cli. I do prefer to learn things in depth so I migrated everything using official docs. So, I have written everything from scratch line by line.

<v0.32.x- before upgrade

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: apps
spec:
  taints:
    - key: karpenter.sh/legacy
      effect: NoSchedule
  labels:
    intent: apps
  kubeletConfiguration:
    clusterDNS: ["10.150.0.10"]
    maxPods: 58
  requirements:
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["m6i.2xlarge", "m6a.2xlarge", "m5.2xlarge", "m5a.2xlarge"]
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ["spot", "on-demand"]
  limits:
    resources:
      cpu: 128
      memory: 512Gi
  ttlSecondsUntilExpired: 5184000 # 60 days ttl
  providerRef:
    name: apps
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: apps
spec:
  subnetSelector:
    karpenter.sh/discovery: "dev"
  securityGroupSelector:
    karpenter.sh/discovery: "dev"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeType: gp3
        volumeSize: 100Gi
        deleteOnTermination: true
        encrypted: true
  tags:
    Name: "karpenter-node-dev"
    KarpenerProvisionerName: "apps"
    IntentLabel: "apps"

Note the taints on the provisioner. I added it after deploying new beta resource so that new nodes will be provisioned through new resources and not the old ones. This is the case since my application were not configured to tolerate such taint.

≥v0.32.x- after upgrade

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: apps
spec:
  template:
    metadata:
      labels:
        intent: apps
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: apps
      requirements:
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["m6i.2xlarge", "m6in.2xlarge", "m6a.2xlarge", "m5.2xlarge"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
      kubelet:
        clusterDNS: ["10.150.0.10"]
        maxPods: 58
  limits:
    cpu: 128
    memory: 512Gi
  disruption:
    expireAfter: 1440h
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: apps
spec:
  amiFamily: AL2
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "dev"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "dev"
  role: "node-karpenter-dev"
  tags:
    Name: "karpenter-node-dev-apps"
    Intent: "apps"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
        deleteOnTermination: true
  detailedMonitoring: true

Note, the role argument. This is a new argument which is required to be specified. It is set to be the name of the Karpenter Node IAM Role. This is the replacement for defaultprofilename that was previously set at helm release level.

Once I had my EC2NodeClass and NodePool configured, I was terminating each of the old nodes one by one.

kubectl get machines # show ec2 instances managed by provisioner
kubectl delete node NODE_ID # delete old node

After all legacy nodes were terminated, I simply delete apps provisioner and corresponding AWSNodeTemplate resource which concluded my upgrade.

Conclusions

You can find the official migration documentation here. I found this doc okey but it wasn’t always clear what needed changing. I hope this article will help you speed up with your upgrade.

I do believe this is a bit of an extreme upgrade in terms of amount of changes. However, it is all doable without any downtime.

In this story, you can see modifications introduced by the new APIs and hopefully a simpler upgrade for yourself. Based on the offical notes; the majority of these API changes will eventually move to the stable v1 API, which enables a broader user base to take full advantage of Karpenter’s capabilities in workload-native node provisioning.

There are some other deprecations and changes that we didn’t cover in this post. Please head to the Karpenter upgrade guide for a comprehensive migration guideline.

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber