Amazon EKS- recover deleted images from EKS node to ECR repository

Find out how to re-push application images back to ECR from EKS node. Fixing “FATA[0000] failed to create a tmp single-platform image, content digest not found” error.

Introduction

In this story you will find out how to recover or push back (to ECR) an image from EKS node.

Recently, I came across an issue where an image has been deleted from ECR repository which was actively used. That particular image couldn’t be rebuild through pipeline as it was old and the only place where it was available was the EKS node where the pod was still running.

You will find out how to use nerdctl to recover your images from the node and what issues you may encounter. Also how to fix “FATA[0000] failed to create a tmp single-platform image” error.

My setup

• EKS Cluster- 1.30.3
• Kubectl cli- 1.30.1
• Containerd- 1.7.11
• Runc- 1.1.11

Recovering Images from EKS node back to private ECR will be accomplished using nerdctl in the next section.

Recovering

To access my EKS node I am using AWS session manager. This can be simply done from console by going to the instance and triggering “connect” option.

Once you are on your node, you want to find your images which can be done by running the following:

sudo -s # become root user
nerdctl images # list all images on the node

Now, you can find your image. In my case I am using 0123456789.dkr.ecr.eu-west-2.amazonaws.com/frontend-prod:v1.0.0 and I want to push it back to my ecr repo.

nerdctl push 0123456789.dkr.ecr.eu-west-2.amazonaws.com/frontend-prod:v1.0.0
FATA[0000] failed to create a tmp single-platform image “0123456789.dkr.ecr.eu-west-2.amazonaws.com/frontend-prod:v1.0.0-tmp-reduced-platform”: content digest sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: not found

As you can see, it resulted in the “failed to create a tmp…” error. To fix that you need to run your push command with additional flag. “ — all-platforms”.

nerdctl push 0123456789.dkr.ecr.eu-west-2.amazonaws.com/frontend-prod:v1.0.0 --all-platforms

This will bring another error which is “401 Unauthorized”. This simply means you are not authenticated with your ECR. To fix this you can run the following:

aws ecr get-login-password - region eu-west-2 | nerdctl login - username AWS - password-stdin 0123456789.dkr.ecr.eu-west-2.amazonaws.com

Now, triggering “nerdctl push…” command again, will return another error which is “403 Forbidden”. That error simply suggests that the IAM Role associated with your EKS node doesn’t have permissions to push image to ECR. To solve this, you need to temporary add “ecr:PutImage” permission to your IAM Role policy.

At this stage when you trigger “nerdctl push…” command, everything will work as expected and your image will be back in your ECR repo!

Hope this will help you recover your images quickly when needed.

Alls commands used in the process:

nerdctl images
nerdctl push 0123456789.dkr.ecr.eu-west-2.amazonaws.com/frontend-prod:v1.0.0 - all-platforms
aws ecr get-login-password - region eu-west-2 | nerdctl login - username AWS - password-stdin 0123456789.dkr.ecr.eu-west-2.amazonaws.com

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber