Amazon GuardDuty- deploy EKS Runtime Monitoring using Terraform

Deploying latest Amazon GuardDuty Support of Amazon EKS Runtime Monitoring.

Introduction

Amazon GuardDuty has been with us for some time since 2017. It has been capable of analysing tens of billions of events per minute across multiple AWS data sources, such as AWS CloudTrail event logs, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, and DNS query logs, Amazon Simple Storage Service (Amazon S3) data plane events, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and Amazon Relational Database Service (Amazon RDS) login events to protect your AWS accounts and resources.

With the latest release which called “Amazon GuardDuty EKS Runtime Monitoring”, we have a mechanism to detect runtime threats from over 30 security findings to protect our EKS clusters. The new EKS Runtime Monitoring uses a fully managed EKS add-on that adds visibility into individual container runtime activities, such as file access, process execution, and network connections. It is deployed as a Kubernetes Daemonset which has a pod running on every node. This agent and effectively GuardDuty can now identify specific containers within your EKS clusters that are potentially compromised and detect attempts to escalate privileges from an individual container to the underlying Amazon EC2 host and the broader AWS environment. GuardDuty EKS Runtime Monitoring findings provide metadata context to identify potential threats and contain them before they escalate.

Implementation

Requirements to make this need functionality work using Terraform:

• Enable GuardDuty EKS Protection with Runtime Monitoring
• Deploying VPC Endpoint of type guardduty-data
• Deploy EKS addon named aws-guardduty-agent

Enablement

Enabling manual configuration so that random resources don’t get created in your AWS account :). Please note, this has to be done manually in the console as Terraform doesn’t provide that functionality. In fact, AWS Terraform provider support around GuardDuty features need to be re-implemented/re-factored so it is more flexible in terms of supporting new features.

Our first step is to enable EKS protection with the following settings:

VPC Endpoint Deployment

data "aws_vpc_endpoint_service" "guardduty" {
  service_type = "Interface"
  filter {
    name   = "service-name"
    values = ["com.amazonaws.${data.aws_region.current.name}.guardduty-data"]
  }
}

resource "aws_vpc_endpoint" "eks_vpc_guardduty" {
  vpc_id            = module.vpc_eks.vpc_id
  service_name      = data.aws_vpc_endpoint_service.guardduty.service_name
  vpc_endpoint_type = "Interface"
  
  policy = data.aws_iam_policy_document.eks_vpc_guardduty.json

  security_group_ids  = [aws_security_group.eks_vpc_endpoint_guardduty.id]
  subnet_ids          = module.vpc_eks.public_subnets
  private_dns_enabled = true
}

resource "aws_security_group" "eks_vpc_endpoint_guardduty" {
  name_prefix = "${local.name_prefix}-vpc-endpoint-guardduty-sg-"
  description = "Security Group used by VPC Endpoints."
  vpc_id      = module.vpc_eks.vpc_id

  tags = {
    "Name"             = "${local.name_prefix}-vpc-endpoint-guardduty-sg-"
    "GuardDutyManaged" = "false"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_vpc_security_group_ingress_rule" "eks_vpc_guardduty_ingress" {
  security_group_id = aws_security_group.eks_vpc_endpoint_guardduty.id
  description       = "Ingress for port 443."

  cidr_ipv4   = "0.0.0.0/0"
  from_port   = 443
  ip_protocol = "tcp"
  to_port     = 443
}

It is also recommended to add the policy to the VPC endpoint created above. The following policy restricts VPC endpoint usage to the specified account only:

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Action": "*",
   "Resource": "*",
   "Effect": "Allow",
   "Principal": "*"
  },
  {
   "Condition": {
    "StringNotEquals": {
     "aws:PrincipalAccount": "123456789012"
    }
   },
   "Action": "*",
   "Resource": "*",
   "Effect": "Deny",
   "Principal": "*"
  }
 ]
}

data "aws_iam_policy_document" "eks_vpc_guardduty" {
  statement {
    actions = ["*"]

    effect = "Allow"

    resources = ["*"]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
  }

  statement {
    actions = ["*"]

    effect = "Deny"

    resources = ["*"]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }

    condition {
      test     = "StringNotEquals"
      variable = "aws:PrincipalAccount"

      values = [data.aws_caller_identity.current.account_id]
    }
  }
}

Important, your VPC endpoint must run in public subnets (which use Internet Gateway) and have private DNS names enabled. This is because agent is looking for guardduty-data.eu-west-2.amazonaws.com endpoint.

EKS addon deployment

resource "aws_eks_addon" "guardduty" {
  count = var.eks_addon_version_guardduty != null ? 1 : 0

  cluster_name      = aws_eks_cluster.cluster.name
  addon_name        = "aws-guardduty-agent"
  addon_version     = "v1.0.0-eksbuild.1"
  resolve_conflicts = "OVERWRITE"

  preserve = true

  tags = {
    "eks_addon" = "guardduty"
  }
}

A successful deployment of the GuardDuty agent will results in the following output:

These pods collect various event types and send them to the GuardDuty backend for threat detection and analysis.

Finally, you will be able to see the status of your cluster in the GuardDuty console view. It shows as follows:

Checkout EKS Runtime Security Findings

Finally, we have a fully deployed and working solution using Terraform. Please note that my cluster didn’t get any security findings for a couple of days so the screenshot below is the generated sample rather than issues I found. I have to say I always run very restrictive policies against my Kubernetes cluster so that security findings are minimal. I am extremely protective of my environments and keep developers’ access to a minimum and this is my principal number one.

Going back to GuardDuty. When GuardDuty detects a potential threat and generates a security finding, you can view the details of the corresponding findings. These security findings indicate either a compromised EC2 instance, container workload, an EKS cluster, or a set of compromised credentials in your AWS environment.

If you want to generate EKS Runtime Monitoring sample findings for testing purposes, see Generating sample findings in GuardDuty in the AWS documentation. Here is an example of potential security issues: a newly created or recently modified binary file in an EKS cluster has been executed. (Screenshot copied for internet).

The ResourceType for an EKS Protection finding type could be an Instance, EKSCluster, or Container. If the Resource type in the finding details is EKSCluster, it indicates that either a pod or a container inside an EKS cluster is potentially compromised. Depending on the potentially compromised resource type, the finding details may contain Kubernetes workload details, EKS cluster details, or instance details.

Conclusion

GuardDuty EKS Protection has been an issue feature to test our and I didn’t encounter any issues. I can’t comment how expensive this is going to be as I am still in a Trial period.

Hopefully, this article will allow to quickly deploy this to your own EKS environments and take advantage of the latest capability.

Enjoy Kubernetes!!!

Sponsor Me

Sponsor @marcincuber on GitHub Sponsors

Like with any other story on Medium written by me, I performed the tasks documented. This is my own research and issues I have encountered.

Thanks for reading everybody. Marcin Cuber