[AWS] Discovering how to design Cluster Autoscaler on EKS.

Một ngày đẹp trời bạn nhận thấy cluster eks nó tự động tăng số lượng node rồi lại tự giảm.
Nếu bạn chưa biết tạo sao thì tìm hiểu Cluster Autoscaler trên EKS

1) Look into Cluster Autoscaler on EKS

2) Install Cluster Autoscale on EKS

Để làm được bài này thì mình nghĩ các bạn nên đọc bài viết bên dưới trước.

[AWS] EKS IAM Roles for Service Accounts (IRSA) using Terraform

# Resource: IAM Policy for Cluster Autoscaler
resource "aws_iam_policy" "cluster_autoscaler_iam_policy" {
  name        = "${local.name}-AmazonEKSClusterAutoscalerPolicy"
  path        = "/"
  description = "EKS Cluster Autoscaler Policy"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
})
}

# Resource: IAM Role for Cluster Autoscaler
## Create IAM Role and associate it with Cluster Autoscaler IAM Policy
resource "aws_iam_role" "cluster_autoscaler_iam_role" {
  name = "${local.name}-cluster-autoscaler"

  # Terraform's "jsonencode" function converts a Terraform expression result to valid JSON syntax.
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Federated = "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn}"
        }
        Condition = {
          StringEquals = {
            "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
          }
        }        
      },
    ]
  })

  tags = {
    tag-key = "cluster-autoscaler"
  }
}


# Associate IAM Policy to IAM Role
resource "aws_iam_role_policy_attachment" "cluster_autoscaler_iam_role_policy_attach" {
  policy_arn = aws_iam_policy.cluster_autoscaler_iam_policy.arn 
  role       = aws_iam_role.cluster_autoscaler_iam_role.name
}

output "cluster_autoscaler_iam_role_arn" {
  description = "Cluster Autoscaler IAM Role ARN"
  value = aws_iam_role.cluster_autoscaler_iam_role.arn
}

resource “aws_iam_policy” “cluster_autoscaler_iam_policy” {}
==> Tạo policy cho phép access và acction trên AutoScaling

resource “aws_iam_role” “cluster_autoscaler_iam_role” {}
==> tạo Assume Role cho phép sử dụng service account để access AutoScaling

Giờ bạn tiền hay lấy các credentials để helm Install.

# Datasource: EKS Cluster Auth 
data "aws_eks_cluster_auth" "cluster" {
  name = data.terraform_remote_state.eks.outputs.cluster_id
}

# HELM Provider
provider "helm" {
  kubernetes {
    host                   = data.terraform_remote_state.eks.outputs.cluster_endpoint
    cluster_ca_certificate = base64decode(data.terraform_remote_state.eks.outputs.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

# Install Cluster Autoscaler using HELM

# Resource: Helm Release 
resource "helm_release" "cluster_autoscaler_release" {
  depends_on = [aws_iam_role.cluster_autoscaler_iam_role ]            
  name       = "${local.name}-ca"

  repository = "https://kubernetes.github.io/autoscaler"
  chart      = "cluster-autoscaler"

  namespace = "kube-system"   

  set {
    name  = "cloudProvider"
    value = "aws"
  }

  set {
    name  = "autoDiscovery.clusterName"
    value = data.terraform_remote_state.eks.outputs.cluster_id
  }

  set {
    name  = "awsRegion"
    value = var.aws_region
  }

  set {
    name  = "rbac.serviceAccount.create"
    value = "true"
  }

  set {
    name  = "rbac.serviceAccount.name"
    value = "cluster-autoscaler"
  }

  set {
    name  = "rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = "${aws_iam_role.cluster_autoscaler_iam_role.arn}"
  }
  # Additional Arguments (Optional) - To Test How to pass Extra Args for Cluster Autoscaler
  #set {
  #  name = "extraArgs.scan-interval"
  #  value = "20s"
  #}    
   
}

Giờ bạn đó có helm chart của cluster-autoscaler để apply vào k8s

Outputs:

cluster_autoscaler_helm_metadata = tolist([
  {
    "app_version" = "1.23.0"
    "chart" = "cluster-autoscaler"
    "name" = "sap-dev-ca"
    "namespace" = "kube-system"
    "revision" = 1
    "values" = "{\"autoDiscovery\":{\"clusterName\":\"SAP-dev-eksdemo\"},\"awsRegion\":\"us-east-1\",\"cloudProvider\":\"aws\",\"rbac\":{\"serviceAccount\":{\"annotations\":{\"eks.amazonaws.com/role-arn\":\"arn:aws:iam::250887682577:role/SAP-dev-cluster-autoscaler\"},\"create\":true,\"name\":\"cluster-autoscaler\"}}}"
    "version" = "9.21.0"
  },
])
cluster_autoscaler_iam_role_arn = "arn:aws:iam::250887682577:role/SAP-dev-cluster-autoscaler"

Giờ recheck 1 số thứ

 kubectl -n kube-system get pods

Bạn có thể coi logs của pod cluster autoscaler

kubectl -n kube-system logs -f $(kubectl -n kube-system get pods | egrep -o 'hr-dev-ca-aws-cluster-autoscaler-[A-Za-z0-9-]+')

Kiểm tra SA

root@work-space-u20:~# kubectl -n kube-system describe sa cluster-autoscaler
Name:                cluster-autoscaler
Namespace:           kube-system
Labels:              app.kubernetes.io/instance=sap-dev-ca
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=aws-cluster-autoscaler
                     helm.sh/chart=cluster-autoscaler-9.21.0
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::250887682577:role/SAP-dev-cluster-autoscaler
                     meta.helm.sh/release-name: sap-dev-ca
                     meta.helm.sh/release-namespace: kube-system
Image pull secrets:  <none>
Mountable secrets:   cluster-autoscaler-token-bzdgg
Tokens:              cluster-autoscaler-token-bzdgg
Events:              <none>

3) Cluster Autoscaler Testing

cluster-autoscaler-sample-app.yaml
>>>>>>>>>>>>>>>>
>>>>>>>>>>>

https://github.com/mrnim94/terraform-aws/tree/master/elk-cluster-autoscaler
bạn có thể thao khảo các file trong repo này.