[EKS] Upgrade eks cluster by eksctl

Posted on By nim No Comments on [EKS] Upgrade eks cluster by eksctl

1) Create eks cluter by eksctl

Đương nhiên là để lab được bài này thì bạn cần create eks cluster trước hihi.

Đầu tiên chúng ta cài đặt 1 cluster eks bằng eksctl.

cat cluster-1.20.yaml

>>>>>
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: devops-catalog
  region: us-east-2
  version: "1.20"
managedNodeGroups:
- name: primary
  instanceType: t2.small
  minSize: 1
  maxSize: 2
  spot: true

https://eksctl.io/usage/schema/
bạn có thể check thêm key value trên schema.

eksctl create cluster --config-file cluster-1.20.yaml

Bạn có thể vào cloudformation để sem chi tiết nó đang làm cái gì.

GIờ thì nó tạo sẽ tạo worker node.

2) Upgrade EKS Cluster.

2.1) Check deprecated

Find Kubernetes resources that have been deprecated

2.1.1) pluto

https://github.com/FairwindsOps/pluto

mình lab trên ubuntu nên mình sẽ truy cập vào link sau và download Binary

Download về máy

wget https://github.com/FairwindsOps/pluto/releases/download/v5.11.1/pluto_5.11.1_linux_amd64.tar.gz
tar zxf pluto_5.11.1_linux_amd64.tar.gz
mv pluto /usr/local/bin
Thằng Pluto này thì nó quét các file deplpoyment có sẵn.

2.1.2) kubent

Cái này thì nó quét trực tiếp cluster của bạn

https://github.com/doitintl/kube-no-trouble

root@LP11-D7891:~# kubent
10:43AM INF >>> Kube No Trouble `kubent` <<<
10:43AM INF version 0.7.0 (git sha d1bb4e5fd6550b533b2013671aa8419d923ee042)
10:43AM INF Initializing collectors and retrieving data
10:43AM INF Target K8s version is 1.21.14-eks-ffeb93d
10:43AM INF Retrieved 227 resources from collector name=Cluster
10:43AM INF Retrieved 6 resources from collector name="Helm v3"
10:43AM INF Loaded ruleset name=custom.rego.tmpl
10:43AM INF Loaded ruleset name=deprecated-1-16.rego
10:43AM INF Loaded ruleset name=deprecated-1-22.rego
10:43AM INF Loaded ruleset name=deprecated-1-25.rego
10:43AM INF Loaded ruleset name=deprecated-1-26.rego
10:43AM INF Loaded ruleset name=deprecated-future.rego
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.22 <<<
------------------------------------------------------------------------------------------
KIND                           NAMESPACE          NAME                               API_VERSION                            REPLACE_WITH (SINCE)
CustomResourceDefinition       <undefined>        externalmetrics.metrics.aws        apiextensions.k8s.io/v1beta1           apiextensions.k8s.io/v1 (1.16.0)

giờ bạn thấy là externalmetrics.metrics.aws và KIND: CustomResourceDefinition bị outdated.

bạn sử dụng command sàu để edit:

kubectl edit CustomResourceDefinition/externalmetrics.metrics.aws

2.2) Upgrade Controller EKS

eksctl upgrade cluster --config-file cluster-1.21.yaml --approve

Giờ bạn sẽ upgrade eks cluster.
dưới đây là file mới

cat cluster-1.21.yaml

#####
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: devops-catalog
  region: us-east-2
  version: "1.21"
managedNodeGroups:
- name: primary-1-21
  instanceType: t2.small
  minSize: 1
  maxSize: 2
  spot: true

Bạn sẽ thấy nó mới chỉ upgrade cài controller.

Còn các node worker thì chưa được upgrade.

root@work-space-u20:~/eks/eksctl-demo# kubectl get node
NAME                                           STATUS   ROLES    AGE    VERSION
ip-192-168-42-164.us-east-2.compute.internal   Ready    <none>   177m   v1.20.15-eks-ba74326

2.3) Create a new node group with a new version.

Giờ bạn cần phải tạo new node group

eksctl create nodegroup --config-file cluster-1.21.yaml
Bạn sẽ thấy nó tạo 1 node group mới
Thấy là anh instance trên EC2
Đây là ảnh bên auto scaling Group.

Nếu bạn muốn tìm hiểu về set max min instance thì tham khảo link này:
http://arun-gupta.github.io/update-eks-cluster/

Bạn muốn tìm hiều spot instance thì link này:
https://eksctl.io/usage/spot-instances/

2.4) Upgrade the important workloads

2.4.1) CoreDNS Plugin

Giờ kiểm tra CoreDNS Plugin:

https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html

kubectl --namespace kube-system describe deployment coredns | grep Image | cut -d "/" -f 3
coredns:v1.8.3-eksbuild.1

Nếu CoreDNS của bản dưới v1.8.3 và muốn upgrade coredns lên version mới hơn 1.8.3

bạn cần config những thứ sau:

kubectl edit clusterrole system:coredns -n kube-system

##add this into content.
...
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
...

của mình đang ở version 1.8.3 thì change image thôi.

kubectl --namespace kube-system set image deployment.apps/coredns coredns=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.8.4-eksbuild.1

làm sao để tim được tag “v1.8.4-eksbuild.1”
1) bạn search quanh trên mạng
2) tạo 1 con Ec2 rồi vào con Ec2 đó pull thử.

Warning

Nếu bạn gặp warning:
Warning – beta.kubernetes.io/os is deprecated since v1.14; use “kubernetes.io/os” instead
bạn đừng lo lắng cứ ignored
https://github.com/coredns/helm/issues/83

2.4.2) Kube Proxy

Kiểm tra Kube Proxy

kubectl -n kube-system get daemonset kube-proxy -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy:v1.20.7-eksbuild.1

https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html

kubectl -n kube-system set image daemonset.apps/kube-proxy kube-proxy=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.21.14-eksbuild.2

2.4.3) VPC CNI Plugin

Kiểm tra VPC CNI Plugin

kubectl -n kube-system describe daemonset aws-node | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.5-eksbuild.1
amazon-k8s-cni:v1.7.5-eksbuild.1

Ở bạn 1 .21 thì họ khuyên như sau:

Bạn có thể vào link này để chọn manifest thích hợp.
https://github.com/aws/amazon-vpc-cni-k8s/releases
cứ chọn latest

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.11.4/config/master/aws-k8s-cni.yaml
root@work-space-u20:~/eks/eksctl-demo# kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2 
amazon-k8s-cni-init:v1.11.4
amazon-k8s-cni:v1.11.4

2.4.4) AWS Load Balancer Controller

Khi upgrade lên eks 1.22 thì gặp lỗi sau trên aws-load-balancer

{"level":"error","ts":1676453510.3522048,"logger":"setup","msg":"unable to create controller","controller":"Ingress","error":"the server could not find the requested resource"}

Bạn cần access vào link dưới để download file yaml:
https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases

Bạn download manifest này xuống.

bạn edit Deployment có tên là aws-load-balancer-controller –> –cluster-name=your-cluster-name

Sau khi sửa xong thì bạn cứ apply thôi.

2.4.4.x) Issues -> secret “aws-load-balancer-webhook-tls” not found

Nếu bạn gặp lỗi:
MountVolume.SetUp failed for volume "cert" : secret "aws-load-balancer-webhook-tls" not found

Chúng ta cần create this secret by hand.

# Generate cert key pair
openssl req -x509 -newkey rsa:4096 -sha256 -days 36500 -nodes \
    -keyout tls.key -out tls.crt -subj "/CN=aws-load-balancer-webhook-service.kube-system.svc" \
    -addext "subjectAltName=DNS:aws-load-balancer-webhook-service.kube-system.svc,DNS:aws-load-balancer-webhook-service.kube-system.svc.cluster.local"

# Add key pair as Kubernetes secrets
kubectl -n kube-system create secret generic aws-load-balancer-webhook-tls \
  --from-file=./tls.crt \
  --from-file=./tls.key

Link tham khảo: https://yhf8377.medium.com/deploying-wordpress-site-using-aws-eks-fargate-efs-and-rds-732e10287fee

2.4.5) Cluster Autoscaler

bạn cứ lấy version mới nhất
Nếu bạn thấy sợ version mới nhất thì deploy version của Cluster Autoscaler = version k8s
https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html


https://github.com/kubernetes/autoscaler/releases

Câu lệnh change version.

 kubectl describe -n kube-system deployment.apps/cluster-autoscaler | grep Image
    Image:      us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.21.3

kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.3
deployment.apps/cluster-autoscaler image updated

3) Delete the old node Group.


eksctl delete nodegroup --config-file cluster-1.21.yaml --only-missing --approve

Chỗ này khá quan trọng:
–only-missing: bạn sẽ detele các node group mà version không giống như trong config file.

4) Delete the eks cluster.

eksctl delete cluster --config-file cluster-1.21.yaml --wait

5) eksctl without –config-file

Nếu động đội của bạn chỉ đưa cho bạn 1 con eks và nhờ bạn upgrade thì:

https://www.stacksimplify.com/aws-eks/eks-cluster/delete-eks-cluster-nodegroup/

eksctl upgrade cluster --name devops-catalog \
--region us-east-2  \
--version 1.22 \
--approve

eksctl create nodegroup --cluster devops-catalog \
--region us-east-2

eksctl get nodegroup --cluster devops-catalog --region us-east-2

Giờ delete node group cũ:

eksctl delete nodegroup --cluster devops-catalog --region us-east-2 --name primary-1-21
eksctl delete cluster --name eks-nimtechnology --region us-east-2 --wait

Bạn có thê tham khảo bài dưới:
https://rharshad.com/aws-eks-upgrade-1.19-to-1.20/

Error cluster-autoscaler

Failed to create AWS Manager

F1031 05:42:07.134307       1 aws_cloud_provider.go:382] Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: arn:aws:sts::xxxxxxxx:assumed-role/eksctl-nimtechnology-nodegr-NodeInstanceRole-FCSNPINNUMJV/i-0f3eb129a6040a5a3 is not authorized to perform: autoscaling:DescribeTags because no identity-based policy allows the autoscaling:DescribeTags action

hOẶC lỗi này:

I0821 06:56:07.191221       1 auto_scaling_groups.go:444] Could not check instance availability, creating placeholder node anyways: AccessDenied: User: arn:aws:sts::XXXXXXXXXX:assumed-role/eksctl-dev-nim-engines-addon-iamservi-Role1-1HRS71P1PTVXR/1692600967063901864 is not authorized to perform: autoscaling:DescribeScalingActivities because no identity-based policy allows the autoscaling:DescribeScalingActivities action

You must create a new policy following the below config and then attach this policy to the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}
Bạn click và tạo 1 policy mới

Bạn có thể tham khảo nhiều hơn ở đây:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md

Ngoài ra gần đây khi mình sử dụng Cluster Autoscaler 1.24.3 và eks 1.24 mình gặp lỗi sau:

aws_cloud_provider.go:369] Failed to generate AWS EC2 Instance Types: UnauthorizedOperation: You are not authorized to perform this operation.

https://github.com/kubernetes/kops/issues/13520
theo như link trên thì thêm ec2:DescribeInstanceTypes vào policy của cluster autoscaler.
Vì nó không gợi ý role name của cluster auto scaler nên bạn mò trong service account là thấy nhé

Failed to watch *v1.Namespace

Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "namespaces" in API group "" at the cluster scope

How to fix:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml#L35-L43

AWS - Amazon Web Service

More Related Articles

[AWS] VPC PEERING – Connecting between other VPCs. AWS - Amazon Web Service
[AWS] Configure Ingress SSL and SSL Redirect on Load Balancer Controller AWS - Amazon Web Service
[Terraform] – Terraform Beginner – Lesson 6: Terraform Import, Tainting Resources, and Debugging AWS - Amazon Web Service
[AWS] Create EKS Cluster and EKS Node Groups in Public and Private Subnets AWS - Amazon Web Service
[AWS] Look into Data Transfer on AWS AWS - Amazon Web Service
[Assume Role/KMS] Using Assume Role to make the other AWS Account access all KMS that/this AWS Account. AWS - Amazon Web Service

Leave a Reply

Your email address will not be published. Required fields are marked *