Skip to content

NimTechnology

Trình bày các công nghệ CLOUD một cách dễ hiểu.

  • Kubernetes & Container
    • Docker
    • Kubernetes
      • Ingress
    • Helm Chart
    • Isito-EnvoyFilter
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Vault
    • Longhorn – Storage
    • VictoriaMetrics
    • MetalLB
    • Kong Gateway
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Spinnaker
    • Jenkins
    • Harbor
    • TeamCity
    • Git
      • Bitbucket
  • Coding
    • Terraform
      • GCP – Google Cloud
      • AWS – Amazon Web Service
    • Golang
    • Laravel
    • Python
    • Jquery & JavaScript
    • Selenium
  • Log & Monitor
    • DataDog
    • Prometheus
    • Grafana
    • ELK
      • Kibana
      • Logstash
  • BareMetal
    • NextCloud
  • Toggle search form

[EKS] Upgrade eks cluster by eksctl

Posted on October 29, 2022March 2, 2023 By nim No Comments on [EKS] Upgrade eks cluster by eksctl

Contents

  • 1) Create eks cluter by eksctl
  • 2) Upgrade EKS Cluster.
    • 2.1) Check deprecated
      • 2.1.1) pluto
      • 2.1.2) kubent
    • 2.2) Upgrade Controller EKS
    • 2.3) Create a new node group with a new version.
    • 2.4) Upgrade the important workloads
      • 2.4.1) CoreDNS Plugin
        • Warning
      • 2.4.2) Kube Proxy
      • 2.4.3) VPC CNI Plugin
      • 2.4.4) AWS Load Balancer Controller
      • 2.4.5) Cluster Autoscaler
  • 3) Delete the old node Group.
  • 4) Delete the eks cluster.
  • 5) eksctl without –config-file
  • Error cluster-autoscaler
    • Failed to create AWS Manager
    • Failed to watch *v1.Namespace

1) Create eks cluter by eksctl

Đương nhiên là để lab được bài này thì bạn cần create eks cluster trước hihi.

Đầu tiên chúng ta cài đặt 1 cluster eks bằng eksctl.

cat cluster-1.20.yaml

>>>>>
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: devops-catalog
  region: us-east-2
  version: "1.20"
managedNodeGroups:
- name: primary
  instanceType: t2.small
  minSize: 1
  maxSize: 2
  spot: true

https://eksctl.io/usage/schema/
bạn có thể check thêm key value trên schema.

eksctl create cluster --config-file cluster-1.20.yaml

Bạn có thể vào cloudformation để sem chi tiết nó đang làm cái gì.

GIờ thì nó tạo sẽ tạo worker node.

2) Upgrade EKS Cluster.

2.1) Check deprecated

Find Kubernetes resources that have been deprecated

2.1.1) pluto

https://github.com/FairwindsOps/pluto

mình lab trên ubuntu nên mình sẽ truy cập vào link sau và download Binary

Download về máy

wget https://github.com/FairwindsOps/pluto/releases/download/v5.11.1/pluto_5.11.1_linux_amd64.tar.gz
tar zxf pluto_5.11.1_linux_amd64.tar.gz
mv pluto /usr/local/bin
Thằng Pluto này thì nó quét các file deplpoyment có sẵn.

2.1.2) kubent

Cái này thì nó quét trực tiếp cluster của bạn

https://github.com/doitintl/kube-no-trouble

root@LP11-D7891:~# kubent
10:43AM INF >>> Kube No Trouble `kubent` <<<
10:43AM INF version 0.7.0 (git sha d1bb4e5fd6550b533b2013671aa8419d923ee042)
10:43AM INF Initializing collectors and retrieving data
10:43AM INF Target K8s version is 1.21.14-eks-ffeb93d
10:43AM INF Retrieved 227 resources from collector name=Cluster
10:43AM INF Retrieved 6 resources from collector name="Helm v3"
10:43AM INF Loaded ruleset name=custom.rego.tmpl
10:43AM INF Loaded ruleset name=deprecated-1-16.rego
10:43AM INF Loaded ruleset name=deprecated-1-22.rego
10:43AM INF Loaded ruleset name=deprecated-1-25.rego
10:43AM INF Loaded ruleset name=deprecated-1-26.rego
10:43AM INF Loaded ruleset name=deprecated-future.rego
__________________________________________________________________________________________
>>> Deprecated APIs removed in 1.22 <<<
------------------------------------------------------------------------------------------
KIND                           NAMESPACE          NAME                               API_VERSION                            REPLACE_WITH (SINCE)
CustomResourceDefinition       <undefined>        externalmetrics.metrics.aws        apiextensions.k8s.io/v1beta1           apiextensions.k8s.io/v1 (1.16.0)

giờ bạn thấy là externalmetrics.metrics.aws và KIND: CustomResourceDefinition bị outdated.

bạn sử dụng command sàu để edit:

kubectl edit CustomResourceDefinition/externalmetrics.metrics.aws

2.2) Upgrade Controller EKS

eksctl upgrade cluster --config-file cluster-1.21.yaml --approve

Giờ bạn sẽ upgrade eks cluster.
dưới đây là file mới

cat cluster-1.21.yaml

#####
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: devops-catalog
  region: us-east-2
  version: "1.21"
managedNodeGroups:
- name: primary-1-21
  instanceType: t2.small
  minSize: 1
  maxSize: 2
  spot: true

Bạn sẽ thấy nó mới chỉ upgrade cài controller.

Còn các node worker thì chưa được upgrade.

root@work-space-u20:~/eks/eksctl-demo# kubectl get node
NAME                                           STATUS   ROLES    AGE    VERSION
ip-192-168-42-164.us-east-2.compute.internal   Ready    <none>   177m   v1.20.15-eks-ba74326

2.3) Create a new node group with a new version.

Giờ bạn cần phải tạo new node group

eksctl create nodegroup --config-file cluster-1.21.yaml
Bạn sẽ thấy nó tạo 1 node group mới
Thấy là anh instance trên EC2
Đây là ảnh bên auto scaling Group.

Nếu bạn muốn tìm hiểu về set max min instance thì tham khảo link này:
http://arun-gupta.github.io/update-eks-cluster/

Bạn muốn tìm hiều spot instance thì link này:
https://eksctl.io/usage/spot-instances/

2.4) Upgrade the important workloads

2.4.1) CoreDNS Plugin

Giờ kiểm tra CoreDNS Plugin:

https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html

kubectl --namespace kube-system describe deployment coredns | grep Image | cut -d "/" -f 3
coredns:v1.8.3-eksbuild.1

Nếu CoreDNS của bản dưới v1.8.3 và muốn upgrade coredns lên version mới hơn 1.8.3

bạn cần config những thứ sau:

kubectl edit clusterrole system:coredns -n kube-system

##add this into content.
...
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
...

của mình đang ở version 1.8.3 thì change image thôi.

kubectl --namespace kube-system set image deployment.apps/coredns coredns=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.8.4-eksbuild.1

làm sao để tim được tag “v1.8.4-eksbuild.1”
1) bạn search quanh trên mạng
2) tạo 1 con Ec2 rồi vào con Ec2 đó pull thử.

Warning

Nếu bạn gặp warning:
Warning – beta.kubernetes.io/os is deprecated since v1.14; use “kubernetes.io/os” instead
bạn đừng lo lắng cứ ignored
https://github.com/coredns/helm/issues/83

2.4.2) Kube Proxy

Kiểm tra Kube Proxy

kubectl -n kube-system get daemonset kube-proxy -o=jsonpath='{$.spec.template.spec.containers[:1].image}'
602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy:v1.20.7-eksbuild.1

https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html

kubectl -n kube-system set image daemonset.apps/kube-proxy kube-proxy=602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.21.14-eksbuild.2

2.4.3) VPC CNI Plugin

Kiểm tra VPC CNI Plugin

kubectl -n kube-system describe daemonset aws-node | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.7.5-eksbuild.1
amazon-k8s-cni:v1.7.5-eksbuild.1

Ở bạn 1 .21 thì họ khuyên như sau:

Bạn có thể vào link này để chọn manifest thích hợp.
https://github.com/aws/amazon-vpc-cni-k8s/releases
cứ chọn latest

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/v1.11.4/config/master/aws-k8s-cni.yaml
root@work-space-u20:~/eks/eksctl-demo# kubectl describe daemonset aws-node -n kube-system | grep Image | cut -d "/" -f 2 
amazon-k8s-cni-init:v1.11.4
amazon-k8s-cni:v1.11.4

2.4.4) AWS Load Balancer Controller

Khi upgrade lên eks 1.22 thì gặp lỗi sau trên aws-load-balancer

{"level":"error","ts":1676453510.3522048,"logger":"setup","msg":"unable to create controller","controller":"Ingress","error":"the server could not find the requested resource"}

Bạn cần access vào link dưới để download file yaml:
https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases

Bạn download manifest này xuống.

bạn edit Deployment có tên là aws-load-balancer-controller –> –cluster-name=your-cluster-name

Sau khi sửa xong thì bạn cứ apply thôi.

2.4.5) Cluster Autoscaler

bạn cứ lấy version mới nhất

https://github.com/kubernetes/autoscaler/releases

Câu lệnh change version.

 kubectl describe -n kube-system deployment.apps/cluster-autoscaler | grep Image
    Image:      us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.21.3

kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.3
deployment.apps/cluster-autoscaler image updated

3) Delete the old node Group.


eksctl delete nodegroup --config-file cluster-1.21.yaml --only-missing --approve

Chỗ này khá quan trọng:
–only-missing: bạn sẽ detele các node group mà version không giống như trong config file.

4) Delete the eks cluster.

eksctl delete cluster --config-file cluster-1.21.yaml --wait

5) eksctl without –config-file

Nếu động đội của bạn chỉ đưa cho bạn 1 con eks và nhờ bạn upgrade thì:

https://www.stacksimplify.com/aws-eks/eks-cluster/delete-eks-cluster-nodegroup/

eksctl upgrade cluster --name devops-catalog \
--region us-east-2  \
--version 1.22 \
--approve

eksctl create nodegroup --cluster devops-catalog \
--region us-east-2

eksctl get nodegroup --cluster devops-catalog --region us-east-2

Giờ delete node group cũ:

eksctl delete nodegroup --cluster devops-catalog --region us-east-2 --name primary-1-21
eksctl delete cluster --name eks-nimtechnology --region us-east-2 --wait

Bạn có thê tham khảo bài dưới:
https://rharshad.com/aws-eks-upgrade-1.19-to-1.20/

Error cluster-autoscaler

Failed to create AWS Manager

F1031 05:42:07.134307       1 aws_cloud_provider.go:382] Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: arn:aws:sts::xxxxxxxx:assumed-role/eksctl-nimtechnology-nodegr-NodeInstanceRole-FCSNPINNUMJV/i-0f3eb129a6040a5a3 is not authorized to perform: autoscaling:DescribeTags because no identity-based policy allows the autoscaling:DescribeTags action

You must create a new policy following the below config and then attach this policy to the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeInstanceTypes"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}
Bạn click và tạo 1 policy mới

Failed to watch *v1.Namespace

Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "namespaces" in API group "" at the cluster scope

How to fix:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml#L35-L43

AWS - Amazon Web Service

Post navigation

Previous Post: [Bitbucket Pipeline] The deployment environment ‘xxx’ in your bitbucket-pipelines.yml file occurs multiple times in the pipeline. Please refer to our documentation for valid environments and their ordering.
Next Post: [DevSecOps] Application Security Testing Tools

More Related Articles

[eks] Kubernetes Cluster-AutoScaler error: Failed to watch *v1.CSIDriver AWS - Amazon Web Service
[EKS] the exciting and helpful things about EKS AWS - Amazon Web Service
[MongoDB] Creating MongoDB Atlas to integrate with your workload on any Cloud AWS - Amazon Web Service
[EKS]networkPlugin cni failed to set up pod “pod—xxxxx” network: add cmd: failed to assign an IP address to container AWS - Amazon Web Service
[AWS] Provision AWS IAM Admin User as EKS Admin AWS - Amazon Web Service
[Metrics Server] Install metrics-server on Kubernetes. AWS - Amazon Web Service

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Tham Gia Group DevOps nhé!
Để Nim có nhiều động lực ra nhiều bài viết.
Để nhận được những thông báo mới nhất.

Recent Posts

  • Experiences for IP Addresses Shortage on EKS Clusters March 29, 2023
  • [Talisman] Discover the sensitive information in your code. March 28, 2023
  • [Prometheus/Grafana] Install Prometheus and Grafana on ubuntu. March 27, 2023
  • [Kong Gateway] WebSocket connection failed March 26, 2023
  • [Nextcloud] Can’t download files to have a size bigger than 2Gi on NextCloud – RaspBerry March 24, 2023

Archives

  • March 2023
  • February 2023
  • January 2023
  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • July 2021
  • June 2021

Categories

  • BareMetal
    • NextCloud
  • CI/CD
    • ArgoCD
    • ArgoWorkflows
    • Git
      • Bitbucket
    • Harbor
    • Jenkins
    • Spinnaker
    • TeamCity
  • Coding
    • Golang
    • Jquery & JavaScript
    • Laravel
    • Python
    • Selenium
    • Terraform
      • AWS – Amazon Web Service
      • GCP – Google Cloud
  • Kubernetes & Container
    • Apache Kafka
      • Kafka
      • Kafka Connect
      • Lenses
    • Docker
    • Helm Chart
    • Isito-EnvoyFilter
    • Kong Gateway
    • Kubernetes
      • Ingress
    • Longhorn – Storage
    • MetalLB
    • Vault
    • VictoriaMetrics
  • Log & Monitor
    • DataDog
    • ELK
      • Kibana
      • Logstash
    • Grafana
    • Prometheus
  • Uncategorized
  • Admin

Copyright © 2023 NimTechnology.