[AWS] Creat Persistent Volume on EKS via EBS.

1) Overview

2) Demo

2.1) Create EKS cluster

Bạn sẽ cần tạo EKS cluster
https://github.com/mrnim94/terraform-aws/tree/master/eks/AWS-EKS-Cluster-Basics

2.2) EBS-CSI and Install Kubernetes Storage

Bược này chúng ta sẽ EBS CSI

Đầu tiên chúng ta có file: c4-01-ebs-csi-datasources.tf

# Datasource: EBS CSI IAM Policy get from EBS GIT Repo (latest)
data "http" "ebs_csi_iam_policy" {
  url = "https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/docs/example-iam-policy.json"

  # Optional request headers
  request_headers = {
    Accept = "application/json"
  }
}

output "ebs_csi_iam_policy" {
  value = data.http.ebs_csi_iam_policy.body
}

Bạn có thể khám phá file này.
https://raw.githubusercontent.com/kubernetes-sigs/aws-ebs-csi-driver/master/docs/example-iam-policy.json

File trên là policy để add và ec2 tương tác với storage.

Bước tiếp theo là:
Create EBS CSI IAM Policy
mục địch của việc này để workload trong eks có thể có quyền action trên resource của AWS.

#data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn
#data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn

# Resource: Create EBS CSI IAM Policy 
resource "aws_iam_policy" "ebs_csi_iam_policy" {
  name        = "${local.name}-AmazonEKS_EBS_CSI_Driver_Policy"
  path        = "/"
  description = "EBS CSI IAM Policy"
  policy = data.http.ebs_csi_iam_policy.body
}

output "ebs_csi_iam_policy_arn" {
  value = aws_iam_policy.ebs_csi_iam_policy.arn 
}

# Resource: Create IAM Role and associate the EBS IAM Policy to it
resource "aws_iam_role" "ebs_csi_iam_role" {
  name = "${local.name}-ebs-csi-iam-role"

  # Terraform's "jsonencode" function converts a Terraform expression result to valid JSON syntax.
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Federated = "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_arn}"
        }
        Condition = {
          StringEquals = {            
            "${data.terraform_remote_state.eks.outputs.aws_iam_openid_connect_provider_extract_from_arn}:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
          }
        }        

      },
    ]
  })

  tags = {
    tag-key = "${local.name}-ebs-csi-iam-role"
  }
}

# Associate EBS CSI IAM Policy to EBS CSI IAM Role
resource "aws_iam_role_policy_attachment" "ebs_csi_iam_role_policy_attach" {
  policy_arn = aws_iam_policy.ebs_csi_iam_policy.arn 
  role       = aws_iam_role.ebs_csi_iam_role.name
}

output "ebs_csi_iam_role_arn" {
  description = "EBS CSI IAM Role ARN"
  value = aws_iam_role.ebs_csi_iam_role.arn
}

EKS Cluster Auth

cài này là để terraform tương tác được eks cluster.

# Datasource: EKS Cluster Auth 
data "aws_eks_cluster_auth" "cluster" {
  name = data.terraform_remote_state.eks.outputs.cluster_id
}

# HELM Provider
provider "helm" {
  kubernetes {
    host                   = data.terraform_remote_state.eks.outputs.cluster_endpoint
    cluster_ca_certificate = base64decode(data.terraform_remote_state.eks.outputs.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}

Install EBS CSI Driver using HELM

# Install EBS CSI Driver using HELM
# Resource: Helm Release 
resource "helm_release" "ebs_csi_driver" {
  depends_on = [aws_iam_role.ebs_csi_iam_role]
  name       = "${local.name}-aws-ebs-csi-driver"
  repository = "https://kubernetes-sigs.github.io/aws-ebs-csi-driver"
  chart      = "aws-ebs-csi-driver"
  namespace = "kube-system"     

  set {
    name = "image.repository"
    value = "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/aws-ebs-csi-driver" # Changes based on Region - This is for us-east-1 Additional Reference: https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html
  }       

  set {
    name  = "controller.serviceAccount.create"
    value = "true"
  }

  set {
    name  = "controller.serviceAccount.name"
    value = "ebs-csi-controller-sa"
  }

  set {
    name  = "controller.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = "${aws_iam_role.ebs_csi_iam_role.arn}"
  }
    
}

Link tham khảo terraform EBS CSI
https://github.com/mrnim94/terraform-aws/tree/master/ebs-terraform-manifests

kubectl get all -n kube-system

2.3) Create Deployment that will use Persistent Volume Claim

2.3.1) STORAGE CLASS

Đầu tiên là tạo STORAGE CLASS

01-storage-class.yaml
>>>>>>
>>>>>

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: 
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer 


# STORAGE CLASS 
# 1. A StorageClass provides a way for administrators 
# to describe the "classes" of storage they offer.
# 2. Here we are offering EBS Storage for EKS Cluster

Bạn để ý là chúng ta sẽ sử dụng provisioner: ebs.csi.aws.com

ngoài ra bạn cũng có thể chọn các type ebs
References: Persistent Storage in EKS failing to provision volume

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  iops: "3000"
  throughput: "125"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

volumeBindingMode: WaitForFirstConsumer
–> Nghĩa là bạn tạo PVC sau đó nó sẽ pending -> khi nào tạo pod và sử dụng PVC thì nó mới tạo PV

This link has many parameters to choose
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/parameters.md

2.3.2) Create Persistent volume claim.

02-persistent-volume-claim.yaml
>>>>>
>>>>>>>

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-mysql-pv-claim
spec: 
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources: 
    requests:
      storage: 4Gi

# NEED FOR PVC
# 1. Dynamic volume provisioning allows storage volumes to be created 
# on-demand. 

# 2. Without dynamic provisioning, cluster administrators have to manually 
# make calls to their cloud or storage provider to create new storage 
# volumes, and then create PersistentVolume objects to represent them in k8s

# 3. The dynamic provisioning feature eliminates the need for cluster 
# administrators to pre-provision storage. Instead, it automatically 
# provisions storage when it is requested by users.

# 4. PVC: Users request dynamically provisioned storage by including 
# a storage class in their PersistentVolumeClaim

lúc này pvc vẫn đang pending

2.3.3) Create deployment based on PVC

apiVersion: v1
kind: ConfigMap
metadata:
  name: usermanagement-dbcreation-script
data: 
  mysql_usermgmt.sql: |-
    DROP DATABASE IF EXISTS webappdb;
    CREATE DATABASE webappdb; 


# CONFIG MAP
# 1. A ConfigMap is an API object used to store non-confidential data in 
# key-value pairs. 

# 2. Pods can consume ConfigMaps as 
## 2.1: environment variables, 
## 2.2: command-line arguments, 
## 2.3: or as configuration files in a volume. We are going to use this in our MySQL Deployment)  


apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec: 
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate 
  template: 
    metadata: 
      labels: 
        app: mysql
    spec: 
      containers:
        - name: mysql
          image: mysql:5.6
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: dbpassword11
          ports:
            - containerPort: 3306
              name: mysql    
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql    
            - name: usermanagement-dbcreation-script
              mountPath: /docker-entrypoint-initdb.d #https://hub.docker.com/_/mysql Refer Initializing a fresh instance                                            
      volumes: 
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: ebs-mysql-pv-claim
        - name: usermanagement-dbcreation-script
          configMap:
            name: usermanagement-dbcreation-script


# VERY IMPORTANT POINTS ABOUT CONTAINERS AND POD VOLUMES: 
## 1. On-disk files in a container are ephemeral
## 2. One problem is the loss of files when a container crashes. 
## 3. Kubernetes Volumes solves above two as these volumes are configured to POD and not container. 
## Only they can be mounted in Container
## 4. Option-1: Using AWS EBS CSI is a super generalized approach 
## for having Persistent Volumes for workloads in Kubernetes
## 5. Option-2: The other approach is to manually create EBS Volume 
## and mount it in pod "spec.volumes.awsElasticBlockStore" (NOT RECOMMENDED)
## https://kubernetes.io/docs/concepts/storage/volumes/#aws-ebs-configuration-example


## ENVIRONMENT VARIABLES
# 1. When you create a Pod, you can set environment variables for the 
# containers that run in the Pod. 
# 2. To set environment variables, include the env or envFrom field in 
# the configuration file.

Sau khi deployment các workload test thì thấy

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: "Retain"

# STORAGE CLASS
# 1. A StorageClass provides a way for administrators
# to describe the "classes" of storage they offer.
# 2. Here we are offering EBS Storage for EKS Cluster

Why change reclaim policy of a PersistentVolume

PersistentVolumes can have various reclaim policies, including “Retain”, “Recycle”, and “Delete”. For dynamically provisioned PersistentVolumes, the default reclaim policy is “Delete”. This means that a dynamically provisioned volume is automatically deleted when a user deletes the corresponding PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume contains precious data. In that case, it is more appropriate to use the “Retain” policy. With the “Retain” policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume will not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.

Nếu storage class đang được sử dụng mà bạn change là sẽ bị lỗi

WebApp# kubectl apply -f 01-storage-class.yaml
The StorageClass "ebs-sc" is invalid: reclaimPolicy: Forbidden: updates to reclaimPolicy are forbidden.

Bạn sẽ cần gỡ storage class và persistentVolumeClain hết và tạo lại storage class

và giờ mình deploy lại.

và nếu trong storage class có cấu hình thứ này:

allowVolumeExpansion: true

Thì bạn có thể thay đổi dụng lượng của pvc bằng cách sửa manifest và apply.

Can not resize the persistentvolumeclaims

persistentvolumeclaims "pvc-networkdisk-smbshare" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

Nếu bị lỗi trên bạn sẽ cần add thêm allowVolumeExpansion: true trong storageclass

Done pvc nhé!

2.4) Using module to deploy EBS-CSI on EKS

mình có tự viết 1 module như trong link này

https://registry.terraform.io/modules/mrnim94/eks-ebs-csi/aws/latest
bạn có thể tham khảo vả sử dụng

3) Research the additional information about EBS

Bạn cũng đã có thể biêt là EBS có 2 volume type phổ biến là gp2 và gp3
Vậy chúng tta cũng tìm hiểu nên sài thằng nào thi ngon.
https://www.cloudforecast.io/blog/amazon-ebs-gp2-vs-gp3-volumes/

trong môi trường Goverment có hỗ trợ EBS
https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-ebs.html
XXX The Provisioned IOPS SSD (io2) EBS volume type is not available.

Trong này có chưa Volume Type của EBS
https://aws.amazon.com/ebs/pricing/