[Spinnaker] Install Spinnaker on Kubernetes through Helm from OpsMx

Video cho anh em dễ tưởng tượng

Contents

1) Installing spinnaker with easy mode
2) Installing spinnaker on production environment.
- 2.1) overlays
- 2.2) bases
3) Monitor spinnaker

1) Installing spinnaker with easy mode

https://github.com/OpsMx/spinnakersummit-2020/tree/main/spinnaker

helm repo add spinnaker https://helmcharts.opsmx.com/
helm install -n spinnaker spinnaker spinnaker/spinnaker

Chúng ta có nội dụng của file value cần lưu ý. Mình thấy chỉ có minio là ăn cấu hình tolerations và nodeSelector trong file value

minio:
  resources:
    requests:
      memory: "1Gi"
  tolerations:
    - key: "node"
      operator: "Equal"
      value: "storage-ssd"
      effect: "NoSchedule"
  nodeSelector:
    node: "storage-ssd"

Con với halyard và redis mình edit manifest đang chay và mình sửa trên Argocd

Bạn edit Halyard trước rồi bạn xoã pod
Tiếp đến là edit redis rồi xoá pod -> làm ntn thì mới halyard nó bắt đầu create các resource spinnaker.

Sau mà đã edit statefulset thì phải delete pod để nó chuyển qua node chỉ định

Theo mình nhớ data spinnaker sẽ lưu ở minio hết bạn có thể không cần persistent volume cho halyard và redis thì bạn chọn value bên dưới
chỉnh storageClass cho minio

halyard:
  # Set to false to disable persistence data volume for halyard
  persistence:
    enabled: false
redis:
# Uncomment if you don't want to create a PVC for redis
  master:
    persistence:
      enabled: false
minio:
  resources:
    requests:
      memory: "1Gi"
  tolerations:
    - key: "node"
      operator: "Equal"
      value: "storage-ssd"
      effect: "NoSchedule"
  nodeSelector:
    node: "storage-ssd"
  persistence:
    enabled: true
    storageClass: longhorn-fast

Minio thì default là 4Gi nhưng lab mình ko nhiều resource nên mình sửa lại

Mình đã thủ dùng repo bên dưới nhưng install ko được, chắc là do thiếu j đó trong file value

Nó sẽ bắt đâu Install như Hình

Sau tất cả các workload đã Active thì bạn tạo ingress cho spin-deck

khi mình cài spinnaker trên lab thì nó ăn CPU nhiều vãi

Có video cho các bạn tham khảo

2) Installing spinnaker on production environment.

Phần này mình sẽ chia sẻ các bạn cài spinnaker 1 cách chuẩn cơm mẹ nấu và có thể đem bán nhóe.

Mình sẽ dụng kết hợp giữa kustomize và helm chart. Bạn sẽ thao khảo bài bên dưới để hiểu về mindset của mình:
https://nimtechnology.com/2022/05/22/kustomize-2/

2.1) overlays

Mình sẽ có thư mục là overlays/dev
nó nghĩa là overlays/<environment>

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base/dev

helmGlobals:
  chartHome: spinnaker-2.2.4/

helmCharts:
- name: spinnaker
  includeCRDs: false
  releaseName: spin
  version: 2.2.4
  repo: https://helmcharts.opsmx.com/
  valuesFile: values_helm.yaml
  namespace: spin

patchesStrategicMerge:
  - custom-spin-redis-master.yaml

values_helm.yaml

halyard:
  # Set to false to disable persistence data volume for halyard
  persistence:
    enabled: true
    storageClass: nfs-client
  # Run all commands of spinnaker
  # https://spinnaker.io/docs/reference/halyard/commands/
  additionalScripts:
    enabled: true
    configMapName: spin-scripts
    configMapKey: config.sh
  # create the files at /home/spinnaker/.hal/default/service-settings/
  additionalServiceSettings:
    clouddriver.yml: |-
      env:
        JAVA_OPTS: "-Xms4000m -Xmx8000m"
  # create the files at /home/spinnaker/.hal/default/profiles/
  additionalProfileConfigMaps:
    data:
      spinnaker-local.yml: |-
        logging:
          level:
            root: ERROR
      clouddriver-local.yml: |-
        serviceLimits:
          cloudProviderOverrides:
            kubernetes:
              rateLimit: 3.0
        kubernetes:
          client:
            maxErrorRetry: 2
      gate-local.yml: |-
        server:
          tomcat:
            protocolHeader: X-Forwarded-Proto
            remoteIpHeader: X-Forwarded-For
            internalProxies: .*
            httpsServerPort: X-Forwarded-Port

kubeConfig:
  enabled: true
  secretName: kubeconfig-spin
  secretKey: config
  contexts:
  # Names of contexts available in the uploaded kubeconfig
  - rke-vdc-infras
  - dev
  deploymentContext: rke-vdc-infras

dockerRegistries:
- name: gcr-account
  address: https://asia.gcr.io
  username: _json_key
  email: fake.email@spinnaker.io
dockerRegistryAccountSecret: gcr-account

# Google Cloud Storage
gcs:
  enabled: true
  project: nimtechnology-prod-cicd-b20d
  bucket: "nimtechnology-prod-cicd-b20d-spinnaker-config"
  ## if jsonKey is set, will create a secret containing it
  jsonKey: ''
  ## override the name of the secret to use for jsonKey, if `jsonKey`
  ## is empty, it will not create a secret assuming you are creating one
  ## external to the chart. the key for that secret should be `key.json`.
  secretName: gcs-account


minio:
  enabled: false

redis:
  master:
    persistence:
      enabled: false

custom-spin-redis-master.yaml
Vì là helm chart ko confing resource reques (ram, cpu) nên mình chơi cách merge manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: spin-redis-master
spec:
  template:
    spec:
      containers:
        - name: spin-redis
          resources:
            requests:
              cpu: "1"
              memory: 4Gi

2.2) bases

tiếp theo mình có 1 folder bash/dev

configmap-spin-script.yaml
-> file này lưu nội dung các file script halyard.
bạn có thể tham khảo các lệnh ở link này https://spinnaker.io/docs/reference/halyard/commands/

apiVersion: v1
data:
  config.sh: |-
    mv /home/spinnaker/.hal/config-init /home/spinnaker/.hal/config

    $HAL_COMMAND  config version edit --version 1.22.6

    $HAL_COMMAND config security ui edit --override-base-url "https://spinnaker-v2.dev.nimtechnology.services"
    $HAL_COMMAND config security api edit --override-base-url "https://spinnaker-v2.dev.nimtechnology.services/gate"

    export CLIENT_ID='506853153623-xxxxxxxxxxxxxxx.apps.googleusercontent.com'
    export CLIENT_SECRET='GOCSPX-GonOTLY2JxxxxxxxxxxxxXLtIzAJm'
    export PROVIDER='google'
    export REDIRECT_URL='https://spinnaker-v2.dev.nimtechnology.services/gate/login'
    export DOMAIN='/(.*)@(nimtechnology)\.vn$/'

    $HAL_COMMAND config security authn oauth2 edit \
      --client-id $CLIENT_ID \
      --client-secret $CLIENT_SECRET \
      --provider $PROVIDER
    $HAL_COMMAND config security authn oauth2 edit --pre-established-redirect-uri $REDIRECT_URL
    $HAL_COMMAND config security authn oauth2 edit --user-info-requirements  email=$DOMAIN
    $HAL_COMMAND config security authn oauth2 enable

    export TOKEN_FROM_SLACK="xoxb-38868754405-751093996193-xxxxxxxxxxxxx"
    export SLACK_BOT=nimtechnology-spinnaker
    echo $TOKEN_FROM_SLACK | $HAL_COMMAND config notification slack edit --bot-name \
      $SLACK_BOT --token
    $HAL_COMMAND config notification slack enable

    $HAL_COMMAND config features edit --artifacts true
    $HAL_COMMAND config artifact github enable
    export GITHUB_ACCOUNT_NAME=nimtechnologyservice
    $HAL_COMMAND config artifact github account add ${GITHUB_ACCOUNT_NAME} \
        --token ghp_xxxxxxxxxxxxxxxxxxxxxxx
    $HAL_COMMAND config features edit --pipeline-templates true
    $HAL_COMMAND config features edit --artifacts true

    export SERVICE_ACCOUNT_DEST=/opt/gcs/key.json
    export ARTIFACT_ACCOUNT_NAME=jenkins-artifacts
    $HAL_COMMAND config artifact gcs enable
    $HAL_COMMAND config artifact gcs account add $ARTIFACT_ACCOUNT_NAME \
        --json-path $SERVICE_ACCOUNT_DEST

    export PROJECT_ID=nimtechnology-prod-cicd-b20d
    export JSON_PATH=/opt/gcs/key.json 
    export MY_SPINNAKER_BUCKET=spin-db2ad87b-c484-4f9c-a6f4-fe89450819a8
    $HAL_COMMAND config canary enable 
    $HAL_COMMAND config canary google enable 
    $HAL_COMMAND config canary google account add spinnaker \
      --project $PROJECT_ID \
      --json-path $JSON_PATH \
      --bucket $MY_SPINNAKER_BUCKET \
      --root-folder kayenta
    $HAL_COMMAND config canary google edit --gcs-enabled true \
      --stackdriver-enabled false

    $HAL_COMMAND config deploy component-sizing orca edit \
      --container-requests-cpu 512m \
      --container-requests-memory 1024Mi

    $HAL_COMMAND config deploy component-sizing clouddriver edit \
      --container-requests-cpu 1600m \
      --container-requests-memory 7000Mi \
      --container-limits-cpu 5000m \
      --container-limits-memory 8100Mi \
      --replicas 2

    $HAL_COMMAND config edit --timezone Asia/Ho_Chi_Minh

    
kind: ConfigMap
metadata:
  name: spin-scripts

ingress-spin.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-uat
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  name: ingress-spinnaker
spec:
  rules:
  - host: spinnaker-v2.dev.nimtechnology.services
    http:
      paths:
      - backend:
          service:
            name: spin-deck
            port:
              number: 9000
        path: /
        pathType: Prefix
      - backend:
          service:
            name: spin-gate
            port:
              number: 8084
        path: /auth
        pathType: Prefix
  tls:
  - hosts:
    - spinnaker-v2.dev.nimtechnology.services
    secretName: tls-spinnaker-v2.dev.nimtechnology.services

secret-gcr.yaml
—> file này chứa tockent của private docker hub of google.

>>>>>this is content not encrypt
{
  "type": "service_account",
  "project_id": "nimtechnology-infra-tf",
  "private_key_id": "90ab58d76d9eb5fef76a01845e44ca1cf6d6c07c",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwgSF5GYSevDQ6C+qU=\n-----END PRIVATE KEY-----\n",
  "client_email": "gcr-pull@nimtechnology-infra-tf.iam.gserviceaccount.com",
  "client_id": "111156186675729476362",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gcr-pull%40nimtechnology-infra-tf.iam.gserviceaccount.com"
}

>>>>>>>>>>this is content secret-gcr.yaml
apiVersion: v1
data:
  _json_key: ewogICJ0eXBlIjo6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLAogICJhdXRoX3Byb3ZpZGVyX3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vb2F1dGgyL3YxL2NlcnRzIiwKICAiY2xpZW50X3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vcm9ib3QvdjEvbWV0YWRhdGEveDUwOS9nY3ItcHVsbCU0MHRpa2ktaW5mcmEtdGYuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0=
kind: Secret
metadata:
  name: gcr-account
type: Opaque

secret-gcs.yaml
–> file storage của gcp

>>>>>this is content not encrypt
{
  "type": "service_account",
  "project_id": "nimtechnology-prod-cicd-b20d",
  "private_key_id": "72dd694fcb8ab3cc3440c5e835924dd4cc9df8e3",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEugIBADANBgkqhkiG9w0B6N4URGmL6yv7SGRfGJKSqvYWrpKhyplor/2eSY7qU+cR7EIj\nOBVClmAOPfwPdJQuNw3HrlenMd4I8Qej22Bz7lSdNlJEnwQTCqCZPJqz3rQPDLR4\nDbodhVdHI3LtqD1qurhlFVjR+0UWgwGAI1zxAoGAWWFBXZ4QnH79WmnBtj1ShEyB\nWsc9QWhK4h/d8P4IxzsMag3FWlhCxcUjujKJzKuc/uK6DoPDhR5L//7pRkcROmpC\nqsxxR9luHvZV7wLPFNqOOQenNu1+VJNbBLmBsjXymcmIrtDny47rSgpxQfF2o23R\nHlo/ipXNlyHntlsb5jU=\n-----END PRIVATE KEY-----\n",
  "client_email": "spinnaker-gcs@nimtechnology-prod-cicd-b20d.iam.gserviceaccount.com",
  "client_id": "104072127438379505554",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/spinnaker-gcs%40nimtechnology-prod-cicd-b20d.iam.gserviceaccount.com"

>>>>>>>>>>this is content secret-gcs.yaml

apiVersion: v1
data:
  key.json: ewogICJ00cyIsCiAgImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL3JvYm90L3YxL21ldGFkYXRhL3g1MDkvc3Bpbm5ha2VyLWdjcyU0MHRpa2ktcHJvZC1jaWNkLWIyMGQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0=
kind: Secret
metadata:
  labels:
    objectset.rio.cattle.io/hash: 8d6bf1c256bb51de9a7a7a76e9200f640239f7af
  name: gcs-account
  namespace: spin
type: Opaque

Giớ đến file kubeconfig để spinnaker có thể deploy lên k8s
secret-kubeconfig.yaml

>>>>>this is content not encrypt
apiVersion: v1
kind: Config
clusters:
- name: "rke-vdc-infras"
  cluster:
    server: "https://dev.infra.nimtechnology.services/k8s/clusters/c-4nzfh"
- name: dev
  cluster:
    server: https://dev.infra.nimtechnology.services/k8s/clusters/c-2tz2k

users:
- name: "rke-vdc-infras"
  user:
    token: "kubeconfig-u-lcfl6vxlrm:jhbrpr7bzxxxxxxxxxxxxxx5m"
- name: dev
  user:
    token: kubeconfig-u-q7227fsnhn:f4ghx855p2cc6fl4hcrxxxxxxxxx9qt7pmngmdh


contexts:
- name: "rke-vdc-infras"
  context:
    user: "rke-vdc-infras"
    cluster: "rke-vdc-infras"
- name: dev
  context:
    user: dev
    cluster: dev

current-context: "rke-vdc-infras"

>>>>>>>>>>this is content secret-gcs.yaml
apiVersion: v1
data:
  config: YXBpVmVyc2lvbj12ZGMtaW5mcmFzIgogIGNvbnRleHQ6CiAgICB1c2VyOiAicmtlLXZkYy1pbmZyYXMiCiAgICBjbHVzdGVyOiAicmtlLXZkYy1pbmZyYXMiCi0gbmFtZTogZGV2CiAgY29udGV4dDoKICAgIHVzZXI6IGRldgogICAgY2x1c3RlcjogZGV2CgpjdXJyZW50LWNvbnRleHQ6ICJya2UtdmRjLWluZnJhcyI=
kind: Secret
metadata:
  name: kubeconfig-spin
  namespace: spin
type: Opaque

cúng cúng là file kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ingress-spin.yaml
  # - secret-kubeconfig.yaml
  # - configmap-spin-script.yaml
  # - secret-gcr.yaml
  # - secret-gcs.yaml