1) Installing spinnaker with easy mode
https://github.com/OpsMx/spinnakersummit-2020/tree/main/spinnaker
helm repo add spinnaker https://helmcharts.opsmx.com/
helm install -n spinnaker spinnaker spinnaker/spinnaker
Chúng ta có nội dụng của file value cần lưu ý. Mình thấy chỉ có minio là ăn cấu hình tolerations và nodeSelector trong file value
minio:
resources:
requests:
memory: "1Gi"
tolerations:
- key: "node"
operator: "Equal"
value: "storage-ssd"
effect: "NoSchedule"
nodeSelector:
node: "storage-ssd"
Con với halyard và redis mình edit manifest đang chay và mình sửa trên Argocd
Bạn edit Halyard trước rồi bạn xoã pod
Tiếp đến là edit redis rồi xoá pod -> làm ntn thì mới halyard nó bắt đầu create các resource spinnaker.
Theo mình nhớ data spinnaker sẽ lưu ở minio hết bạn có thể không cần persistent volume cho halyard và redis thì bạn chọn value bên dưới
chỉnh storageClass cho minio
halyard:
# Set to false to disable persistence data volume for halyard
persistence:
enabled: false
redis:
# Uncomment if you don't want to create a PVC for redis
master:
persistence:
enabled: false
minio:
resources:
requests:
memory: "1Gi"
tolerations:
- key: "node"
operator: "Equal"
value: "storage-ssd"
effect: "NoSchedule"
nodeSelector:
node: "storage-ssd"
persistence:
enabled: true
storageClass: longhorn-fast
Minio thì default là 4Gi nhưng lab mình ko nhiều resource nên mình sửa lại
Mình đã thủ dùng repo bên dưới nhưng install ko được, chắc là do thiếu j đó trong file value
Nó sẽ bắt đâu Install như Hình
Sau tất cả các workload đã Active thì bạn tạo ingress cho spin-deck
khi mình cài spinnaker trên lab thì nó ăn CPU nhiều vãi
Có video cho các bạn tham khảo
2) Installing spinnaker on production environment.
Phần này mình sẽ chia sẻ các bạn cài spinnaker 1 cách chuẩn cơm mẹ nấu và có thể đem bán nhóe.
Mình sẽ dụng kết hợp giữa kustomize và helm chart. Bạn sẽ thao khảo bài bên dưới để hiểu về mindset của mình:
https://nimtechnology.com/2022/05/22/kustomize-2/
2.1) overlays
Mình sẽ có thư mục là overlays/dev
nó nghĩa là overlays/<environment>
kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../base/dev helmGlobals: chartHome: spinnaker-2.2.4/ helmCharts: - name: spinnaker includeCRDs: false releaseName: spin version: 2.2.4 repo: https://helmcharts.opsmx.com/ valuesFile: values_helm.yaml namespace: spin patchesStrategicMerge: - custom-spin-redis-master.yaml
values_helm.yaml
halyard:
# Set to false to disable persistence data volume for halyard
persistence:
enabled: true
storageClass: nfs-client
# Run all commands of spinnaker
# https://spinnaker.io/docs/reference/halyard/commands/
additionalScripts:
enabled: true
configMapName: spin-scripts
configMapKey: config.sh
# create the files at /home/spinnaker/.hal/default/service-settings/
additionalServiceSettings:
clouddriver.yml: |-
env:
JAVA_OPTS: "-Xms4000m -Xmx8000m"
# create the files at /home/spinnaker/.hal/default/profiles/
additionalProfileConfigMaps:
data:
spinnaker-local.yml: |-
logging:
level:
root: ERROR
clouddriver-local.yml: |-
serviceLimits:
cloudProviderOverrides:
kubernetes:
rateLimit: 3.0
kubernetes:
client:
maxErrorRetry: 2
gate-local.yml: |-
server:
tomcat:
protocolHeader: X-Forwarded-Proto
remoteIpHeader: X-Forwarded-For
internalProxies: .*
httpsServerPort: X-Forwarded-Port
kubeConfig:
enabled: true
secretName: kubeconfig-spin
secretKey: config
contexts:
# Names of contexts available in the uploaded kubeconfig
- rke-vdc-infras
- dev
deploymentContext: rke-vdc-infras
dockerRegistries:
- name: gcr-account
address: https://asia.gcr.io
username: _json_key
email: fake.email@spinnaker.io
dockerRegistryAccountSecret: gcr-account
# Google Cloud Storage
gcs:
enabled: true
project: nimtechnology-prod-cicd-b20d
bucket: "nimtechnology-prod-cicd-b20d-spinnaker-config"
## if jsonKey is set, will create a secret containing it
jsonKey: ''
## override the name of the secret to use for jsonKey, if `jsonKey`
## is empty, it will not create a secret assuming you are creating one
## external to the chart. the key for that secret should be `key.json`.
secretName: gcs-account
minio:
enabled: false
redis:
master:
persistence:
enabled: false
custom-spin-redis-master.yaml
Vì là helm chart ko confing resource reques (ram, cpu) nên mình chơi cách merge manifest
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: spin-redis-master
spec:
template:
spec:
containers:
- name: spin-redis
resources:
requests:
cpu: "1"
memory: 4Gi
2.2) bases
tiếp theo mình có 1 folder bash/dev
configmap-spin-script.yaml
-> file này lưu nội dung các file script halyard.
bạn có thể tham khảo các lệnh ở link này https://spinnaker.io/docs/reference/halyard/commands/
apiVersion: v1
data:
config.sh: |-
mv /home/spinnaker/.hal/config-init /home/spinnaker/.hal/config
$HAL_COMMAND config version edit --version 1.22.6
$HAL_COMMAND config security ui edit --override-base-url "https://spinnaker-v2.dev.nimtechnology.services"
$HAL_COMMAND config security api edit --override-base-url "https://spinnaker-v2.dev.nimtechnology.services/gate"
export CLIENT_ID='506853153623-xxxxxxxxxxxxxxx.apps.googleusercontent.com'
export CLIENT_SECRET='GOCSPX-GonOTLY2JxxxxxxxxxxxxXLtIzAJm'
export PROVIDER='google'
export REDIRECT_URL='https://spinnaker-v2.dev.nimtechnology.services/gate/login'
export DOMAIN='/(.*)@(nimtechnology)\.vn$/'
$HAL_COMMAND config security authn oauth2 edit \
--client-id $CLIENT_ID \
--client-secret $CLIENT_SECRET \
--provider $PROVIDER
$HAL_COMMAND config security authn oauth2 edit --pre-established-redirect-uri $REDIRECT_URL
$HAL_COMMAND config security authn oauth2 edit --user-info-requirements email=$DOMAIN
$HAL_COMMAND config security authn oauth2 enable
export TOKEN_FROM_SLACK="xoxb-38868754405-751093996193-xxxxxxxxxxxxx"
export SLACK_BOT=nimtechnology-spinnaker
echo $TOKEN_FROM_SLACK | $HAL_COMMAND config notification slack edit --bot-name \
$SLACK_BOT --token
$HAL_COMMAND config notification slack enable
$HAL_COMMAND config features edit --artifacts true
$HAL_COMMAND config artifact github enable
export GITHUB_ACCOUNT_NAME=nimtechnologyservice
$HAL_COMMAND config artifact github account add ${GITHUB_ACCOUNT_NAME} \
--token ghp_xxxxxxxxxxxxxxxxxxxxxxx
$HAL_COMMAND config features edit --pipeline-templates true
$HAL_COMMAND config features edit --artifacts true
export SERVICE_ACCOUNT_DEST=/opt/gcs/key.json
export ARTIFACT_ACCOUNT_NAME=jenkins-artifacts
$HAL_COMMAND config artifact gcs enable
$HAL_COMMAND config artifact gcs account add $ARTIFACT_ACCOUNT_NAME \
--json-path $SERVICE_ACCOUNT_DEST
export PROJECT_ID=nimtechnology-prod-cicd-b20d
export JSON_PATH=/opt/gcs/key.json
export MY_SPINNAKER_BUCKET=spin-db2ad87b-c484-4f9c-a6f4-fe89450819a8
$HAL_COMMAND config canary enable
$HAL_COMMAND config canary google enable
$HAL_COMMAND config canary google account add spinnaker \
--project $PROJECT_ID \
--json-path $JSON_PATH \
--bucket $MY_SPINNAKER_BUCKET \
--root-folder kayenta
$HAL_COMMAND config canary google edit --gcs-enabled true \
--stackdriver-enabled false
$HAL_COMMAND config deploy component-sizing orca edit \
--container-requests-cpu 512m \
--container-requests-memory 1024Mi
$HAL_COMMAND config deploy component-sizing clouddriver edit \
--container-requests-cpu 1600m \
--container-requests-memory 7000Mi \
--container-limits-cpu 5000m \
--container-limits-memory 8100Mi \
--replicas 2
$HAL_COMMAND config edit --timezone Asia/Ho_Chi_Minh
kind: ConfigMap
metadata:
name: spin-scripts
ingress-spin.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-uat
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
name: ingress-spinnaker
spec:
rules:
- host: spinnaker-v2.dev.nimtechnology.services
http:
paths:
- backend:
service:
name: spin-deck
port:
number: 9000
path: /
pathType: Prefix
- backend:
service:
name: spin-gate
port:
number: 8084
path: /auth
pathType: Prefix
tls:
- hosts:
- spinnaker-v2.dev.nimtechnology.services
secretName: tls-spinnaker-v2.dev.nimtechnology.services
secret-gcr.yaml
—> file này chứa tockent của private docker hub of google.
>>>>>this is content not encrypt
{
"type": "service_account",
"project_id": "nimtechnology-infra-tf",
"private_key_id": "90ab58d76d9eb5fef76a01845e44ca1cf6d6c07c",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwgSF5GYSevDQ6C+qU=\n-----END PRIVATE KEY-----\n",
"client_email": "gcr-pull@nimtechnology-infra-tf.iam.gserviceaccount.com",
"client_id": "111156186675729476362",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gcr-pull%40nimtechnology-infra-tf.iam.gserviceaccount.com"
}
>>>>>>>>>>this is content secret-gcr.yaml
apiVersion: v1
data:
_json_key: ewogICJ0eXBlIjo6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLAogICJhdXRoX3Byb3ZpZGVyX3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vb2F1dGgyL3YxL2NlcnRzIiwKICAiY2xpZW50X3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vcm9ib3QvdjEvbWV0YWRhdGEveDUwOS9nY3ItcHVsbCU0MHRpa2ktaW5mcmEtdGYuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0=
kind: Secret
metadata:
name: gcr-account
type: Opaque
secret-gcs.yaml
–> file storage của gcp
>>>>>this is content not encrypt
{
"type": "service_account",
"project_id": "nimtechnology-prod-cicd-b20d",
"private_key_id": "72dd694fcb8ab3cc3440c5e835924dd4cc9df8e3",
"private_key": "-----BEGIN PRIVATE KEY-----\nMIIEugIBADANBgkqhkiG9w0B6N4URGmL6yv7SGRfGJKSqvYWrpKhyplor/2eSY7qU+cR7EIj\nOBVClmAOPfwPdJQuNw3HrlenMd4I8Qej22Bz7lSdNlJEnwQTCqCZPJqz3rQPDLR4\nDbodhVdHI3LtqD1qurhlFVjR+0UWgwGAI1zxAoGAWWFBXZ4QnH79WmnBtj1ShEyB\nWsc9QWhK4h/d8P4IxzsMag3FWlhCxcUjujKJzKuc/uK6DoPDhR5L//7pRkcROmpC\nqsxxR9luHvZV7wLPFNqOOQenNu1+VJNbBLmBsjXymcmIrtDny47rSgpxQfF2o23R\nHlo/ipXNlyHntlsb5jU=\n-----END PRIVATE KEY-----\n",
"client_email": "spinnaker-gcs@nimtechnology-prod-cicd-b20d.iam.gserviceaccount.com",
"client_id": "104072127438379505554",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/spinnaker-gcs%40nimtechnology-prod-cicd-b20d.iam.gserviceaccount.com"
>>>>>>>>>>this is content secret-gcs.yaml
apiVersion: v1
data:
key.json: ewogICJ00cyIsCiAgImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL3JvYm90L3YxL21ldGFkYXRhL3g1MDkvc3Bpbm5ha2VyLWdjcyU0MHRpa2ktcHJvZC1jaWNkLWIyMGQuaWFtLmdzZXJ2aWNlYWNjb3VudC5jb20iCn0=
kind: Secret
metadata:
labels:
objectset.rio.cattle.io/hash: 8d6bf1c256bb51de9a7a7a76e9200f640239f7af
name: gcs-account
namespace: spin
type: Opaque
Giớ đến file kubeconfig để spinnaker có thể deploy lên k8s
secret-kubeconfig.yaml
>>>>>this is content not encrypt
apiVersion: v1
kind: Config
clusters:
- name: "rke-vdc-infras"
cluster:
server: "https://dev.infra.nimtechnology.services/k8s/clusters/c-4nzfh"
- name: dev
cluster:
server: https://dev.infra.nimtechnology.services/k8s/clusters/c-2tz2k
users:
- name: "rke-vdc-infras"
user:
token: "kubeconfig-u-lcfl6vxlrm:jhbrpr7bzxxxxxxxxxxxxxx5m"
- name: dev
user:
token: kubeconfig-u-q7227fsnhn:f4ghx855p2cc6fl4hcrxxxxxxxxx9qt7pmngmdh
contexts:
- name: "rke-vdc-infras"
context:
user: "rke-vdc-infras"
cluster: "rke-vdc-infras"
- name: dev
context:
user: dev
cluster: dev
current-context: "rke-vdc-infras"
>>>>>>>>>>this is content secret-gcs.yaml
apiVersion: v1
data:
config: YXBpVmVyc2lvbj12ZGMtaW5mcmFzIgogIGNvbnRleHQ6CiAgICB1c2VyOiAicmtlLXZkYy1pbmZyYXMiCiAgICBjbHVzdGVyOiAicmtlLXZkYy1pbmZyYXMiCi0gbmFtZTogZGV2CiAgY29udGV4dDoKICAgIHVzZXI6IGRldgogICAgY2x1c3RlcjogZGV2CgpjdXJyZW50LWNvbnRleHQ6ICJya2UtdmRjLWluZnJhcyI=
kind: Secret
metadata:
name: kubeconfig-spin
namespace: spin
type: Opaque
cúng cúng là file kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ingress-spin.yaml # - secret-kubeconfig.yaml # - configmap-spin-script.yaml # - secret-gcr.yaml # - secret-gcs.yaml
3) Monitor spinnaker
https://github.com/spinnaker/spinnaker-monitoring/tree/master/spinnaker-monitoring-third-party/third_party/prometheus
https://github.com/uneeq-oss/spinnaker-mixin (bạn có thể download release và trong đó có sẵn file json)
