[ArgoCD/KSOPS/AWS] Encrypt secrets before pushing them to GitHub.

Posted on By nim No Comments on [ArgoCD/KSOPS/AWS] Encrypt secrets before pushing them to GitHub.

Trước đây mình có 1 bài hướng dẫn các bạn encypt 1 file manifest bất kì trước khi đẩy lên github và sau đó thì Argocd sẽ decrypt manifest và apply to k8s

[ArgoCD/KSOPS] Encrypting Resource on kustomize and Argocd.
Bài này thì nó chúng ta không depend vào aws hay google để encypt hoặc decypt data.

Ksop with KMS on Argocd

Lần này chúng ta sẽ sử dụng KSOPS và KMS của aws.

Chúng ta cần cài sops

Cách cũ:

###-->>> https://pypi.org/project/sops/

apt-get update -y
sudo apt-get install gcc git libffi-dev libssl-dev libyaml-dev make openssl python3-dev python3-pip
sudo pip install --upgrade sops

Cách mới

SOPS_LATEST_VERSION=$(curl -s "https://api.github.com/repos/getsops/sops/releases/latest" | grep -Po '"tag_name": "v\K[0-9.]+')
curl -Lo sops.deb "https://github.com/getsops/sops/releases/latest/download/sops_${SOPS_LATEST_VERSION}_amd64.deb"
sudo apt --fix-broken install ./sops.deb
rm -rf sops.deb
sops -version
Bạn cần tạo trước KMS trên AWS

Bạn cần tạo file config cho SOPS

cat <<EOF > ./.sops.yaml
creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    kms: arn:aws:kms:us-west-2:2508876XXXX:key/c2affea6-1a23-4730-811c-4e00f71b4e1d
    #aws_profile: default #not working
EOF

aws_profile là profile trong credential của aws.
Rất tiếc là không thể sử dụng key là aws_profile

I can not use sops with aws_profile, an issue is open 
https://github.com/mozilla/sops/issues/439
https://www.bardina.net/sops-aws-kms-multi-account/

Tuy nhiên bạn có thể dụng env

export AWS_PROFILE=production

Hoặc là –>

So when your aws credentials file contains multiple profiles you need to export two environments variables before sops command.

AWS_SDK_LOAD_CONFIG=1 AWS_DEFAULT_PROFILE=sandbox sops -e -i settings/lab/connections.sh
#AWS_PROFILE : forces sops to use different profile from default
#AWS_SDK_LOAD_CONFIG=1 : AWS_PROFILE is only read if AWS_SDK_LOAD_CONFIG is enabled

giờ mình sẽ tạo file configmap.yaml

vi configmap.yaml

####Content
apiVersion: v1
data:
  exec.enable: 'true'
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
  namespace: argocd

root@LP11-D7891:~/sops# sops --encrypt --in-place configmap.yaml
INFO found a configuration for 'configmap.yaml' in '../.sops.yaml'
INFO: generating and storing data encryption key

root@LP11-D7891:~/sops# cat configmap.yaml
apiVersion: ENC[AES256_GCM,data:wsg=,iv:rp+e1hzMVMOoUXgu3qvnRj60xWPsBSD9bhSA7g7eL10=,tag:HdtqnI7dWsWV3W8kUB8Vcw==,type:str]
data:
    exec.enable: ENC[AES256_GCM,data:rM28xg==,iv:KpAEhMo0H4ML4CRBqu4sHBFobOtdqt0BOU9AlctM+50=,tag:Snti0P/oX41ZjOCJs8dQ5w==,type:str]
kind: ENC[AES256_GCM,data:VwA00IjAflnu,iv:ENpziSwMg5UkHkzCzFJzba0jcp0RQD2GzyFdfNVUFpo=,tag:+DfGjQob1YD7JatrXGHhag==,type:str]
metadata:
    labels:
        app.kubernetes.io/instance: ENC[AES256_GCM,data:CPO1WFAg,iv:q6js9e1eOJEEaHqvg0F0uYm4qYW3Sso4h2WTa2UmJ4A=,tag:jDCOnNxXrZ0Z9T/4DTL1Xw==,type:str]
        app.kubernetes.io/name: ENC[AES256_GCM,data:1uNgQJXxFw9R,iv:XHUNAZlcpCWj+c9GcV1NkgMDRJFYdxmEzyG8OM86eEg=,tag:xP6479VbKUahQZFjrrR1Jw==,type:str]
        app.kubernetes.io/part-of: ENC[AES256_GCM,data:mKEI3+eq,iv:JPcDPynaSL/vts3uu+KDOa7KFoXBSBvKTKbLLctbtMg=,tag:96tyadobNFj2ohvFxtn0wQ==,type:str]
    name: ENC[AES256_GCM,data:gxyAHqwXycSb,iv:BTM0lb34otc5b+5QbrCvzvMGw+0CpiEhSXww3KD+ovU=,tag:6utKNvvLy8nJFil5KoBikg==,type:str]
    namespace: ENC[AES256_GCM,data:VGbLLxuZ,iv:fYZsXcpB8RJZIEkbnE/WASNA22QsLfiy73wMH2B94Uk=,tag:vW8dDaQNDBN0sAVciQT0oA==,type:str]
sops:
    attention: This section contains key material that should only be modified with
        extra care. See `sops -h`.
    version: '1.18'
    unencrypted_suffix: _unencrypted
    kms:
    -   arn: arn:aws:kms:us-west-2:2508876XXXX:key/c2affea6-1a23-4730-811c-4e00f71b4e1d
        enc: AQICAHieQzzkJQgHz+zSKXuZTbF0N9fOD29+n/pVBcZo8lS4dAGCwuVHO/wUItW+8/YwjIlwAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMk6eWsUcYbrSk+TlUAgEQgDuTVf+uO5uY+mPb3Gkr61BHp9Nm6z+hMEJa6qX7WTDChwqNV01cXIuW9ByXNMLlO+j5olQrkGPOjqv3Ig==
        created_at: '2022-12-01T17:06:14Z'
    lastmodified: '2022-12-01T17:06:14Z'
    mac: ENC[AES256_GCM,data:Mv+AWbwYHYo+eV0GmhpomKhOI1OyjY4CdDVFZSwwuosKJwcecKvhR34rsnRrQhvLHAir6LlZhqPJfg2pO5kze8OSvStVkEBExM8CsO3n2TSIvqO7EoQdor6OVIdAF/z5Og3ZIcEWtcGxhZJJKUqhBwD3fIHAh9Knt4Zf9lfNz38=,iv:pv3l+88TfOhVaPf5S7Y1fQlKo8rgEuWRxsDr/MawZhc=,tag:Nc76oH9Zb4Spp5de21Am5g==,type:str]

Giờ mình đã encrypt được file

Giờ integrate with argocd:

Bạn có thêm tham khảo link này cài argocd thông qua helm
https://github.com/viaduct-ai/kustomize-sops/issues/156

Configure through Helm chart

repo argo:
https://argoproj.github.io/argo-helm

values.yaml

configs:
  cm:
    create: true
    kustomize.buildOptions: "--enable-alpha-plugins"

repoServer:
  name: repo-server
  env:
    - name: XDG_CONFIG_HOME
      value: /.config
    - name: AWS_DEFAULT_REGION
      value: eu-central-1
    - name: AWS_ACCESS_KEY_ID
      value: "XXXXXXXXXQSHIIZXXXXXXX"
    - name: AWS_SECRET_ACCESS_KEY
      value: "XXXXXXXXXXXDd9zLTqlMWSAEXXXXXXXX"
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
      name: custom-tools
      subPath: ksops
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.2.1
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
              mv ksops /custom-tools/;
              mv kustomize /custom-tools/;
              echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

Merge the extra configuration to the main manifest

Còn 1 style nữa bạn sẽ thường apply file manifest của argocd:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.6.4/manifests/install.yaml

Lúc này chúng ta sử dụng patchesStrategicMerge:

Tiếp đến là 2 file:

agocd-cm-plugin.yaml
>>>>
>>>>>>

apiVersion: v1
data:
  exec.enable: "true"
  kustomize.buildOptions: '--enable-alpha-plugins'
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]
kind: ConfigMap
metadata:
  name: argocd-cm
argocd-kms-ksops.yaml
>>>>>>>>>
>>>>>>>>>>>>>>


apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      initContainers:
        - args:
            - echo "Installing KSOPS...";
              mv ksops /custom-tools/;
              mv kustomize /custom-tools/;
              echo "Done.";
          command:
            - /bin/sh
            - '-c'
          image: 'viaductoss/ksops:v4.2.1'
          name: install-ksops
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      containers:
        - name: argocd-repo-server
          env:
            - name: XDG_CONFIG_HOME
              value: /.config
            - name: AWS_DEFAULT_REGION
              value: us-west-2
            - name: AWS_ACCESS_KEY_ID
              value: XXXXX3KUMTPJXXXXX
            - name: AWS_SECRET_ACCESS_KEY
              value: "XXXXXXbUmov+CVx6susUWzewXBEAXXXXXX"
          volumeMounts:
            - mountPath: /usr/local/bin/kustomize
              name: custom-tools
              subPath: kustomize
            - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
              name: custom-tools
              subPath: ksops
      volumes:
        - emptyDir: {}
          name: custom-tools

Encrypt Data in Your Application and Integrate with Argocd.

Giờ cần tạo push file mà bạn đã encrypt lên git và create 1 application trên argocd để kiểm chứng bạn có thể tham khảo link bên dưới
https://nimtechnology.com/2022/07/03/argocd-ksops-encrypting-resource-on-kustomise-and-argocd/#3_Encrypt_Data_in_Your_Application_and_Integrate_with_Argocd

Cannot merge or replace

Mình gặp lỗi này:

rpc error: code = Unknown desc = `kustomize build .meta-structure/coralogix/otel-agent --enable-alpha-plugins` failed exit status 1: Error: merging from generator &{0xc00015c730 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"otel-coralogix", Namespace:"coralogix"} does not exist; cannot merge or replace

Bạn sẽ cần remove hay xóa namespace trong file trước khi encrypt nhé

Real case.

Mình sẽ cài jenkins

Đầu tiên mình dùng kustomization integrate helm chart

Bạn để ý là mình có call sang ksops.
kustomization.yaml
>>>>>>>>>>>>>>>

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
- name: jenkins
  includeCRDs: false
  releaseName: jenkins-controller-android
  version: 4.1.8
  repo: https://charts.jenkins.io
  # valuesFile: values_helm.yaml
  namespace: default

generators:
 - ./kops.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: ksops-secret
files:
  - encrypted_file/jenkins-controller-android.yaml

Nếu bây giờ bạn sync luôn sẽ bị báo lỗi.

rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build .jenkins/jenkins-helm-ksops --enable-alpha-plugins` failed exit status 1: Error: merging from generator &{0xc00026e730 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"ConfigMap", isClusterScoped:false}, Name:"jenkins-controller-android", Namespace:"default"} exists; behavior must be merge or replace

https://github.com/viaduct-ai/kustomize-sops#encrypted-secret-overlays-w-generator-options

Lúc này bạn sẽ cần cân nhắc giữa replace hay merge

annotations:
    # replace the base secret data/stringData values with these encrypted data/stringData values
    kustomize.config.k8s.io/behavior: replace
annotations:
    # merge the base secret data/stringData values with these encrypted data/stringData values
    kustomize.config.k8s.io/behavior: merge
Mình chọn merge mà thay đổi phần minh muốn thay
ArgoCD

More Related Articles

[Argocd] Run “bash/sh” to execute pods on kubernetes via Argocd. ArgoCD
[Argocd – Azure] Login to ArgoCD using Microsoft or Azure Accounts. ArgoCD
[Argocd] Creating an Application of Argocd is related to helm public and repo helm ArgoCD
[AWS] Login Argocd via Cognito in AWS ArgoCD
[ArgoCD/KSOPS] Encrypting Resource on kustomize and Argocd. ArgoCD
[kubectl/Argocd] How to create a kubectl config file for serviceaccount or from the cluster secret of Argocd ArgoCD

Leave a Reply

Your email address will not be published. Required fields are marked *