Manage Kubernetes Secrets With External Secrets Operator

Hiện tại mình sẽ cần sync secret từ data secret nào đó lên thành secret trên k8s
chúng ta sẽ sử dụng external secret

Nếu bạn muốn config external secret với Google thì coi clip này!

Mình sẽ focus vào external secret với AWS

Contents

1) Install External Secrets via Helm chart

helm repo add external-secrets \
    https://charts.external-secrets.io

helm repo update

helm upgrade --install \
    external-secrets \
    external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace

2) Create User and Assume Role access Parameter Stores

# Create IAM User
aws iam create-user --user-name readonly_ssm

# Set password for readonly_ssm user
aws iam create-login-profile --user-name readonly_ssm --password @SSMUserReadOnly --no-password-reset-required

#Credentials for IAM User and make a note of them
aws iam create-access-key --user-name readonly_ssm

# Make a note of Access Key ID and Secret Access Key
User: readonly_ssm
{
    "AccessKey": {
        "UserName": "readonly_ssm",
        "AccessKeyId": "AKIATU2QSHIITN5ALD7E",
        "Status": "Active",
        "SecretAccessKey": "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",
        "CreateDate": "2022-08-31T06:47:24+00:00"
    }
}

còn phần tạo assume role access vào Parameter thì tham khảo bài sau:

[Demo] Instructing configure AssumeRole – IAM on AWS

Giờ bạn tạo ra 1 secret và lưu secret access và access key

apiVersion: v1
data:
  secret-access-key: UEd3QVBkMlZPWjhkSUZIejRBciXXXXXXlaTGtsQkJUWE1lKw==
  access-key: QUtJQVRVMlFTXXXXX41QUxEN0U=
kind: Secret
metadata:
  name: awssm-secret
  namespace: external-secrets
type: Opaque

3) External Secret integrates with AWS

3.1) External Secret integrates with Parameter Stores

Bạn có thêm thao khảo docs chính
https://external-secrets.io/v0.5.9/provider-aws-parameter-store/

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: ParameterStore
      # define a specific role to limit access
      # to certain secrets
      role: arn:aws:iam::250887682577:policy/AmazonParameterStoreReadOnlyPolicy
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

Để tạo được secret trong parameter stores bạn tham khảo bài này

[AWS] Saving your secret on AWS Systems Manager – Parameter Store

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  data:
  - secretKey: firstname
    remoteRef:
      key: /dev/mysql/hostname

Sau khi mình apply hì secret đã được tự động sync

Lưu ý cực kỳ to:
Vị dụ bạn muốn sync Secret từ Parameter Store —> sang –> Namespace là: nimtechnology
thì các secret chứa access và secret key AWS, ExternalSecret, SecretStore phải nằm trên namespace nimtechnology

3.2) External Secret integrates with Secrets Manager

Getting Started with External Secrets Operator on Kubernetes using AWS Secrets Manager

Bạn có thể tham khảo bài này

https://waswani.medium.com/integrating-secrets-manager-with-aws-eks-79c93e70c74e
https://www.bigtreetc.com/column/eks-secrets/
https://external-secrets.io/v0.5.7/api-secretstore/

https://blog.container-solutions.com/tutorial-how-to-set-external-secrets-with-aws

3.2.1) Create secret on AWS Secrets Manager.

đầu tiên bạn phải tạo 1 secret để test

3.2.2) Using Access Key

Mình vị dự bạn muốn sync 1 secret từ Secrets Manager xuống –> namespace a
thì các config dưới đây bạn cần create trên namepace a đó

Bạn sẽ cần tạo 1 secret để lưu access key và secret key trong đó.

apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: <namepace a>
type: Opaque
data:
  access-key: QUtJQTJZMlRCNYRTQ=
  secret: czVseUJtNHhjRlJNNXpERmZtcHBZSVsTnlaMW9mZ2lpUQ==

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: ss-<namepace a>
  namespace: <namepace a>
spec:
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access-key
            name: aws-secret
          secretAccessKeySecretRef:
            key: secret
            name: aws-secret
      region: us-west-2
      service: SecretsManager

3.2.2.1) If access key and Secrets Manager stand on the same Acount ID.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-<namepace a>
  namespace: <namepace a>
spec:
  dataFrom:
    - extract:
        conversionStrategy: Default
        decodingStrategy: None
        key: MDC/STAGING/PLATFORM
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: ss-<namepace a>
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: <namepace a>

3.2.2.2) If the access key and Secrets Manager stand on the different account IDs.

ở trường hợp này bạn cần cấp permission trên Secrets Manager allow cho account id của bạn.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: es-<namepace a>
  namespace: <namepace a>
spec:
  dataFrom:
    - extract:
        conversionStrategy: Default
        decodingStrategy: None
        key: arn:aws:secretsmanager:us-west-2:XXXX2008XXXX:secret:Nim/STAGING/PLATFORM-v4kaek
  refreshInterval: 1m
  secretStoreRef:
    kind: SecretStore
    name: ss-<namepace a>
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: <namepace a>

Error – ExternalSecret isn’t not OK

could not set ExternalSecret controller reference: Object staging-xxx/staging-xxx is already owned by another ExternalSecret controller staging-xxx”,”stacktrace”:”sigs.k8s.io/controller-runtime/pkg/internal/controller.

{"level":"error","ts":1676537761.9858818,"logger":"controllers.ExternalSecret","msg":"could not update Secret","ExternalSecret":"staging-xxx/es-staging-xxx","error":"could not set ExternalSecret controller reference: Object staging-xxx/staging-xxx is already owned by another ExternalSecret controller staging-xxx","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:278\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}
6
{"level":"error","ts":1676537761.994499,"msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"es-staging-xxx","namespace":"staging-xxx"},"namespace":"staging-xxx","name":"es-staging-xxx","reconcileID":"442003a1-a7cb-477d-b69b-ac252b311c9e","error":"could not set ExternalSecret controller reference: Object staging-xxx/staging-xxx is already owned by another ExternalSecret controller staging-xxx","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"}

Bạn cần phải xóa toàn bộ các component liên quan đến Old external secret khi migrate sang external secret operator

KIND                           NAMESPACE               NAME                                 API_VERSION                            REPLACE_WITH (SINCE)
ClusterRole                    <undefined>             kubernetes-external-secrets          rbac.authorization.k8s.io/v1beta1      rbac.authorization.k8s.io/v1 (1.8.0)
ClusterRoleBinding             <undefined>             kubernetes-external-secrets          rbac.authorization.k8s.io/v1beta1      rbac.authorization.k8s.io/v1 (1.8.0)
ClusterRoleBinding             <undefined>             kubernetes-external-secrets-auth     rbac.authorization.k8s.io/v1beta1      rbac.authorization.k8s.io/v1 (1.8.0)
CustomResourceDefinition       <undefined>             externalmetrics.metrics.aws          apiextensions.k8s.io/v1beta1           apiextensions.k8s.io/v1 (1.16.0)