GCP: How to add TLS certificate to GKE from Google Secret Manager

The only hurdle every beginner in GKE comes across is how to add TLS certificate to GKE from Google Secret Manager. If you are managing multiple sub domains under one organization then having one TLS certificate stored in Google Secret Manager is a better way to distribute the TLS certificate across all new clusters in GKE in a single GCP project. There are three programmatic solutions that work with helm for handling secrets from Google Secret Manager. The native solution among them is Secret Store CSI Driver. Every public cloud provider has support for Secret Store CSI Driver. For GCP, we have secrets-store-csi-driver-provider-gcp but it lacks detailed documentation with examples on different ways we can use secrets-store-csi-driver-provider-gcp to access secrets in Google Secret Manager. I read through several articles but none of them are showing how to add TLS certificate to GKE from Google Secret Manager.

Although i found the article Shhhh… Kubernetes Secrets Are Not Really Secret! written by Deepu K Sasidharan to be helpful in understanding the concepts about Kubernetes Secrets. Below is the image from the article,

Image credit from https://auth0.com/blog/kubernetes-secrets-management/

I had struggled with adding TLS certificate to GKE from Google Secret Manager. I was able to find some clues on TLS certificate from the issue number 96 in secrets-store-csi-driver-provider-gcp project in GitHub.

Please follow the installation steps from the link. Make sure to enable sync as Kubernetes secret and secret auto rotation. If not enabled it will cause issues when there are changes to the secrets in Google Secret Manager.

After installation, perform the IAM binding from the usage steps from the link.

After usage steps, you need to create a service account yaml file for storing the service account annotation. Below is the example `mypodserviceaccount.yaml file,

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mypodserviceaccount
  namespace: default
  annotations:
    iam.gke.io/gcp-service-account: gke-workload@$PROJECT_ID.iam.gserviceaccount.com

In your pod yaml file add a volume mount path to store secrets and in volumes add the volume name for the secrets. Below is the example `mypod.yaml` file,

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: default
spec:
  serviceAccountName: mypodserviceaccount
  containers:
  - image: gcr.io/google.com/cloudsdktool/cloud-sdk:slim
    imagePullPolicy: IfNotPresent
    name: mypod
    resources:
      requests:
        cpu: 100m
    stdin: true
    stdinOnce: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
      - mountPath: "/var/secrets"
        name: mysecret
  volumes:
  - name: mysecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-tls"

From the `mypod.yaml` file, the volume mount where the secret file is stored in the path `var/secrets/`is,

volumeMounts:
      - mountPath: "/var/secrets"
        name: mysecret

From the `mypod.yaml` file, the volume name where the secret file is referenced from the secret provider class `my-tls` is,

volumes:
  - name: mysecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "my-tls"

The secret provider class yaml file for storing the secrets. Below is the example `my-tls-secret-provider.yaml` file,

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: my-tls
spec:
  provider: gcp
  secretObjects:
  - secretName: my-tls-csi
    type: kubernetes.io/tls
    data: 
    - objectName: "mycert.pem"
      key: tls.key
    - objectName: "mycert.pem"
      key: tls.crt
  parameters:
    secrets: |
      - resourceName: "projects/$PROJECT_ID/secrets/testsecret/versions/latest"
         fileName: "mycert.pem"

You need to store the testsecret file in the Google Secret Manager in the following format,

-----BEGIN KEY----- 
MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
-----END KEY-----
-----BEGIN CERTIFICATE----- 
MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
-----END CERTIFICATE----

Then, only the key-value pairs of tls.key and tls.cert are extracted and created in the Kubernetes Secrets as `my-tls-csi` secret. You need to use this secret in your load balancer in GKE cluster.

Reference the Setting up HTTPS (TLS) between client and load balancer on how to setup the tls secret in a load balancer. Below is the example `my-ingress-controller.yaml` file,

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  tls:
  - secretName: my-tls-csi
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: SERVICE_NAME
            port:
              number: 60000

Recommendation: Always try out first if the secret is working via first manual creation of the secret in Kubernetes secrets then only try the secrets-store-csi-driver-provider-gcp programmatic way.

Credits,

1. The examples are from the secrets-store-csi-driver-provider-gcp project in GitHub.

If you like my articles, please follow me on Medium and you can also support me by buying me a coffee.