Kubernetes Lesson 20 – Secrets | Dataplexa
Core Kubernetes Concepts · Lesson 20

Secrets

Kubernetes Secrets are designed to hold sensitive data — passwords, tokens, TLS certificates, API keys. They work almost identically to ConfigMaps, but with one critical difference that most people get wrong: they're not actually secret by default, and understanding why is the most important thing in this lesson.

The Uncomfortable Truth About Kubernetes Secrets

When you create a Kubernetes Secret, the value is stored in etcd encoded in base64. Base64 is not encryption. It's encoding. Anyone who can run kubectl get secret my-secret -o yaml in the namespace can decode it in two seconds with echo "dXNlcm5hbWU=" | base64 --decode. Secrets are base64 so they can carry binary data and special characters cleanly — not to protect them.

The real security comes from three separate mechanisms layered on top: RBAC (who can read Secrets in this namespace), etcd encryption at rest (encrypt the etcd database itself), and external secret management (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). A cluster with no RBAC and no etcd encryption has Secrets that are trivially readable by anyone with cluster access.

⚠️ Never commit Secret manifests to Git

A Secret YAML file contains base64-encoded values. Anyone who sees that file can decode them. Never commit Secret manifests to version control — not even a private repo. Use a tool like Sealed Secrets, SOPS, or an external secrets operator to encrypt values before committing, or generate Secrets outside Git entirely using a secrets manager.

Secret Types

Kubernetes has several built-in Secret types. Most of the time you'll use Opaque, but the typed ones unlock special integrations — like automatic image pull authentication.

Type Used for Required keys
Opaque General-purpose — any key-value pairs you define. Default type. None — fully user-defined
kubernetes.io/tls TLS certificate and private key pairs — used by Ingress controllers tls.crt, tls.key
kubernetes.io/dockerconfigjson Docker registry credentials — used to pull private images .dockerconfigjson
kubernetes.io/service-account-token Auto-created token for ServiceAccount authentication to the API server token, ca.crt, namespace

Creating Secrets the Right Way

The scenario: You're a backend engineer deploying a new payment processing service. It needs to connect to a PostgreSQL database, call an external payment gateway using an API key, and authenticate with your private container registry to pull the image. Three different secrets — all need to be in the cluster before the Pod starts. Here's how to create all three without ever putting a plaintext value in Git.

kubectl create secret generic postgres-credentials \
  --from-literal=POSTGRES_USER=payment_user \
  --from-literal=POSTGRES_PASSWORD=s3cur3P@ssw0rd \
  --from-literal=POSTGRES_DB=payments \
  -n payments
# generic = Opaque type — the default for arbitrary key-value secrets
# --from-literal: values are provided inline on the command line
# The values are base64-encoded automatically by kubectl before storing in etcd
# This command is safe because the value never touches a file — but it does appear in shell history

kubectl create secret generic postgres-credentials \
  --from-env-file=.env \
  -n payments
# --from-env-file: reads a KEY=value file (like a .env file) and creates one secret entry per line
# Better than --from-literal for multiple values — .env file is gitignored, not committed

kubectl create secret generic payment-gateway-key \
  --from-literal=API_KEY=pk_live_51NxKjQ2eZvKYlo \
  -n payments

kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.company.com \
  --docker-username=ci-bot \
  --docker-password=ghp_xxxxxxxxxxxx \
  --docker-email=ci@company.com \
  -n payments
# docker-registry: creates a kubernetes.io/dockerconfigjson type secret
# Used in imagePullSecrets to authenticate with private container registries
$ kubectl create secret generic postgres-credentials \
  --from-literal=POSTGRES_USER=payment_user \
  --from-literal=POSTGRES_PASSWORD=s3cur3P@ssw0rd \
  --from-literal=POSTGRES_DB=payments \
  -n payments
secret/postgres-credentials created

$ kubectl get secret postgres-credentials -n payments -o yaml
apiVersion: v1
data:
  POSTGRES_DB: cGF5bWVudHM=
  POSTGRES_PASSWORD: czNjdXIzUEBzc3cwcmQ=
  POSTGRES_USER: cGF5bWVudF91c2Vy
kind: Secret
metadata:
  name: postgres-credentials
  namespace: payments
type: Opaque

$ echo "cGF5bWVudF91c2Vy" | base64 --decode
payment_user

What just happened?

base64 is not encryptioncGF5bWVudF91c2Vy looks opaque, but one pipe to base64 --decode and it's plaintext. This is the demo everyone in security gives when they want to explain why "Kubernetes Secrets aren't secret." The base64 encoding exists so that binary data and special characters (like the @ in the password) can be stored as clean ASCII in etcd — not to obfuscate the value.

Shell history warning — When you use --from-literal with a password, the full command including the password is written to your shell history (~/.bash_history or ~/.zsh_history). In production, prefer --from-env-file with a gitignored file, or pipe the value: kubectl create secret generic my-secret --from-literal=KEY=$(vault read -field=value secret/my-key).

type: Opaque — When you use kubectl create secret generic, the type is set to Opaque automatically. This is the catch-all type for arbitrary secrets. Kubernetes doesn't validate the keys or structure — you define both.

Writing a Secret Manifest — With Pre-encoded Values

If you need a Secret in YAML form (for applying via a pipeline or storing in an encrypted secrets tool), the values in the data field must be base64-encoded. Kubernetes also provides a stringData field that accepts plaintext and encodes it for you at apply time.

The scenario: Your CI/CD pipeline needs to apply a Secret manifest as part of its deployment step. The pipeline reads values from a secrets manager (Vault or AWS SSM) and renders a YAML file at deploy time. You need to understand both data and stringData to write that template correctly.

echo -n "payment_user" | base64
# -n: suppress the trailing newline — CRITICAL — without -n, the newline gets encoded too
# This produces: cGF5bWVudF91c2Vy
# The trailing newline bug is one of the most common Secret encoding mistakes

echo -n "s3cur3P@ssw0rd" | base64
# Produces: czNjdXIzUEBzc3cwcmQ=

echo -n "payments" | base64
# Produces: cGF5bWVudHM=
apiVersion: v1
kind: Secret
metadata:
  name: postgres-credentials     # Secret name — referenced by Pods that consume it
  namespace: payments
  labels:
    app: payment-processor
type: Opaque                      # Opaque = arbitrary key-value secret, no schema enforced
data:                             # data: values MUST be base64-encoded
  POSTGRES_USER: cGF5bWVudF91c2Vy          # base64("payment_user")
  POSTGRES_PASSWORD: czNjdXIzUEBzc3cwcmQ=  # base64("s3cur3P@ssw0rd")
  POSTGRES_DB: cGF5bWVudHM=                # base64("payments")

---

apiVersion: v1
kind: Secret
metadata:
  name: payment-gateway-credentials
  namespace: payments
type: Opaque
stringData:                       # stringData: accepts PLAINTEXT — Kubernetes encodes it at apply time
  API_KEY: "pk_live_51NxKjQ2eZvKYlo"       # Write the raw value here — no base64 needed
  API_SECRET: "sk_live_AbCdEfGhIjKlMn"     # Kubernetes converts to base64 before storing in etcd
                                            # stringData is WRITE-ONLY — it won't appear in kubectl get output
                                            # The encoded value appears in data after apply
$ kubectl apply -f postgres-secret.yaml
secret/postgres-credentials created
secret/payment-gateway-credentials created

$ kubectl get secret payment-gateway-credentials -n payments -o yaml
apiVersion: v1
data:
  API_KEY: cGtfbGl2ZV81MU54S2pRMmVadktZbG8=
  API_SECRET: c2tfbGl2ZV9BYkNkRWZHaElqS2xNbg==
kind: Secret
metadata:
  name: payment-gateway-credentials
  namespace: payments
type: Opaque

What just happened?

data vs stringDatadata requires you to pre-encode values as base64. stringData takes plaintext and Kubernetes encodes it during apply. After the apply, the values only appear in the data field — stringData is consumed and disappears. This makes stringData convenient for templated YAML from CI pipelines — the pipeline substitutes the plaintext value, Kubernetes handles encoding.

The -n flag on echo — Without -n, echo appends a newline character before encoding. So echo "password" | base64 actually encodes "password\n" — and when the container decodes it, the value has a trailing newline. This has broken database connection strings and authentication headers in real production environments. Always use echo -n.

Consuming Secrets in Pods

Secrets are consumed exactly like ConfigMaps — either as environment variables or as volume-mounted files. The YAML syntax is nearly identical, just swapping configMapKeyRef for secretKeyRef. The delivery mechanism is the same; what's different is the intent, the access controls, and where Kubernetes stores them in memory.

The scenario: The payment processor service needs its PostgreSQL credentials as environment variables and its payment gateway API key. It also needs to pull its Docker image from a private registry. You're wiring all three secrets into the Deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-processor
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-processor
  template:
    metadata:
      labels:
        app: payment-processor
    spec:
      imagePullSecrets:                  # imagePullSecrets: authenticate to private container registry
        - name: registry-credentials     # Reference the docker-registry Secret we created earlier
                                         # Without this, pods fail with ErrImagePull on private images

      containers:
        - name: payment-processor
          image: registry.company.com/payment-processor:3.1.0
          ports:
            - containerPort: 8080
          env:
            - name: POSTGRES_USER        # Env var name in the container
              valueFrom:
                secretKeyRef:            # secretKeyRef: pull a single key from a Secret
                  name: postgres-credentials  # Which Secret to read from
                  key: POSTGRES_USER     # Which key within the Secret
                  optional: false        # optional: false means Pod fails if Secret doesn't exist
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: POSTGRES_PASSWORD
                  optional: false
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: postgres-credentials
                  key: POSTGRES_DB
          envFrom:
            - secretRef:                 # secretRef: inject ALL keys from a Secret as env vars
                name: payment-gateway-credentials  # All keys become env vars: API_KEY, API_SECRET
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "300m"
              memory: "256Mi"
$ kubectl apply -f payment-processor-deployment.yaml
deployment.apps/payment-processor created

$ kubectl get pods -n payments
NAME                                 READY   STATUS    RESTARTS   AGE
payment-processor-8c4f7d-j9pkx       1/1     Running   0          14s
payment-processor-8c4f7d-r2skl       1/1     Running   0          14s

$ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- env | grep POSTGRES
POSTGRES_DB=payments
POSTGRES_PASSWORD=s3cur3P@ssw0rd
POSTGRES_USER=payment_user

What just happened?

Secrets are decoded automatically — Inside the container, the env var value is the decoded plaintext — not the base64 string. Kubernetes decodes the Secret values before injecting them. The container sees POSTGRES_PASSWORD=s3cur3P@ssw0rd, not the base64 gibberish. Same applies to volume-mounted Secrets — the files contain decoded plaintext.

imagePullSecrets — This field lives at the Pod spec level (not container level) and tells the kubelet which credentials to use when pulling the container image. Without it, a Pod trying to pull from a private registry will fail immediately with ErrImagePull or ImagePullBackOff. You can also attach imagePullSecrets to a ServiceAccount so all Pods using that account inherit the credentials automatically.

optional: false — Setting optional: false on a secretKeyRef means the Pod will not start if the referenced Secret doesn't exist or doesn't have that key. This is fail-fast behaviour — better to surface a misconfiguration immediately than to have a Pod start with a missing password and fail only when it tries its first database query. Set optional: true only if the secret being absent is an acceptable runtime condition.

Mounting Secrets as Files

Some applications expect credentials as files — TLS certificates, SSH keys, service account JSON files. The volume mount approach for Secrets works exactly like ConfigMaps but with one important addition: Kubernetes stores Secret volumes in tmpfs (in-memory filesystem) rather than on disk. This means the Secret data is never written to the node's disk storage, which reduces the risk of it appearing in disk forensics.

The scenario: The payment processor needs to establish a TLS-encrypted connection to the PostgreSQL database. The database requires mutual TLS — the client must present a certificate. You're mounting the TLS Secret as files so the Java JDBC driver can read the cert and key from the filesystem.

kubectl create secret tls postgres-tls \
  --cert=client.crt \
  --key=client.key \
  -n payments
# tls subcommand: creates a kubernetes.io/tls type Secret
# --cert: path to the PEM-encoded certificate file
# --key: path to the PEM-encoded private key file
# Kubernetes stores them under the keys tls.crt and tls.key
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-processor
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payment-processor
  template:
    metadata:
      labels:
        app: payment-processor
    spec:
      containers:
        - name: payment-processor
          image: registry.company.com/payment-processor:3.1.0
          volumeMounts:
            - name: tls-certs             # Must match volumes[].name below
              mountPath: /etc/ssl/certs   # Path where the cert files will appear in the container
              readOnly: true              # readOnly: always use true for secrets mounted as files
      volumes:
        - name: tls-certs
          secret:                         # secret: mount a Secret as a volume (vs configMap:)
            secretName: postgres-tls      # Which Secret to expose as this volume
            defaultMode: 0400             # defaultMode: file permissions in octal
                                          # 0400 = owner read-only — prevents accidental writes
                                          # Private keys should always be 0400 or 0600
$ kubectl apply -f payment-processor-tls.yaml
deployment.apps/payment-processor configured

$ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- ls -la /etc/ssl/certs
total 0
drwxrwxrwt 3 root root  120 Mar 10 09:44 .
drwxr-xr-x 1 root root 4096 Mar 10 09:44 ..
lrwxrwxrwx 1 root root   14 Mar 10 09:44 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root   14 Mar 10 09:44 tls.key -> ..data/tls.key

$ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- df -h /etc/ssl/certs
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.8G     0  7.8G   0% /etc/ssl/certs

What just happened?

tmpfs confirmed — The df -h output shows the mount is on tmpfs — a RAM-backed filesystem. The cert and key files exist in memory, not on the node's disk. If the Pod is killed, the files vanish. This is one of the concrete security benefits of volume-mounted Secrets over env vars — env var values can appear in /proc/[pid]/environ on the node, which is a disk-backed file.

Symlinks to ..data/ — The files are symlinks pointing to a ..data/ directory. This is how Kubernetes implements live Secret updates for volume mounts — it atomically swaps the ..data/ symlink target to a new directory containing the updated content. The files your app reads always point to the latest version.

defaultMode: 0400 — File permissions matter for private keys. A key file with permissions 0644 (world-readable) will be rejected by most TLS libraries with an error like "permissions are too open." The defaultMode: 0400 field sets the file to owner-read-only, which matches what SSH and TLS tools expect.

The Secrets Security Landscape

Here's how the security layers stack — from the weakest default to the strongest hardened setup:

Kubernetes Secrets Security Layers

Level 0
No protection (default fresh cluster)
Secrets stored as base64 in etcd. No etcd encryption. Anyone with kubectl access can read all Secrets. etcd backup files contain plaintext secrets.
Level 1
+ RBAC on Secrets
Restrict get/list on Secrets to specific service accounts and users. Developers can deploy but can't read the Secret values. Only the application SA and admins have access.
Level 2
+ etcd Encryption at Rest
Enable EncryptionConfiguration on the API server. Secrets are AES-encrypted before writing to etcd. Even direct etcd access or backup file leakage doesn't expose plaintext values.
Level 3
+ External Secrets Manager
Use External Secrets Operator, Vault Agent Injector, or AWS Secrets Manager CSI driver. Secret values live in Vault/AWS/GCP — Kubernetes Secrets are either never created or auto-synced and rotated. Zero plaintext ever stored in etcd.

Teacher's Note: What level should you be at?

For a production cluster that handles any kind of sensitive data, Level 1 (RBAC) is the absolute minimum — and it costs nothing to implement. Level 2 (etcd encryption) is standard practice on managed Kubernetes services like EKS, GKE, and AKS — they often enable it by default. Level 3 is where security-conscious companies operating at scale land, especially those in regulated industries (finance, healthcare).

A quick trick to see all Secret values in a namespace in one command — and understand why RBAC matters: kubectl get secrets -n payments -o json | jq -r '.items[] | .metadata.name + ": " + (.data | to_entries[] | .key + "=" + (.value | @base64d))'. If you can run that command in your production namespace, so can an attacker who compromises any Pod with a permissive service account.

The lesson: Kubernetes Secrets are a delivery mechanism. The security model around them is your responsibility — not Kubernetes's.

Practice Questions

1. Kubernetes Secret values are stored in etcd using what encoding? And why does this NOT mean they are encrypted?



2. Which Secret manifest field accepts plaintext values and has Kubernetes automatically base64-encode them at apply time — so you don't have to pre-encode values yourself?



3. When a Secret is mounted as a volume into a container, Kubernetes uses what type of in-memory filesystem so the Secret data is never written to the node's disk?



Quiz

1. You need to base64-encode the string mypassword to put it in a Secret's data field. Which command produces the correct result without a trailing newline bug?


2. Which Pod spec field references a kubernetes.io/dockerconfigjson Secret to allow the kubelet to pull images from a private container registry?


3. A container spec uses secretKeyRef with optional: false to reference a key in a Secret that does not exist. What happens?


Up Next · Lesson 21

Environment Variables

All the ways to inject config into containers — static values, ConfigMap refs, Secret refs, Pod metadata, and field references — in one complete lesson.