Kubernetes Course
Secrets
Kubernetes Secrets are designed to hold sensitive data — passwords, tokens, TLS certificates, API keys. They work almost identically to ConfigMaps, but with one critical difference that most people get wrong: they're not actually secret by default, and understanding why is the most important thing in this lesson.
The Uncomfortable Truth About Kubernetes Secrets
When you create a Kubernetes Secret, the value is stored in etcd encoded in base64. Base64 is not encryption. It's encoding. Anyone who can run kubectl get secret my-secret -o yaml in the namespace can decode it in two seconds with echo "dXNlcm5hbWU=" | base64 --decode. Secrets are base64 so they can carry binary data and special characters cleanly — not to protect them.
The real security comes from three separate mechanisms layered on top: RBAC (who can read Secrets in this namespace), etcd encryption at rest (encrypt the etcd database itself), and external secret management (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). A cluster with no RBAC and no etcd encryption has Secrets that are trivially readable by anyone with cluster access.
⚠️ Never commit Secret manifests to Git
A Secret YAML file contains base64-encoded values. Anyone who sees that file can decode them. Never commit Secret manifests to version control — not even a private repo. Use a tool like Sealed Secrets, SOPS, or an external secrets operator to encrypt values before committing, or generate Secrets outside Git entirely using a secrets manager.
Secret Types
Kubernetes has several built-in Secret types. Most of the time you'll use Opaque, but the typed ones unlock special integrations — like automatic image pull authentication.
| Type | Used for | Required keys |
|---|---|---|
| Opaque | General-purpose — any key-value pairs you define. Default type. | None — fully user-defined |
| kubernetes.io/tls | TLS certificate and private key pairs — used by Ingress controllers | tls.crt, tls.key |
| kubernetes.io/dockerconfigjson | Docker registry credentials — used to pull private images | .dockerconfigjson |
| kubernetes.io/service-account-token | Auto-created token for ServiceAccount authentication to the API server | token, ca.crt, namespace |
Creating Secrets the Right Way
The scenario: You're a backend engineer deploying a new payment processing service. It needs to connect to a PostgreSQL database, call an external payment gateway using an API key, and authenticate with your private container registry to pull the image. Three different secrets — all need to be in the cluster before the Pod starts. Here's how to create all three without ever putting a plaintext value in Git.
kubectl create secret generic postgres-credentials \
--from-literal=POSTGRES_USER=payment_user \
--from-literal=POSTGRES_PASSWORD=s3cur3P@ssw0rd \
--from-literal=POSTGRES_DB=payments \
-n payments
# generic = Opaque type — the default for arbitrary key-value secrets
# --from-literal: values are provided inline on the command line
# The values are base64-encoded automatically by kubectl before storing in etcd
# This command is safe because the value never touches a file — but it does appear in shell history
kubectl create secret generic postgres-credentials \
--from-env-file=.env \
-n payments
# --from-env-file: reads a KEY=value file (like a .env file) and creates one secret entry per line
# Better than --from-literal for multiple values — .env file is gitignored, not committed
kubectl create secret generic payment-gateway-key \
--from-literal=API_KEY=pk_live_51NxKjQ2eZvKYlo \
-n payments
kubectl create secret docker-registry registry-credentials \
--docker-server=registry.company.com \
--docker-username=ci-bot \
--docker-password=ghp_xxxxxxxxxxxx \
--docker-email=ci@company.com \
-n payments
# docker-registry: creates a kubernetes.io/dockerconfigjson type secret
# Used in imagePullSecrets to authenticate with private container registries
$ kubectl create secret generic postgres-credentials \ --from-literal=POSTGRES_USER=payment_user \ --from-literal=POSTGRES_PASSWORD=s3cur3P@ssw0rd \ --from-literal=POSTGRES_DB=payments \ -n payments secret/postgres-credentials created $ kubectl get secret postgres-credentials -n payments -o yaml apiVersion: v1 data: POSTGRES_DB: cGF5bWVudHM= POSTGRES_PASSWORD: czNjdXIzUEBzc3cwcmQ= POSTGRES_USER: cGF5bWVudF91c2Vy kind: Secret metadata: name: postgres-credentials namespace: payments type: Opaque $ echo "cGF5bWVudF91c2Vy" | base64 --decode payment_user
What just happened?
base64 is not encryption — cGF5bWVudF91c2Vy looks opaque, but one pipe to base64 --decode and it's plaintext. This is the demo everyone in security gives when they want to explain why "Kubernetes Secrets aren't secret." The base64 encoding exists so that binary data and special characters (like the @ in the password) can be stored as clean ASCII in etcd — not to obfuscate the value.
Shell history warning — When you use --from-literal with a password, the full command including the password is written to your shell history (~/.bash_history or ~/.zsh_history). In production, prefer --from-env-file with a gitignored file, or pipe the value: kubectl create secret generic my-secret --from-literal=KEY=$(vault read -field=value secret/my-key).
type: Opaque — When you use kubectl create secret generic, the type is set to Opaque automatically. This is the catch-all type for arbitrary secrets. Kubernetes doesn't validate the keys or structure — you define both.
Writing a Secret Manifest — With Pre-encoded Values
If you need a Secret in YAML form (for applying via a pipeline or storing in an encrypted secrets tool), the values in the data field must be base64-encoded. Kubernetes also provides a stringData field that accepts plaintext and encodes it for you at apply time.
The scenario: Your CI/CD pipeline needs to apply a Secret manifest as part of its deployment step. The pipeline reads values from a secrets manager (Vault or AWS SSM) and renders a YAML file at deploy time. You need to understand both data and stringData to write that template correctly.
echo -n "payment_user" | base64
# -n: suppress the trailing newline — CRITICAL — without -n, the newline gets encoded too
# This produces: cGF5bWVudF91c2Vy
# The trailing newline bug is one of the most common Secret encoding mistakes
echo -n "s3cur3P@ssw0rd" | base64
# Produces: czNjdXIzUEBzc3cwcmQ=
echo -n "payments" | base64
# Produces: cGF5bWVudHM=
apiVersion: v1
kind: Secret
metadata:
name: postgres-credentials # Secret name — referenced by Pods that consume it
namespace: payments
labels:
app: payment-processor
type: Opaque # Opaque = arbitrary key-value secret, no schema enforced
data: # data: values MUST be base64-encoded
POSTGRES_USER: cGF5bWVudF91c2Vy # base64("payment_user")
POSTGRES_PASSWORD: czNjdXIzUEBzc3cwcmQ= # base64("s3cur3P@ssw0rd")
POSTGRES_DB: cGF5bWVudHM= # base64("payments")
---
apiVersion: v1
kind: Secret
metadata:
name: payment-gateway-credentials
namespace: payments
type: Opaque
stringData: # stringData: accepts PLAINTEXT — Kubernetes encodes it at apply time
API_KEY: "pk_live_51NxKjQ2eZvKYlo" # Write the raw value here — no base64 needed
API_SECRET: "sk_live_AbCdEfGhIjKlMn" # Kubernetes converts to base64 before storing in etcd
# stringData is WRITE-ONLY — it won't appear in kubectl get output
# The encoded value appears in data after apply
$ kubectl apply -f postgres-secret.yaml secret/postgres-credentials created secret/payment-gateway-credentials created $ kubectl get secret payment-gateway-credentials -n payments -o yaml apiVersion: v1 data: API_KEY: cGtfbGl2ZV81MU54S2pRMmVadktZbG8= API_SECRET: c2tfbGl2ZV9BYkNkRWZHaElqS2xNbg== kind: Secret metadata: name: payment-gateway-credentials namespace: payments type: Opaque
What just happened?
data vs stringData — data requires you to pre-encode values as base64. stringData takes plaintext and Kubernetes encodes it during apply. After the apply, the values only appear in the data field — stringData is consumed and disappears. This makes stringData convenient for templated YAML from CI pipelines — the pipeline substitutes the plaintext value, Kubernetes handles encoding.
The -n flag on echo — Without -n, echo appends a newline character before encoding. So echo "password" | base64 actually encodes "password\n" — and when the container decodes it, the value has a trailing newline. This has broken database connection strings and authentication headers in real production environments. Always use echo -n.
Consuming Secrets in Pods
Secrets are consumed exactly like ConfigMaps — either as environment variables or as volume-mounted files. The YAML syntax is nearly identical, just swapping configMapKeyRef for secretKeyRef. The delivery mechanism is the same; what's different is the intent, the access controls, and where Kubernetes stores them in memory.
The scenario: The payment processor service needs its PostgreSQL credentials as environment variables and its payment gateway API key. It also needs to pull its Docker image from a private registry. You're wiring all three secrets into the Deployment manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processor
namespace: payments
spec:
replicas: 2
selector:
matchLabels:
app: payment-processor
template:
metadata:
labels:
app: payment-processor
spec:
imagePullSecrets: # imagePullSecrets: authenticate to private container registry
- name: registry-credentials # Reference the docker-registry Secret we created earlier
# Without this, pods fail with ErrImagePull on private images
containers:
- name: payment-processor
image: registry.company.com/payment-processor:3.1.0
ports:
- containerPort: 8080
env:
- name: POSTGRES_USER # Env var name in the container
valueFrom:
secretKeyRef: # secretKeyRef: pull a single key from a Secret
name: postgres-credentials # Which Secret to read from
key: POSTGRES_USER # Which key within the Secret
optional: false # optional: false means Pod fails if Secret doesn't exist
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_PASSWORD
optional: false
- name: POSTGRES_DB
valueFrom:
secretKeyRef:
name: postgres-credentials
key: POSTGRES_DB
envFrom:
- secretRef: # secretRef: inject ALL keys from a Secret as env vars
name: payment-gateway-credentials # All keys become env vars: API_KEY, API_SECRET
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "300m"
memory: "256Mi"
$ kubectl apply -f payment-processor-deployment.yaml deployment.apps/payment-processor created $ kubectl get pods -n payments NAME READY STATUS RESTARTS AGE payment-processor-8c4f7d-j9pkx 1/1 Running 0 14s payment-processor-8c4f7d-r2skl 1/1 Running 0 14s $ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- env | grep POSTGRES POSTGRES_DB=payments POSTGRES_PASSWORD=s3cur3P@ssw0rd POSTGRES_USER=payment_user
What just happened?
Secrets are decoded automatically — Inside the container, the env var value is the decoded plaintext — not the base64 string. Kubernetes decodes the Secret values before injecting them. The container sees POSTGRES_PASSWORD=s3cur3P@ssw0rd, not the base64 gibberish. Same applies to volume-mounted Secrets — the files contain decoded plaintext.
imagePullSecrets — This field lives at the Pod spec level (not container level) and tells the kubelet which credentials to use when pulling the container image. Without it, a Pod trying to pull from a private registry will fail immediately with ErrImagePull or ImagePullBackOff. You can also attach imagePullSecrets to a ServiceAccount so all Pods using that account inherit the credentials automatically.
optional: false — Setting optional: false on a secretKeyRef means the Pod will not start if the referenced Secret doesn't exist or doesn't have that key. This is fail-fast behaviour — better to surface a misconfiguration immediately than to have a Pod start with a missing password and fail only when it tries its first database query. Set optional: true only if the secret being absent is an acceptable runtime condition.
Mounting Secrets as Files
Some applications expect credentials as files — TLS certificates, SSH keys, service account JSON files. The volume mount approach for Secrets works exactly like ConfigMaps but with one important addition: Kubernetes stores Secret volumes in tmpfs (in-memory filesystem) rather than on disk. This means the Secret data is never written to the node's disk storage, which reduces the risk of it appearing in disk forensics.
The scenario: The payment processor needs to establish a TLS-encrypted connection to the PostgreSQL database. The database requires mutual TLS — the client must present a certificate. You're mounting the TLS Secret as files so the Java JDBC driver can read the cert and key from the filesystem.
kubectl create secret tls postgres-tls \
--cert=client.crt \
--key=client.key \
-n payments
# tls subcommand: creates a kubernetes.io/tls type Secret
# --cert: path to the PEM-encoded certificate file
# --key: path to the PEM-encoded private key file
# Kubernetes stores them under the keys tls.crt and tls.key
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processor
namespace: payments
spec:
replicas: 2
selector:
matchLabels:
app: payment-processor
template:
metadata:
labels:
app: payment-processor
spec:
containers:
- name: payment-processor
image: registry.company.com/payment-processor:3.1.0
volumeMounts:
- name: tls-certs # Must match volumes[].name below
mountPath: /etc/ssl/certs # Path where the cert files will appear in the container
readOnly: true # readOnly: always use true for secrets mounted as files
volumes:
- name: tls-certs
secret: # secret: mount a Secret as a volume (vs configMap:)
secretName: postgres-tls # Which Secret to expose as this volume
defaultMode: 0400 # defaultMode: file permissions in octal
# 0400 = owner read-only — prevents accidental writes
# Private keys should always be 0400 or 0600
$ kubectl apply -f payment-processor-tls.yaml deployment.apps/payment-processor configured $ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- ls -la /etc/ssl/certs total 0 drwxrwxrwt 3 root root 120 Mar 10 09:44 . drwxr-xr-x 1 root root 4096 Mar 10 09:44 .. lrwxrwxrwx 1 root root 14 Mar 10 09:44 tls.crt -> ..data/tls.crt lrwxrwxrwx 1 root root 14 Mar 10 09:44 tls.key -> ..data/tls.key $ kubectl exec -it payment-processor-8c4f7d-j9pkx -n payments -- df -h /etc/ssl/certs Filesystem Size Used Avail Use% Mounted on tmpfs 7.8G 0 7.8G 0% /etc/ssl/certs
What just happened?
tmpfs confirmed — The df -h output shows the mount is on tmpfs — a RAM-backed filesystem. The cert and key files exist in memory, not on the node's disk. If the Pod is killed, the files vanish. This is one of the concrete security benefits of volume-mounted Secrets over env vars — env var values can appear in /proc/[pid]/environ on the node, which is a disk-backed file.
Symlinks to ..data/ — The files are symlinks pointing to a ..data/ directory. This is how Kubernetes implements live Secret updates for volume mounts — it atomically swaps the ..data/ symlink target to a new directory containing the updated content. The files your app reads always point to the latest version.
defaultMode: 0400 — File permissions matter for private keys. A key file with permissions 0644 (world-readable) will be rejected by most TLS libraries with an error like "permissions are too open." The defaultMode: 0400 field sets the file to owner-read-only, which matches what SSH and TLS tools expect.
The Secrets Security Landscape
Here's how the security layers stack — from the weakest default to the strongest hardened setup:
Kubernetes Secrets Security Layers
get/list on Secrets to specific service accounts and users. Developers can deploy but can't read the Secret values. Only the application SA and admins have access.EncryptionConfiguration on the API server. Secrets are AES-encrypted before writing to etcd. Even direct etcd access or backup file leakage doesn't expose plaintext values.Teacher's Note: What level should you be at?
For a production cluster that handles any kind of sensitive data, Level 1 (RBAC) is the absolute minimum — and it costs nothing to implement. Level 2 (etcd encryption) is standard practice on managed Kubernetes services like EKS, GKE, and AKS — they often enable it by default. Level 3 is where security-conscious companies operating at scale land, especially those in regulated industries (finance, healthcare).
A quick trick to see all Secret values in a namespace in one command — and understand why RBAC matters: kubectl get secrets -n payments -o json | jq -r '.items[] | .metadata.name + ": " + (.data | to_entries[] | .key + "=" + (.value | @base64d))'. If you can run that command in your production namespace, so can an attacker who compromises any Pod with a permissive service account.
The lesson: Kubernetes Secrets are a delivery mechanism. The security model around them is your responsibility — not Kubernetes's.
Practice Questions
1. Kubernetes Secret values are stored in etcd using what encoding? And why does this NOT mean they are encrypted?
2. Which Secret manifest field accepts plaintext values and has Kubernetes automatically base64-encode them at apply time — so you don't have to pre-encode values yourself?
3. When a Secret is mounted as a volume into a container, Kubernetes uses what type of in-memory filesystem so the Secret data is never written to the node's disk?
Quiz
1. You need to base64-encode the string mypassword to put it in a Secret's data field. Which command produces the correct result without a trailing newline bug?
2. Which Pod spec field references a kubernetes.io/dockerconfigjson Secret to allow the kubelet to pull images from a private container registry?
3. A container spec uses secretKeyRef with optional: false to reference a key in a Secret that does not exist. What happens?
Up Next · Lesson 21
Environment Variables
All the ways to inject config into containers — static values, ConfigMap refs, Secret refs, Pod metadata, and field references — in one complete lesson.