Terraform Course
Secrets Management
Every real infrastructure deployment has secrets — database passwords, API keys, TLS certificates, OAuth tokens. The question is not whether you have them but where they live. In code is wrong. In state is unavoidable but manageable. In a dedicated secrets manager is right. This lesson covers the patterns that keep secrets out of the wrong places and how to use AWS Secrets Manager and SSM Parameter Store with Terraform.
This lesson covers
Where secrets must never live → AWS Secrets Manager with Terraform → SSM Parameter Store SecureString → Creating secrets with Terraform → Rotating secrets without redeploying → Keeping secrets out of state entirely → HashiCorp Vault basics
Where Secrets Must Never Live
Before patterns, a clear statement of what is forbidden. These three locations are where secrets end up when engineers take shortcuts — and where they get stolen.
In .tf files or .tfvars files committed to Git
Git history is permanent. A secret committed to a repository — even briefly — can be retrieved from history indefinitely. Anyone with repository read access, any CI/CD runner that clones the repo, any engineer who ever pulls — all have access to the credential. Rotating the secret does not help if the old value is still in Git history. The only fix is revocation, history rewriting, and credential rotation across all affected systems.
In terraform.tfvars without gitignore protection
Even a separate terraform.tfvars file is dangerous if it is ever committed. The file itself might be in .gitignore, but a developer running git add . or an IDE that auto-stages files can accidentally commit it. If your workflow requires a .tfvars file, it must be in .gitignore and the CI/CD pipeline must inject values from a secure store — never from a file in the repository.
In Terraform state without backend encryption
Terraform state stores every resource attribute in plaintext JSON — including passwords, private keys, and access tokens that AWS or Azure returns after creating a resource. An unencrypted state file in an S3 bucket with overly permissive IAM is one misconfiguration away from a full credential dump. State encryption via the backend and strict IAM access to the state bucket are non-negotiable for any environment that handles sensitive infrastructure.
AWS Secrets Manager — Reading Existing Secrets
The most common secrets management pattern in Terraform is reading an existing secret from AWS Secrets Manager at plan time and using it as an input to a resource. The secret is stored and managed in Secrets Manager — Terraform never owns it, never creates it, and never stores the value in configuration files.
New terms:
- aws_secretsmanager_secret — the secret container resource. Holds metadata about the secret — name, description, rotation configuration, KMS key. The actual secret value is a separate resource:
aws_secretsmanager_secret_version. - aws_secretsmanager_secret_version — the data source that reads the current value of a secret. The
secret_stringattribute contains the secret value — typically a JSON string for multi-value secrets like database credentials. - jsondecode() — Terraform function that parses a JSON string into a map. Used to extract individual fields from a Secrets Manager secret that stores multiple values as JSON, such as
{"username":"admin","password":"secret"}.
# Reading an existing secret from Secrets Manager
# The secret was created by a human operator or a separate process — not by Terraform
# Step 1: Look up the secret metadata by name
data "aws_secretsmanager_secret" "db" {
name = "prod/rds/master-credentials" # Only the name — never the value — in .tf files
}
# Step 2: Read the current version of the secret value
data "aws_secretsmanager_secret_version" "db" {
secret_id = data.aws_secretsmanager_secret.db.id # Reference by ID from step 1
}
# Step 3: Parse the JSON secret string into a usable map
locals {
# Secrets Manager stores credentials as JSON: {"username":"admin","password":"s3cr3t"}
db_credentials = jsondecode(data.aws_secretsmanager_secret_version.db.secret_string)
}
# Step 4: Use the credentials — they are read at plan time, never stored in code
resource "aws_db_instance" "main" {
identifier = "prod-db"
engine = "postgres"
engine_version = "15.3"
instance_class = "db.t3.medium"
allocated_storage = 100
db_name = "appdb"
username = local.db_credentials.username # From Secrets Manager — not hardcoded
password = local.db_credentials.password # From Secrets Manager — not hardcoded
# Even though the password came from Secrets Manager,
# it will still appear in the state file — ensure state is encrypted
skip_final_snapshot = false
tags = {
Environment = "prod"
ManagedBy = "Terraform"
}
}
$ terraform plan
data.aws_secretsmanager_secret.db: Reading...
data.aws_secretsmanager_secret.db: Read complete [id=arn:aws:secretsmanager:us-east-1:123:secret:prod/rds/master-credentials]
data.aws_secretsmanager_secret_version.db: Reading...
data.aws_secretsmanager_secret_version.db: Read complete [id=prod/rds/master-credentials|AWSCURRENT]
+ aws_db_instance.main {
+ identifier = "prod-db"
+ username = (sensitive value)
+ password = (sensitive value)
}
Plan: 1 to add, 0 to change, 0 to destroy.
# The secret value was read from Secrets Manager during the plan
# It never appeared in any .tf file
# It shows as (sensitive value) in plan output
# It WILL be stored in the state file — ensure state is encryptedWhat just happened?
- The secret was fetched at plan time, not stored anywhere in the repository. The
.tffiles contain only the secret name —prod/rds/master-credentials. The actual username and password live exclusively in Secrets Manager. No Git history contains them. No configuration file contains them. - The two-step data source pattern is intentional. First you look up the secret by name to get its ARN and metadata. Then you read the current version using that ARN. This separation allows the secret name to be a variable — the same Terraform configuration can read different secrets for dev vs prod by passing different secret names.
- Warning: the value still ends up in state. AWS must store the password somewhere to manage the RDS instance. Terraform records it in state. This is why state encryption is non-negotiable when managing databases — not optional, not "nice to have".
SSM Parameter Store SecureString
AWS Systems Manager Parameter Store is a lighter-weight alternative to Secrets Manager for storing configuration values and simple secrets. SecureString parameters are encrypted with KMS — suitable for API keys, connection strings, and non-rotated credentials.
New terms:
- SecureString — an SSM Parameter Store parameter type that encrypts the value with a KMS key. The value is decrypted on retrieval by callers with the appropriate IAM and KMS permissions. Standard String parameters are stored in plaintext — never use them for secrets.
- with_decryption = true — required on the
aws_ssm_parameterdata source to receive the decrypted value. Without it, SecureString parameters return the encrypted ciphertext instead of the plaintext value.
# Reading a SecureString from SSM Parameter Store
data "aws_ssm_parameter" "api_key" {
name = "/prod/app/third-party-api-key"
with_decryption = true # Required to get the plaintext value — omit and you get ciphertext
}
data "aws_ssm_parameter" "db_password" {
name = "/prod/rds/master-password"
with_decryption = true
}
# Use in a resource
resource "aws_lambda_function" "app" {
filename = "app.zip"
function_name = "prod-app"
role = aws_iam_role.lambda.arn
handler = "index.handler"
runtime = "nodejs18.x"
environment {
variables = {
# Inject SSM values as environment variables — Lambda decrypts at runtime
API_KEY = data.aws_ssm_parameter.api_key.value # From SSM SecureString
DB_PASSWORD = data.aws_ssm_parameter.db_password.value
}
}
}
# Secrets Manager vs SSM Parameter Store — which to use?
# Use Secrets Manager when:
# - You need automatic rotation with rotation lambdas
# - You store complex JSON credentials (database username + password)
# - You need cross-account secret sharing
# - Compliance requires dedicated secrets management (PCI-DSS, HIPAA)
#
# Use SSM Parameter Store when:
# - Simple key-value secrets without rotation requirements
# - Configuration values mixed with secrets (same hierarchy)
# - Cost is a factor — SSM SecureString is free, Secrets Manager charges per secret/month
# - You want a hierarchical namespace: /prod/app/key, /dev/app/key
Creating Secrets with Terraform
Sometimes Terraform needs to create the secret container in Secrets Manager while keeping the actual value out of Terraform state. The pattern: Terraform creates the secret shell with no initial value, then a separate process (a rotation lambda, a bootstrap script, or a human operator) populates the value.
# Pattern: Terraform creates the secret container — not the secret value
# The value is populated out of band — by rotation, a bootstrap job, or a human
resource "aws_secretsmanager_secret" "db_credentials" {
name = "prod/rds/master-credentials"
description = "RDS master credentials for the production database"
# Encrypt the secret with a customer-managed KMS key
kms_key_id = aws_kms_key.secrets.arn
# Soft delete protection — prevents accidental deletion of the secret
recovery_window_in_days = 7 # 7-day window before permanent deletion
tags = {
Environment = "prod"
ManagedBy = "Terraform"
}
}
# Rotation configuration — Secrets Manager calls the Lambda on schedule
resource "aws_secretsmanager_secret_rotation" "db_credentials" {
secret_id = aws_secretsmanager_secret.db_credentials.id
rotation_lambda_arn = aws_lambda_function.db_rotation.arn
rotation_rules {
automatically_after_days = 30 # Rotate the password every 30 days automatically
}
}
# The initial value is set outside Terraform — via AWS console, CLI, or bootstrap script
# aws secretsmanager put-secret-value \
# --secret-id prod/rds/master-credentials \
# --secret-string '{"username":"admin","password":"initial-value"}'
#
# This keeps the initial secret value out of Terraform state entirely
# Subsequent rotations are handled by the Lambda — Terraform never sees the value
What just happened?
- Terraform owns the secret container, not the secret value. The
aws_secretsmanager_secretresource creates the name, KMS key association, rotation schedule, and metadata. The actual password is never in any.tffile and never in the Terraform state file. The shell is infrastructure. The value is a secret — they are managed separately. - Automatic rotation means Terraform never needs to touch the value again. The rotation Lambda updates the password in Secrets Manager and in RDS on the defined schedule. Terraform's job was to set up the rotation plumbing — not to know what the password is at any given moment.
Rotating Secrets Without Redeploying Infrastructure
A common misconception: if a secret rotates in Secrets Manager, Terraform needs to re-apply to pick up the new value. This is only true if Terraform is the one injecting the secret into a resource at apply time. When the application reads the secret directly from Secrets Manager at runtime — not via Terraform — rotation requires no Terraform involvement at all.
# Architecture that allows secret rotation without any Terraform apply
# Approach: Grant the application's IAM role permission to read from Secrets Manager
# The application calls Secrets Manager directly at runtime — not via Terraform
resource "aws_iam_role_policy" "app_secrets" {
name = "app-secrets-access"
role = aws_iam_role.app.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"secretsmanager:GetSecretValue", # Read the secret value at runtime
"secretsmanager:DescribeSecret" # Read secret metadata
]
Resource = aws_secretsmanager_secret.db_credentials.arn # Only this specific secret
}]
})
}
# The application code reads the secret at startup:
# const secret = await secretsManagerClient.getSecretValue({ SecretId: 'prod/rds/master-credentials' })
# const creds = JSON.parse(secret.SecretString)
# const dbPassword = creds.password
#
# When the secret rotates:
# → Secrets Manager calls the rotation Lambda
# → Lambda updates the RDS password and the secret value atomically
# → On next application startup, the app fetches the new value automatically
# → Zero Terraform involvement — zero infrastructure redeployment
# Contrast with the injection approach (Terraform injects at apply time):
# → Secret value goes into Terraform state
# → To use a rotated secret, must run terraform apply again
# → Application downtime window between rotation and redeploy
# → The runtime approach is always preferred for high-availability systems
HashiCorp Vault Basics with Terraform
HashiCorp Vault is a dedicated secrets management platform — more powerful than Secrets Manager for complex requirements like dynamic database credentials, PKI certificate management, and multi-cloud secrets. Terraform has a Vault provider for reading secrets during plan/apply.
# Using HashiCorp Vault with Terraform
terraform {
required_providers {
vault = {
source = "hashicorp/vault"
version = "~> 3.0"
}
}
}
# Configure the Vault provider — address and authentication method
provider "vault" {
address = "https://vault.acme.internal:8200"
# Authentication via AWS IAM role — no token hardcoded
# The Vault server validates the caller's AWS identity against Vault policies
# Set VAULT_TOKEN or VAULT_ROLE environment variable in CI/CD
}
# Read a secret from Vault's KV secrets engine
data "vault_generic_secret" "db" {
path = "secret/prod/database" # Path in Vault where the secret is stored
}
# Use the secret value from Vault
resource "aws_db_instance" "main" {
identifier = "prod-db"
engine = "postgres"
instance_class = "db.t3.medium"
username = data.vault_generic_secret.db.data["username"] # From Vault
password = data.vault_generic_secret.db.data["password"] # From Vault
# Note: Vault dynamic secrets generate a unique credential for each lease
# Dynamic credentials expire automatically — far more secure than static passwords
# Requires the Vault database secrets engine to be configured for the RDS instance
}
Common Secrets Management Mistakes
Using aws_secretsmanager_secret_version as a resource to set a secret value
When you use aws_secretsmanager_secret_version as a resource (not a data source), the secret value is an argument — which means it goes into Terraform state in plaintext. If you must use Terraform to set an initial value, use the lifecycle ignore_changes = [secret_string] immediately so future rotations do not cause Terraform to revert to the initial value. Better still — set the initial value out of band and never put it in Terraform at all.
Injecting secrets into EC2 user data or container environment variables via Terraform
When a secret is passed to user_data on an EC2 instance or as an environment variable in an ECS task definition via Terraform, the value goes into state. It also typically ends up in CloudTrail API logs and instance metadata. Applications should retrieve secrets from Secrets Manager or SSM at runtime — not have them injected at provisioning time by Terraform.
Forgetting with_decryption = true on SSM SecureString data sources
Without with_decryption = true, the aws_ssm_parameter data source returns the KMS-encrypted ciphertext instead of the plaintext value. Your application receives a base64 blob and fails silently or with a cryptic error. Always set with_decryption = true when reading SecureString parameters.
The golden rule of secrets and Terraform
Terraform should manage the existence of secrets — the name, the encryption key, the rotation schedule, the IAM access policies. Terraform should not manage the secret value if you can avoid it. When applications read secrets directly from Secrets Manager or Vault at runtime, secret rotation requires no Terraform involvement, no redeployment, and no state file updates. This is the architecture that scales and stays secure under operational pressure.
Practice Questions
1. Which Terraform function parses the JSON string stored in a Secrets Manager secret so you can access individual fields like username and password?
2. Which argument must you set on an aws_ssm_parameter data source to receive the plaintext value of a SecureString parameter instead of the ciphertext?
3. What architecture allows a secret to rotate in Secrets Manager without requiring a terraform apply to pick up the new value?
Quiz
1. What is the recommended pattern for creating an AWS Secrets Manager secret with Terraform while keeping the secret value out of Terraform state?
2. When should you choose SSM Parameter Store over AWS Secrets Manager for storing a secret?
3. You read a database password from Secrets Manager using a data source and pass it to aws_db_instance. Is the password protected from appearing in the state file?
Up Next · Lesson 31
Terraform with AWS
Secrets managed. Lesson 31 goes deep into the AWS provider — the full authentication credential chain, multi-account deployments with assume_role, multi-region with provider aliases, default_tags, and the key resource patterns every AWS Terraform engineer needs to know.