Ansible Lesson 29 – Managing Secrets Securely | Dataplexa
Section III · Lesson 29

Managing Secrets Securely

In this lesson

Secrets strategy External secrets managers HashiCorp Vault AWS Secrets Manager CI/CD pipelines

Managing secrets securely means choosing the right tool for each secret, at each scale, in each environment — and building the habits that prevent credentials from leaking through the gaps between tools. Ansible Vault from Lesson 28 is the right foundation for most teams. But production infrastructure at scale introduces additional requirements: secrets that rotate automatically, audit logs showing who accessed what, dynamic credentials that expire after use, and integration with cloud-native secrets services. This lesson covers the full secrets management landscape — from the simplest vault-password-file setup to HashiCorp Vault and AWS Secrets Manager integration — so you can choose the right approach for your team's security requirements and operational maturity.

Choosing Your Secrets Strategy

Not every team needs HashiCorp Vault. The right strategy depends on team size, compliance requirements, and operational maturity. Over-engineering secrets management wastes time; under-engineering it causes incidents. These four tiers map to progressively more complex and capable approaches.

Tier 1 — Small team

Ansible Vault + password file

All secrets in vars/secrets.yml, encrypted with ansible-vault encrypt. Password in .vault_pass on each engineer's machine. Shared via a team password manager. Zero external dependencies.

Tier 2 — CI/CD pipeline

Ansible Vault + CI environment secrets

Same encrypted files in Git. Vault password stored as a CI/CD secret (GitHub Actions secret, GitLab CI variable). Pipeline injects it at runtime. No password file on disk.

Tier 3 — Cloud-native

AWS / GCP / Azure Secrets Manager

Secrets stored in cloud-native services. Ansible retrieves them at runtime using the aws_secret or azure_keyvault_secret lookup plugins. Automatic rotation, fine-grained IAM, and full audit logging.

Tier 4 — Enterprise

HashiCorp Vault

Centralised secrets engine with dynamic credentials, lease expiry, policy-based access control, and a complete audit trail. Works across cloud and on-premise. The industry standard for enterprise secret management.

Tier 2 — Vault in CI/CD Pipelines

The most common production setup for teams that use Ansible Vault is injecting the vault password through the CI/CD pipeline's own secrets mechanism. The encrypted files are in Git; the key is not. The pipeline assembles them at runtime without any engineer having to manage a password file manually.

Step 1

Store the vault password as a CI/CD secret

In GitHub Actions: Settings → Secrets → Actions → New repository secret. Name it VAULT_PASSWORD. The value is the same password in your local .vault_pass. It is stored encrypted by GitHub and never visible in logs.

Step 2

Write the workflow file

# .github/workflows/deploy.yml
name: Deploy to production

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Ansible and dependencies
        run: |
          pip install ansible
          ansible-galaxy install -r requirements.yml

      - name: Write vault password file
        run: echo "${{ secrets.VAULT_PASSWORD }}" > .vault_pass
        # The file exists only for the duration of this job

      - name: Run deployment playbook
        run: |
          ansible-playbook deploy.yml \
            -i inventory/production/ \
            --vault-password-file .vault_pass

      - name: Remove vault password file
        if: always()    # runs even if the playbook fails
        run: rm -f .vault_pass

Step 3

Alternative — use environment variable instead of a file

      - name: Run deployment playbook (env var method)
        env:
          ANSIBLE_VAULT_PASSWORD: ${{ secrets.VAULT_PASSWORD }}
        run: |
          # Write password from env var to a temp file Ansible can read
          echo "$ANSIBLE_VAULT_PASSWORD" > /tmp/.vault_pass
          ansible-playbook deploy.yml \
            -i inventory/production/ \
            --vault-password-file /tmp/.vault_pass
          rm /tmp/.vault_pass

The Hotel Key Card Analogy

Managing secrets in a CI/CD pipeline is like hotel key cards. The hotel (CI/CD system) holds the master key (vault password) in a secure cabinet. When a guest (pipeline job) needs access to a room (encrypted secrets), the hotel issues a temporary key card valid only for that stay (job execution). The card is returned or destroyed when the guest checks out (job completes). No guest ever possesses the master key directly — they only ever hold a temporary, scoped credential issued by the system that manages it.

Tier 3 — AWS Secrets Manager Integration

Cloud-native secrets managers offer capabilities beyond what Ansible Vault provides: automatic rotation, fine-grained IAM access controls, versioned secret history, and a full audit log in CloudTrail. The amazon.aws collection provides a lookup plugin that retrieves secrets at playbook runtime — no local copy of the secret ever exists.

# Install the AWS collection
ansible-galaxy collection install amazon.aws
---
- name: Deploy application with AWS Secrets Manager
  hosts: appservers
  become: true

  vars:
    # Fetch secrets at runtime from AWS Secrets Manager
    # The IAM role on the control node must have secretsmanager:GetSecretValue
    db_password: "{{ lookup('amazon.aws.aws_secret',
                    'prod/myapp/db_password',
                    region='us-east-1') }}"

    api_key: "{{ lookup('amazon.aws.aws_secret',
                 'prod/myapp/api_key',
                 region='us-east-1') }}"

  tasks:
    - name: Deploy application configuration
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/app/config.conf
      # In app.conf.j2:
      # db_password = {{ db_password }}
      # api_key     = {{ api_key }}
      no_log: true    # prevent secret values from appearing in output

How the lookup works

The lookup plugin runs on the control node at the moment the variable is evaluated. It calls the AWS Secrets Manager API using the control node's IAM credentials, retrieves the current secret value, and returns it as a string. The secret never touches the filesystem — it exists only in memory for the duration of the playbook run. When AWS rotates the secret, the next playbook run automatically picks up the new value.

Tier 4 — HashiCorp Vault Integration

HashiCorp Vault is the industry standard for enterprise secrets management. It provides dynamic credentials (database passwords that are created per-request and expire automatically), policy-based access control, and a complete audit log. Ansible integrates with it via the hashi_vault lookup plugin in the community.hashi_vault collection.

# Install the HashiCorp Vault collection
ansible-galaxy collection install community.hashi_vault

# Set the Vault address and token in environment (or ansible.cfg)
export VAULT_ADDR=https://vault.example.com
export VAULT_TOKEN=s.abc123xyz
---
- name: Deploy with HashiCorp Vault secrets
  hosts: databases
  become: true

  vars:
    # KV secrets engine — retrieve a static secret
    db_password: "{{ lookup('community.hashi_vault.hashi_vault',
                    'secret=secret/data/myapp/db_password:value
                     url=https://vault.example.com
                     auth_method=token') }}"

    # Dynamic database credentials — created per-request, expire automatically
    # Requires a Vault database secrets engine configured for PostgreSQL
    pg_credentials: "{{ lookup('community.hashi_vault.hashi_vault',
                         'secret=database/creds/myapp-role
                          url=https://vault.example.com
                          auth_method=approle
                          role_id={{ vault_role_id }}
                          secret_id={{ vault_secret_id }}') }}"

  tasks:
    - name: Create database user with dynamic credentials
      community.postgresql.postgresql_user:
        name: "{{ pg_credentials.username }}"
        password: "{{ pg_credentials.password }}"
        state: present
      become_user: postgres
      no_log: true
      # These credentials expire in 1 hour — Vault creates and revokes them

HashiCorp Vault auth methods for Ansible

token A Vault token directly. Simple for development. Tokens expire — not ideal for long-running CI pipelines without renewal logic.
approle A role ID and secret ID pair — designed for machine-to-machine authentication. The standard method for CI/CD pipelines and control nodes accessing Vault.
aws_iam Authenticate using the control node's AWS IAM role. Zero credentials to manage — the EC2 instance identity itself becomes the Vault credential.
ldap Authenticate using Active Directory or LDAP credentials. Suitable when engineers run playbooks interactively and already have AD credentials.

Secrets Hygiene — Universal Rules

These rules apply regardless of which secrets strategy you choose. They are the baseline that every Ansible project handling real credentials must meet.

🚫

Never log secrets — always use no_log: true

Any task whose arguments contain a credential must set no_log: true. Ansible output goes to terminals, CI logs, monitoring systems, and log aggregators — all of which persist the output. A single uncovered password task can expose credentials in a dozen places.

🔄

Rotate credentials when team membership changes

When an engineer leaves, rotate every secret they had access to — vault passwords, API keys, and service account credentials. This is a process, not a technical control, but it prevents the most common post-departure exposure scenario.

🔍

Scan Git history before making a repo public

Tools like git-secrets, truffleHog, and gitleaks scan every commit in a repository's history for credential patterns. Run one before changing a private repo to public — secrets committed and then deleted still exist in git history.

🏷️

Name all vault variables with a vault_ prefix

vault_db_password not db_password. This makes encrypted variables visually distinct in any file — critical for code reviews where a reviewer needs to quickly identify which values are protected.

📂

Separate secrets files by sensitivity level

Maintain separate vault files for different sensitivity tiers: vars/secrets.yml for application secrets and vars/infra_secrets.yml for infrastructure credentials. This allows Vault IDs to assign different passwords to different files and limits exposure when one password is compromised.

📋

Document what each secret is for in a README

Every secret in vars/secrets.yml should have a corresponding entry in a non-committed SECRETS.md (or in the encrypted file's comments) explaining what the secret is used for and how to rotate it. When a rotation is required at 2am, this document is invaluable.

Vault Password Scripts

Instead of a static password file, you can point vault_password_file at an executable script. Ansible runs the script and uses whatever it prints to stdout as the vault password. This is how you connect Ansible Vault to any external secrets manager — the script handles the retrieval, Ansible handles the decryption.

#!/usr/bin/env python3
# scripts/get_vault_password.py
# Makes this file executable: chmod +x scripts/get_vault_password.py

import boto3
import json
import sys

def get_vault_password():
    """Retrieve vault password from AWS Secrets Manager."""
    client = boto3.client('secretsmanager', region_name='us-east-1')
    try:
        response = client.get_secret_value(SecretId='ansible/vault-password')
        # Secret stored as plain string
        print(response['SecretString'])
    except Exception as e:
        print(f"Error retrieving vault password: {e}", file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    get_vault_password()
# ansible.cfg — point to the script instead of a static file
[defaults]
vault_password_file = ./scripts/get_vault_password.py
# The script must be executable
chmod +x scripts/get_vault_password.py

# Test the script directly before using it with Ansible
./scripts/get_vault_password.py    # should print the vault password to stdout

# Now ansible-playbook automatically calls the script at runtime
ansible-playbook site.yml -i inventory/production/

Deleted Secrets Still Exist in Git History

If a plaintext secret is ever committed to a Git repository — even for a moment, even to a private repo — it must be treated as compromised. Simply deleting the file and committing again does not remove it from the repository's history. Anyone who clones the repo can access every commit, including the one that contained the secret. The correct response is to rotate the exposed credential immediately and use git filter-branch or BFG Repo Cleaner to purge the secret from history — then force-push and require all team members to re-clone. This process takes hours and causes operational disruption. Prevention via pre-commit hooks and CI scanning is far less painful.

Key Takeaways

Choose the right tier for your team's scale — Ansible Vault plus a password file is sufficient for most small teams. Layer on cloud-native secrets managers or HashiCorp Vault when you need automatic rotation, audit logs, or dynamic credentials.
CI/CD pipelines should inject the vault password via their own secret mechanism — GitHub Actions secrets, GitLab CI variables, or environment variables. Never commit the password to the repo or hardcode it in the workflow file.
Vault password scripts connect Ansible Vault to any external secrets manager — point vault_password_file at an executable script that retrieves the password from wherever you store it.
Always use no_log: true on tasks that handle credentials — regardless of which secrets manager you use, task output exposes the decrypted value unless suppressed.
Treat any committed plaintext secret as compromised immediately — rotate it before removing it from history. Deletion without rotation leaves the window of exposure open indefinitely.

Teacher's Note

If your project uses GitHub Actions, set up the Tier 2 pattern from this lesson today — create a VAULT_PASSWORD repository secret and update your workflow file to write it to .vault_pass before running Ansible. The five-minute setup eliminates the need for any engineer to manage a local password file for CI deployments.

Practice Questions

1. Which task attribute must be set on every task that handles a decrypted credential to prevent the value from appearing in Ansible's output or CI logs?



2. Which ansible.cfg setting can point to an executable script that retrieves the vault password from an external secrets manager at runtime?



3. A plaintext API key was accidentally committed to a Git repository and then deleted in the next commit. What must be done with the API key before anything else?



Quiz

1. A GitHub Actions workflow needs to run an Ansible playbook that uses Ansible Vault-encrypted variables. What is the correct approach for providing the vault password?


2. A security team requires that database passwords used by Ansible never persist longer than one hour and are unique per deployment run. Which secrets approach satisfies this?


3. An engineer committed a plaintext database password to a private GitHub repo by mistake, noticed immediately, and deleted the file in the next commit. Is the secret safe?


Up Next · Lesson 30

Application Deployment

Learn to automate full application deployment cycles — rolling releases, blue-green deployments, symlink-based release management, health checks, and automated rollback when a deployment fails.