Rollback Strategies

A rollback is the act of returning a production environment to a previously known-good state after a deployment introduces a problem. It is not a failure of CI/CD — it is evidence that the safety net works. Every team that deploys software to production will eventually need to roll back. The question is not whether rollback will be needed, but whether the team has a defined, practiced, and fast procedure for doing it when the moment comes. A rollback that takes four hours under pressure is qualitatively different from one that takes four minutes and runs automatically.

Rollback vs Roll Forward — Choosing the Right Response

Not every production problem calls for a rollback. The first decision after detecting an incident is whether to roll back to the previous version or roll forward with a fix. Each approach has a different risk profile and a different time cost, and the right choice depends on the nature and severity of the problem.

Artifact-Based Rollback — The Fastest Path to Recovery

The most reliable rollback mechanism in a CI/CD pipeline is artifact-based rollback: redeploying a specific previous artifact directly from the registry, identified by its commit SHA or version tag. Because the previous artifact is already built, tested, and stored — and because the deployment process is automated and scripted — the rollback is essentially a re-run of the deploy job with a different artifact reference. No code changes required, no build step, no pipeline from scratch.

This is why artifact immutability and retention policies (covered in Lesson 14) are not administrative details — they are prerequisites for fast rollback. If previous artifacts are overwritten or deleted on a short retention schedule, the option to roll back to a specific version disappears. The artifact must still exist in the registry, under its original tag, in exactly the state it was when it was deployed.

Database Migrations — The Rollback Complication

Application code is easy to roll back. Database schema changes are not. A database migration that adds a column, renames a table, or drops a field changes the persistent state of the system in ways that may be irreversible — or at least expensive to reverse. Rolling back the application code while leaving the migrated schema in place can produce a mismatch that causes the previous version to fail immediately.

The industry-standard approach to making migrations rollback-safe is the expand-contract pattern. Instead of making a breaking schema change in a single migration, the change is split across multiple deployments. First, the new column or table is added alongside the old one (expand). The application is updated to write to both. After several deployment cycles of confirmed stability, the old column is removed (contract). At any point during the expand phase, rolling back the application code is safe because the old schema is still present. By the time the contract migration runs, the team is confident no rollback is needed.

Automated Rollback — Making Recovery Instant

Automated rollback triggers a redeployment of the previous artifact without human intervention, based on a defined failure signal — a failed smoke test, a spike in error rate from the observability platform, a health check that stops returning 200. The pipeline (or a monitoring integration) detects the signal, identifies the last known-good artifact, and deploys it immediately.

Automated rollback is the gold standard for production stability, but it requires two preconditions: the failure signal must be reliable enough that false positives do not trigger unnecessary rollbacks, and the rollback itself must be tested regularly so the team knows it works when needed. A rollback procedure that has never been tested in production is a procedure the team cannot trust in an incident. Platforms like Kubernetes, AWS ECS, and deployment tools like Argo Rollouts support automated rollback natively.

jobs:
  deploy-production:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy new artifact
        run: ./deploy.sh production ${{ github.sha }}

      - name: Run smoke tests
        id: smoke
        run: ./smoke-test.sh https://myapp.com
        continue-on-error: true          # Don't fail the job yet — check the result first

      - name: Rollback on smoke test failure
        if: steps.smoke.outcome == 'failure'
        run: |
          echo "Smoke tests failed — rolling back to ${{ vars.LAST_GOOD_SHA }}"
          ./deploy.sh production ${{ vars.LAST_GOOD_SHA }}   # Redeploy previous known-good artifact
          ./smoke-test.sh https://myapp.com                  # Verify rollback succeeded

      - name: Fail the job if smoke tests failed
        if: steps.smoke.outcome == 'failure'
        run: exit 1                      # Mark the workflow as failed after rollback completes

Practice Questions

Answer in your own words — then check against the expected answer.

Lesson Quiz