Monitoring CI/CD Pipelines

Pipeline monitoring is the practice of collecting, visualising, and alerting on data about the CI/CD system itself — not just the applications it deploys, but the pipeline's own health, speed, reliability, and delivery performance. A team that does not measure its pipeline cannot systematically improve it. Pipeline durations trend upward unnoticed. Flaky tests accumulate. Deployment frequency drops. Change failure rates rise. These patterns are invisible without measurement — and without measurement, the pipeline degrades slowly while the team attributes each symptom to something else. Monitoring the pipeline is how teams turn CI/CD from a tool into a continuously improving system.

DORA Metrics — The Industry Standard for Delivery Performance

The DORA metrics — from the DevOps Research and Assessment programme — are the four measurements most strongly correlated with software delivery performance and organisational outcomes. They are the closest thing the industry has to a standard for measuring how well a CI/CD system is working. Teams that track these metrics have a shared vocabulary for describing delivery capability and a consistent basis for improvement.

Pipeline Health Signals — What to Measure Beyond DORA

DORA metrics measure delivery outcomes. Pipeline health signals measure the pipeline's internal condition — the leading indicators that predict whether DORA metrics will deteriorate before they actually do. A team that tracks only DORA metrics sees problems after they have already affected delivery. A team that also tracks pipeline health signals sees problems forming and can intervene before they land.

Alerting on Pipeline Failures — Closing the Loop Quickly

A pipeline failure that nobody notices for an hour is an hour of blocked delivery. A main branch that is broken and stays broken while the team works on unrelated things is a compounding problem — every subsequent commit stacks on top of the broken state, making the eventual fix harder. Pipeline failure alerting ensures that a broken main branch is treated as an incident: visible, urgent, and resolved before new work is stacked on top of it.

The most common pattern is Slack notification on pipeline failure for the main branch, sent immediately to a dedicated channel that the team monitors. GitHub Actions supports this natively through webhook-based notification steps or through community actions. The notification should include the commit SHA, the author, the failing job name, and a direct link to the pipeline run — everything the engineer needs to start investigating without navigating the GitHub UI first.

Deployment Observability — What Happens After the Pipeline

The pipeline's job does not end when the deployment completes. A deployment that succeeds at the infrastructure level — new pods running, health checks passing — can still introduce production issues that only become visible through application-level observability: error rates rising, latency increasing, specific user flows failing. Deployment markers are the mechanism that connects pipeline events to observability data.

When a pipeline completes a deployment, it should emit a deployment event to the observability platform — Datadog, Grafana, New Relic, or equivalent. This event appears as a vertical marker on every dashboard, aligning the timing of any metric change with the exact deployment that caused it. An error rate spike that begins 3 minutes after a deployment marker is almost certainly caused by that deployment. Without the marker, the correlation requires manual archaeology through logs and deployment history. With it, the on-call engineer sees the cause immediately in the dashboard they are already watching.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy to production
        run: ./deploy.sh production ${{ github.sha }}

      - name: Send deployment marker to Datadog
        if: success()
        run: |
          curl -X POST "https://api.datadoghq.com/api/v1/events" \
            -H "DD-API-KEY: ${{ secrets.DATADOG_API_KEY }}" \
            -d '{
              "title": "Deployment: api",
              "text": "Deployed ${{ github.sha }} to production",
              "tags": ["env:production", "service:api"],
              "alert_type": "info"
            }'
          # Appears as a marker on all Datadog dashboards — correlates metric changes to this deploy

      - name: Alert on deployment failure
        if: failure()
        uses: slackapi/slack-github-action@v1.26.0
        with:
          channel-id: 'C0DEPLOYALERTS'
          slack-message: |
            :red_circle: *Production deployment FAILED*
            Service: `api`
            Commit: `${{ github.sha }}`
            Author: ${{ github.actor }}
            <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View pipeline run>
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}

Practice Questions

Answer in your own words — then check against the expected answer.

Lesson Quiz