CI/CD Course
Mini Project
In this lesson
This is the final lesson of the Dataplexa CI/CD Course. Thirty-nine lessons have covered every foundational concept, architectural pattern, security practice, deployment strategy, and organisational discipline that constitutes modern CI/CD. This lesson applies all of it. Rather than introducing new concepts, it presents a realistic engineering scenario and walks through the complete pipeline design and implementation — from requirements analysis through production deployment — using the tools, patterns, and principles covered throughout the course. Work through it actively. Every decision point has a rationale. Every YAML block has a lesson reference. This is CI/CD from commit to production, end to end.
The Scenario
You are the first platform engineer at Vaultex, a Series A fintech startup building an API that handles payment processing for e-commerce merchants. The engineering team has grown from 3 to 12 developers in six months. They currently deploy manually — a developer SSH-es into the production server, pulls the latest code, and runs a restart script. Deployments happen at 11pm to avoid business hours. The last deployment took three hours and caused 45 minutes of downtime. Two developers have handed in notice, citing deployment anxiety as a contributing factor.
The CTO has tasked you with building a complete CI/CD system from scratch. The application is a Node.js REST API deployed on AWS ECS. The database is PostgreSQL on AWS RDS. The source code lives in a GitHub monorepo containing the API service and a shared utilities package. The company processes payment data, so there are compliance requirements: every production deployment must have a named approver, all pipeline runs must be auditable, and credentials must never appear in logs.
Vaultex — Current State vs Target State
Requirements Analysis — Mapping Needs to Pipeline Decisions
Before writing a single line of YAML, every requirement should map to a specific pipeline decision. This is the discipline that separates a pipeline designed for the context from one assembled from generic templates.
Requirements → Pipeline Decisions
main. All workflow runs logged automatically via GitHub audit log.permissions: contents: read at workflow level. GitHub automatically masks secret values in log output.services/api/** and packages/utils/**. Changes to the shared utils package trigger the API service pipeline since the API depends on it.aws ecs wait services-stable blocks the pipeline until confirmed.continue-on-error: true. On failure, ECS is reverted to the previous task definition revision. Previous image retained in ECR for 30 days via lifecycle policy.The Full Pipeline Implementation
The Vaultex pipeline is structured in five stages. Each stage maps to a specific section of this course. The stage order follows the cheapest-first principle: fast quality checks before the build, the build before tests, tests before deployment, deployment before production.
Stage 1 — Quality Gate (PR push trigger)
# .github/workflows/api-pipeline.yml
name: Vaultex API Pipeline
on:
push:
branches: [main]
paths:
- 'services/api/**'
- 'packages/utils/**' # Changes to shared package also trigger API pipeline
pull_request:
branches: [main]
paths:
- 'services/api/**'
- 'packages/utils/**'
permissions: # Lesson 26 — minimal token permissions at workflow level
contents: read
jobs:
quality: # Lesson 17 — static analysis first, fastest feedback
name: Quality Gate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Format check
run: npx prettier --check "services/api/src/**" "packages/utils/src/**"
- name: Lint
run: npx eslint "services/api/src/**" "packages/utils/src/**"
- name: Type check
run: npx tsc --noEmit --project services/api/tsconfig.json
- name: Secret scanning
uses: gitleaks/gitleaks-action@v2 # Lesson 25 — detect accidentally committed secretsStage 2 — Build and Security Scan
build: # Lesson 12 — build once, produce the artifact
name: Build and Scan
needs: quality # Lesson 18 — only runs if quality gate passes
runs-on: ubuntu-latest
permissions:
contents: read
packages: write # Lesson 26 — job-level override for ECR push
id-token: write # Lesson 23 — OIDC token for AWS auth
outputs:
image-tag: ${{ github.sha }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3 # Lesson 28 — BuildKit for layer cache export
- name: Authenticate to AWS via OIDC # Lesson 23 — no stored credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_BUILD_ROLE }}
aws-region: eu-west-1
- name: Login to ECR
uses: aws-actions/amazon-ecr-login@v2
- name: Build multi-stage Docker image # Lesson 27 — multi-stage for lean runtime image
uses: docker/build-push-action@v5
with:
context: .
file: services/api/Dockerfile
push: false # Build but don't push yet — scan first
tags: ${{ secrets.ECR_REGISTRY }}/vaultex-api:${{ github.sha }}
cache-from: type=registry,ref=${{ secrets.ECR_REGISTRY }}/vaultex-api:cache
cache-to: type=registry,ref=${{ secrets.ECR_REGISTRY }}/vaultex-api:cache,mode=max
load: true
- name: Scan image for vulnerabilities # Lesson 27 — scan before pushing to registry
uses: aquasecurity/trivy-action@v0.16.0
with:
image-ref: ${{ secrets.ECR_REGISTRY }}/vaultex-api:${{ github.sha }}
exit-code: '1'
severity: 'CRITICAL,HIGH'
ignore-unfixed: true
- name: Dependency audit # Lesson 13 — SCA on every build
run: npm audit --audit-level=high --prefix services/api
- name: Push image to ECR # Only pushes after scan passes
run: docker push ${{ secrets.ECR_REGISTRY }}/vaultex-api:${{ github.sha }}Stage 3 — Test Suite (Parallel)
unit-tests: # Lesson 15 — unit tests, fast, no dependencies
name: Unit Tests
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '20', cache: 'npm' }
- run: npm ci
- run: npm run test:unit --prefix services/api -- --coverage
- name: Enforce coverage threshold
run: npx jest --coverage --coverageThreshold='{"global":{"lines":80}}'
--prefix services/api
integration-tests: # Lesson 27 — real PostgreSQL via service container
name: Integration Tests
needs: build
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_USER: vaultex_test
POSTGRES_PASSWORD: testpass
POSTGRES_DB: vaultex_test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: '20', cache: 'npm' }
- run: npm ci
- name: Run database migrations against test DB # Lesson 20 — migrations before tests
run: npx flyway migrate
env:
FLYWAY_URL: jdbc:postgresql://localhost:5432/vaultex_test
FLYWAY_USER: vaultex_test
FLYWAY_PASSWORD: testpass
- run: npm run test:integration --prefix services/api
env:
DATABASE_URL: postgresql://vaultex_test:testpass@localhost:5432/vaultex_test
sast: # Lesson 16 — SAST runs in parallel with tests
name: SAST
needs: build
runs-on: ubuntu-latest
permissions:
contents: read
security-events: write # Required for CodeQL to upload findings
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with: { languages: javascript }
- uses: github/codeql-action/analyze@v3Stage 4 — Deploy to Staging and Verify
deploy-staging:
name: Deploy to Staging
needs: [unit-tests, integration-tests, sast] # All three parallel jobs must pass
runs-on: ubuntu-latest
environment: staging # Lesson 26 — staging-scoped secrets only
if: github.ref == 'refs/heads/main' # Only deploy on merge to main
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_STAGING_DEPLOY_ROLE }}
aws-region: eu-west-1
- name: Run database migrations on staging # Lesson 20 — expand-contract enforced
run: |
aws ecs run-task \
--cluster vaultex-staging \
--task-definition vaultex-api-migrate \
--overrides '{"containerOverrides":[{"name":"migrate","environment":[
{"name":"IMAGE_TAG","value":"${{ github.sha }}"}]}]}'
- name: Update ECS service — staging # Lesson 29 — ECS rolling deployment
run: |
aws ecs update-service \
--cluster vaultex-staging \
--service vaultex-api \
--force-new-deployment \
--task-definition vaultex-api:${{ github.sha }}
aws ecs wait services-stable \
--cluster vaultex-staging \
--services vaultex-api # Lesson 29 — always wait for confirmation
- name: Smoke tests — staging # Lesson 16 — smoke test after every deploy
run: |
./scripts/smoke-test.sh \
https://api.staging.vaultex.io \
${{ secrets.SMOKE_TEST_API_KEY }}
- name: Send deployment marker to Datadog # Lesson 30 — correlate metrics to deploys
if: success()
run: |
curl -X POST "https://api.datadoghq.com/api/v1/events" \
-H "DD-API-KEY: ${{ secrets.DATADOG_API_KEY }}" \
-d '{"title":"Deploy: vaultex-api staging","text":"${{ github.sha }}",
"tags":["env:staging","service:api"],"alert_type":"info"}'Stage 5 — Production Deployment with Approval Gate and Rollback
deploy-production:
name: Deploy to Production
needs: deploy-staging
runs-on: ubuntu-latest
environment: production # Lesson 26 — required reviewer, self-review prevented,
# deployment branch restricted to main
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.AWS_PROD_DEPLOY_ROLE }} # Separate role for prod
aws-region: eu-west-1
- name: Run database migrations on production
run: |
aws ecs run-task \
--cluster vaultex-production \
--task-definition vaultex-api-migrate \
--overrides '{"containerOverrides":[{"name":"migrate","environment":[
{"name":"IMAGE_TAG","value":"${{ github.sha }}"}]}]}'
- name: Record current task definition for rollback
id: current
run: |
CURRENT=$(aws ecs describe-services \
--cluster vaultex-production \
--services vaultex-api \
--query 'services[0].taskDefinition' \
--output text)
echo "task-def=$CURRENT" >> $GITHUB_OUTPUT
- name: Deploy to production ECS
run: |
aws ecs update-service \
--cluster vaultex-production \
--service vaultex-api \
--force-new-deployment \
--task-definition vaultex-api:${{ github.sha }}
aws ecs wait services-stable \
--cluster vaultex-production \
--services vaultex-api
- name: Smoke tests — production # Lesson 20 — automated rollback on failure
id: smoke
run: |
./scripts/smoke-test.sh \
https://api.vaultex.io \
${{ secrets.SMOKE_TEST_API_KEY }}
continue-on-error: true
- name: Rollback on smoke test failure
if: steps.smoke.outcome == 'failure'
run: |
echo "Smoke tests failed — rolling back to ${{ steps.current.outputs.task-def }}"
aws ecs update-service \
--cluster vaultex-production \
--service vaultex-api \
--task-definition ${{ steps.current.outputs.task-def }}
aws ecs wait services-stable \
--cluster vaultex-production \
--services vaultex-api
- name: Alert on failure
if: steps.smoke.outcome == 'failure'
uses: slackapi/slack-github-action@v1.26.0
with:
channel-id: ${{ secrets.SLACK_ONCALL_CHANNEL }}
slack-message: |
:red_circle: *PRODUCTION DEPLOYMENT FAILED — ROLLING BACK*
Service: vaultex-api | SHA: `${{ github.sha }}`
Approver: ${{ github.actor }}
<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
- name: Fail job if smoke tests failed
if: steps.smoke.outcome == 'failure'
run: exit 1
- name: Send deployment marker to Datadog
if: success()
run: |
curl -X POST "https://api.datadoghq.com/api/v1/events" \
-H "DD-API-KEY: ${{ secrets.DATADOG_API_KEY }}" \
-d '{"title":"Deploy: vaultex-api production","text":"${{ github.sha }}",
"tags":["env:production","service:api"],"alert_type":"success"}'What just happened?
Five stages built from the ground up apply 25 lessons of material in sequence. Quality checks catch style and syntax issues in seconds. The build produces a single scanned artifact using OIDC — no stored credentials. Three parallel test jobs exercise unit behaviour, real database integration, and security patterns simultaneously. Staging deployment verifies the full stack before production is touched. Production requires a named approver, deploys zero-downtime, captures the pre-deployment task definition, runs smoke tests, rolls back automatically on failure, and emits a deployment marker to Datadog whether it succeeds or fails. Vaultex's developers can now merge code at 2pm on a Tuesday and have it in production by 2:15pm — reviewed, tested, audited, and automatically recovered if anything goes wrong.
What Vaultex Gets — A Day-in-the-Life After the Pipeline
An Ordinary Wednesday — Six Months After the Pipeline is Built
Final Warning: A Pipeline Is Never Finished
The Vaultex pipeline built in this lesson is a strong starting point — not a destination. As the team grows, the pipeline will need path filtering for more services, additional compliance controls, performance benchmarks, contract tests between emerging microservices, and a platform team to own the reusable workflows. Test suites will grow and require more aggressive parallelisation. New CVEs will appear in dependencies that Dependabot will need to manage automatically. Security requirements will tighten. The pipeline is not infrastructure you build once and forget — it is a system you continuously maintain, measure, and improve. The practices in this course give you the framework. What you do with that framework, and how consistently you apply it under the pressure of real engineering work, determines whether CI/CD becomes the foundation of a fast-moving engineering organisation or another well-intentioned system that nobody quite trusts.
Key Takeaways from This Lesson — and This Course
Teacher's Note
You have completed 40 lessons covering every layer of modern CI/CD. The next step is not more reading — it is building. Take the pipeline from this lesson, adapt it to your current project, and ship something to production through it this week. That first automated deployment is worth more than ten more lessons.
Practice Questions
Answer in your own words — then check against the expected answer.
1. What is the process — performed before writing any pipeline YAML — of mapping each business constraint, technical requirement, and compliance need to a specific pipeline decision, ensuring the resulting pipeline is designed for its context rather than assembled from generic templates?
2. What GitHub Actions step property allows the automated rollback logic to execute even when the smoke test step fails — by preventing the failure from immediately terminating the job, so that subsequent steps can check the outcome and respond accordingly?
3. In the Vaultex monorepo, why does the API pipeline's path filter include changes to the shared utilities package — even though that package is not the API service itself?
Lesson Quiz
1. In the Vaultex build stage, the Docker image is built with push: false, scanned with Trivy, and then pushed separately only if the scan passes. What security property does this sequence guarantee?
2. The Vaultex pipeline has three test jobs: unit tests (4 min), integration tests (7 min), and SAST (5 min). All three have needs: build but no dependency on each other. How long does the test stage take and why?
3. The Vaultex production deploy job targets a protected GitHub environment with required reviewers and self-review prevention. What specific compliance requirement from the scenario does this directly satisfy?
Course Complete
Dataplexa CI/CD Course
You have completed all 40 lessons — from the fundamentals of CI/CD through enterprise deployment patterns, security architecture, and real-world pipeline design. The knowledge is yours. Now go build something and ship it.
Return to Course Index