CI/CD Course
Test Types in Pipelines
In this lesson
Lesson 15 introduced the test pyramid — unit, integration, and end-to-end tests — as the structural foundation of an automated test suite. But a mature CI/CD pipeline contains more than these three types. Smoke tests, contract tests, performance tests, and security tests each serve a distinct purpose and belong at a specific point in the pipeline. Understanding what each type verifies, how expensive it is to run, and where it delivers the most value is what separates a pipeline that catches real problems from one that creates a false sense of security.
Smoke Tests — Fast Deployment Verification
A smoke test is a minimal set of checks run immediately after a deployment to verify that the application started correctly and its most critical paths are functional. The name comes from electronics — when you power on a new circuit board for the first time, you check whether it smokes before running any detailed diagnostics. In software, the equivalent is: did the application start? Is the health endpoint returning 200? Can a user log in?
Smoke tests are not comprehensive. They are designed to take under two minutes and answer a binary question: is this deployment fundamentally broken or not? They run after every deployment — to staging and to production. A failed smoke test on staging blocks promotion to production. A failed smoke test on production triggers an immediate automated rollback. Speed is more important than depth here; a smoke test that takes 15 minutes to run defeats its own purpose.
The Vital Signs Analogy
When a patient comes out of surgery, the first thing the recovery team checks is not whether their knee replacement is perfectly aligned — it is pulse, blood pressure, and breathing. These vital signs do not tell you everything, but they tell you immediately whether the patient is stable. Smoke tests are vital signs for a deployment. They do not verify every feature; they verify that the application is alive and breathing before the detailed checks begin.
Contract Tests — Keeping Services in Sync
Contract testing addresses one of the most common failure modes in microservices architectures: two services that were built and tested independently but break when they actually talk to each other because their API expectations have diverged. A contract test verifies that a service honours the interface that its consumers depend on — without requiring both services to be running at the same time.
The most widely used contract testing tool is Pact. The consumer service generates a "pact file" — a record of the requests it makes and the responses it expects. The provider service runs its own pipeline test to verify it can satisfy that pact. If the provider changes its API in a way that breaks the consumer's expectations, the provider's pipeline fails — before either service is deployed. This catches integration breakage at the point of change, not at runtime when both services are live in production.
Performance Tests — Catching Regressions Before Users Do
Performance tests measure how the application behaves under load — response times, throughput, error rates, and resource consumption under realistic or peak traffic conditions. They catch a category of problem that functional tests are blind to: a change that is logically correct but introduces a slow database query, an N+1 problem, or a memory leak that only manifests under concurrent load.
Performance tests are expensive to run — a meaningful load test takes minutes to hours and requires a realistic environment. For this reason they are rarely run on every PR. The most practical pattern is to run a lightweight performance benchmark on every merge to main — checking that key endpoint response times have not regressed beyond a defined threshold — and reserve full load tests for scheduled nightly runs or pre-release gates. Tools like k6, Locust, and Artillery integrate cleanly with GitHub Actions.
Security Tests — Shifting Left on Vulnerabilities
Security testing in a CI/CD pipeline takes several forms, each catching a different class of vulnerability. The goal in all cases is the same: find security issues at the point of change rather than in a quarterly penetration test or, worse, after a breach.
Security Test Types and Their Pipeline Role
npm audit, Snyk, or Dependabot.gitleaks can be added as a pre-commit hook or pipeline step.Placing Each Test Type in the Pipeline
The most important decision in test pipeline design is not which tests to write — it is when to run them. Running every test on every PR event produces a slow, expensive pipeline that developers avoid. Running tests too late in the process means failures surface after deployment rather than before. The right placement for each test type is determined by its speed, its dependencies, and the cost of the failure it catches.
Test Placement Reference — Pipeline Stage by Test Type
Multi-Stage Test Pipeline — GitHub Actions
jobs:
unit-and-sast: # Fast gate — runs on every PR push
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm test # Unit tests
- run: npx semgrep --config=auto . # SAST scan
integration: # Slower gate — also on every PR push
needs: unit-and-sast # Only runs if the fast gate passes
runs-on: ubuntu-latest
services:
postgres: # Spin up a real DB for integration tests
image: postgres:15
env:
POSTGRES_PASSWORD: testpass
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm run test:integration
smoke-test: # Runs after deployment to staging
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- run: |
curl --fail https://staging.myapp.com/health # Fail if health check returns non-200
curl --fail https://staging.myapp.com/api/pingWhat just happened?
Three test jobs are structured in sequence: fast unit and SAST checks run first, integration tests only run if those pass, and a smoke test runs after the staging deployment confirms the application is alive. Each stage acts as a gate — a failure stops the pipeline before spending time on the next, more expensive stage.
Warning: Running All Tests on Every PR Creates a Pipeline Nobody Uses
A pipeline that runs every test type — including full end-to-end suites, load tests, and DAST scans — on every pull request push will take 30–60 minutes per run. Developers stop waiting for it, start merging speculatively, and the pipeline becomes a formality rather than a quality gate. The discipline of test placement — fast checks on every push, slow checks only where they add unique value — is what keeps the pipeline fast enough to be genuinely useful. A 5-minute PR pipeline that catches 90% of problems is more valuable than a 45-minute pipeline that catches 95%.
Key Takeaways from This Lesson
Teacher's Note
If a team tells you their pipeline takes 40 minutes, ask them what runs on every PR — the answer almost always reveals expensive tests in the wrong place, not a slow codebase.
Practice Questions
Answer in your own words — then check against the expected answer.
1. What is the name of the minimal, fast check that runs immediately after every deployment to verify the application started correctly and its most critical paths are responding — designed to take under two minutes and trigger an automatic rollback on failure?
2. What is the acronym for the type of security test that analyses source code without running it — looking for insecure patterns like SQL injection vectors and hardcoded credentials — and runs on every pull request?
3. What testing approach — commonly implemented with a tool called Pact — verifies that a service provider's API satisfies the expectations of its consumers, catching integration breakage without requiring both services to be deployed simultaneously?
Lesson Quiz
1. A security test type cannot run on a PR because it requires a live, deployed instance of the application to send adversarial requests against. Which test type is this, and why does it need a running environment?
2. A team wants to catch performance regressions in their API response times but cannot afford to run a full load test on every pull request. What is the recommended placement pattern for performance tests in the pipeline?
3. A frontend service and a backend API are developed by separate teams. A contract testing tool like Pact is introduced. What does each side of the contract test actually verify?
Up Next · Lesson 17
Code Quality & Static Analysis
Tests verify behaviour. Static analysis verifies structure — catching complexity, style violations, and potential bugs without running a single line of code.