CI/CD Course
CI/CD with Kubernetes
In this lesson
Kubernetes is the dominant container orchestration platform — the system that runs containers at scale, manages their lifecycle, handles traffic routing, restarts failed instances, and rolls out new versions without downtime. For CI/CD, Kubernetes is the deployment target: the pipeline builds a Docker image, pushes it to a registry, and then instructs Kubernetes to run the new image across the cluster. How that instruction is given — through direct kubectl commands, through Helm chart releases, or through a GitOps operator like Argo CD — determines the deployment model's auditability, rollback capability, and operational complexity.
Three Deployment Models for Kubernetes
There is no single correct way to deploy to Kubernetes from a CI/CD pipeline. Three models are in common use, each with a different trade-off between simplicity, flexibility, and operational maturity.
Kubernetes Deployment Models — Comparison
kubectl apply -f manifests/ to apply YAML manifest files directly to the cluster. Simple, no additional tooling, works well for small teams with few services. Becomes unwieldy as the number of environments and services grows — manifests must be templated or duplicated per environment.helm upgrade --install with environment-specific values files. Supports rollback with helm rollback, tracks release history, and handles complex applications with many interdependent resources. The standard for teams managing multiple environments.The Thermostat Analogy
A push-based deployment is like manually turning the heating dial every time the temperature drops — you detect the gap and intervene. GitOps is like a smart thermostat: you set the desired temperature once, and the thermostat continuously monitors the actual temperature, automatically turning the heating on or off to maintain the target. The desired state is declared; the system enforces it. If someone manually opens a window and the temperature drops, the thermostat corrects it without being asked. If someone manually changes a Kubernetes resource outside of Git, the GitOps operator reverts it to match the declared state.
Direct kubectl Deployment — The Simplest Starting Point
For teams just getting started with Kubernetes deployments, kubectl apply from the pipeline is the most straightforward approach. The pipeline authenticates to the cluster using a kubeconfig stored as a GitHub Secret or via OIDC, then applies manifests or patches the image tag on an existing deployment.
Direct kubectl Deployment — GitHub Actions
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
permissions:
contents: read
id-token: write # For OIDC auth to cloud provider
steps:
- uses: actions/checkout@v4
- name: Authenticate to GKE via OIDC
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Get cluster credentials
uses: google-github-actions/get-gke-credentials@v2
with:
cluster_name: prod-cluster
location: europe-west1
- name: Update image tag in deployment
run: |
kubectl set image deployment/api \
api=ghcr.io/myorg/api:${{ github.sha }} \
--namespace production
- name: Wait for rollout to complete
run: |
kubectl rollout status deployment/api \
--namespace production \
--timeout=300s # Fail pipeline if rollout takes more than 5 minutes
What just happened?
The pipeline authenticated to GCP via OIDC — no stored credentials — fetched cluster credentials, updated the deployment's image tag to the current commit SHA, then waited for Kubernetes to complete the rolling update. If the rollout does not complete within 5 minutes — because new pods are crashing or failing health checks — the pipeline fails and the on-call engineer is alerted. The deployment is not declared successful until Kubernetes confirms it.
Helm — Templated Releases with Rollback
Helm is the Kubernetes package manager. A Helm chart defines all the Kubernetes resources an application needs — Deployment, Service, Ingress, ConfigMap, HorizontalPodAutoscaler — as templated YAML files. Values are injected at install or upgrade time from a values file or command-line overrides. The same chart deploys to staging with values-staging.yaml and to production with values-production.yaml, with the image tag passed as a parameter.
Helm tracks every release as a versioned history entry in the cluster. helm rollback my-app 3 rolls back to release revision 3 in seconds — no pipeline run required, no Git revert needed. For teams managing multiple services and environments, Helm's release management, values hierarchy, and rollback capability make it significantly more maintainable than raw kubectl apply at scale.
GitOps with Argo CD — The Operator Model
GitOps inverts the deployment model. Instead of the pipeline pushing changes to the cluster, an operator inside the cluster pulls changes from a Git repository. Argo CD watches a Git repository containing Kubernetes manifests or Helm charts. When the repository changes — because the CI pipeline updated the image tag in a values file — Argo CD detects the diff between the declared state in Git and the actual state in the cluster, and automatically applies the change.
The pipeline's role in a GitOps workflow changes significantly. Instead of authenticating to the cluster and running kubectl or helm commands directly, the pipeline updates the image tag in the Git repository and commits the change. Argo CD handles the rest. This means the pipeline never needs cluster credentials — a major security improvement. Every cluster change is traceable to a Git commit. Drift between the declared and actual state is detected and corrected automatically. Manual changes to the cluster are reverted unless they are first committed to Git.
GitOps Pipeline — Update Image Tag in Git, Let Argo CD Deploy
jobs:
update-gitops-repo:
runs-on: ubuntu-latest
needs: build-and-push # Runs after the Docker image is built and pushed
steps:
- name: Check out the GitOps config repository
uses: actions/checkout@v4
with:
repository: myorg/gitops-config # Separate repo holding cluster state
token: ${{ secrets.GITOPS_PAT }} # PAT with write access to config repo
- name: Update image tag in values file
run: |
# Replace the image tag in the production values file
sed -i "s|tag:.*|tag: ${{ github.sha }}|" \
apps/api/values-production.yaml
- name: Commit and push the updated tag
run: |
git config user.name "CI Bot"
git config user.email "ci@myorg.com"
git add apps/api/values-production.yaml
git commit -m "deploy: api ${{ github.sha }} to production"
git push
# Argo CD detects this commit and applies the change to the cluster
# No cluster credentials needed in this pipeline at all
What just happened?
The pipeline updated one line in a values file in the GitOps config repository and committed it. Argo CD, running inside the cluster, detects the diff within seconds and applies the new image tag to the production deployment. The pipeline never touched the cluster directly — no kubeconfig, no kubectl, no Helm. The entire deployment history is the Git log of the config repository, and every production change is a reviewed, committed, auditable Git commit.
Warning: Not Waiting for Rollout Completion Hides Deployment Failures
A pipeline step that runs kubectl set image and immediately reports success has only told Kubernetes to start the rollout — it has not confirmed that the rollout succeeded. Kubernetes may be silently failing to schedule new pods, crashing on startup, or failing readiness probes while the pipeline reports a green build and the team believes the deployment is complete. Always follow a deployment command with kubectl rollout status --timeout or its equivalent. A pipeline that waits for Kubernetes confirmation is one where a green build actually means something. A pipeline that does not wait is one where production failures surface through monitoring alerts rather than pipeline failures — by which point the broken version is already serving traffic.
Key Takeaways from This Lesson
kubectl rollout status --timeout turns a deployment command into a deployment confirmation. Without it, a green pipeline does not mean a healthy deployment.
helm rollback reverts to any previous release revision in seconds, without requiring a pipeline run or a Git revert, making it the preferred deployment tool for complex multi-resource applications.
Teacher's Note
If you are choosing between Helm and GitOps, start with Helm — it is simpler to adopt and gives you rollback immediately. Add Argo CD when you find yourself spending significant time debugging "what is actually running in production right now" — that is the problem GitOps is designed to eliminate.
Practice Questions
Answer in your own words — then check against the expected answer.
1. What is the name of the GitOps operator that runs inside a Kubernetes cluster, watches a Git repository containing the declared cluster state, and automatically applies changes when the repository is updated — eliminating the need for the CI pipeline to hold cluster credentials?
2. What kubectl command — combined with a --timeout flag — waits for a Kubernetes Deployment rollout to complete and fails the pipeline if the rollout does not succeed within the specified time window?
3. What Helm command reverts a release to a specific previous revision — enabling instant rollback to any prior deployment state without requiring a pipeline run or a Git revert?
Lesson Quiz
1. A security team requires that the CI pipeline must never hold Kubernetes cluster credentials. Which deployment model satisfies this requirement, and how?
2. A pipeline runs kubectl set image deployment/api api=ghcr.io/myorg/api:abc123 and reports success. Five minutes later, monitoring alerts fire — the new pods are crashing on startup due to a missing environment variable. The team is confused because the pipeline was green. What went wrong?
3. A team manages 8 microservices each deployed to 3 environments. They currently maintain separate YAML manifest directories per service per environment — 24 directories — and update image tags manually in each. Which deployment tool would most reduce this maintenance burden and why?
Up Next · Lesson 30
Monitoring CI/CD Pipelines
A pipeline that cannot be measured cannot be improved. Lesson 30 closes Section III by covering how to instrument pipelines for observability — tracking DORA metrics, pipeline health, and deployment outcomes.