Jenkins Lesson 31 – Backup and Restore | Dataplexa

Section III · Lesson 31

Backup and Restore

Every Jenkins server will eventually break, get migrated, or need to be rebuilt from scratch. The teams that recover in 20 minutes are the ones with backups. The teams that spend two days rebuilding are the ones without them.

This lesson covers

What to back up → Manual backup with tar → The ThinBackup plugin → Restoring from backup → Testing your backup → Backup strategies for Docker and Kubernetes Jenkins

Jenkins stores everything in one place — JENKINS_HOME. Back up that directory and you can restore your entire Jenkins installation on a new server in minutes. That single fact makes Jenkins backup simpler than most people realise. The complexity is in making it automatic, reliable, and tested.

The Analogy

A Jenkins backup is like a photograph of your entire work desk — every document, every sticky note, every tool, every setting exactly as it was. When disaster strikes, you don't recreate everything from memory. You look at the photograph and put it all back exactly as it was. An untested backup is a photograph you took with the lens cap on. You don't find out until you need it most.

What Lives in JENKINS_HOME

Understanding what's in JENKINS_HOME helps you decide what to back up, what to exclude, and what's safe to skip. On a Linux native install, this is /var/lib/jenkins. In Docker, it's /var/jenkins_home.

Directory / File	Contents	Back up?
config.xml	Master Jenkins configuration — security settings, global config	✓ Critical
jobs/	All job configurations (config.xml per job) and build history	✓ Critical
credentials.xml	All stored credentials — encrypted at rest	✓ Critical
secrets/	Encryption keys for the credential store — without these, credentials.xml is unreadable	⚠ Critical + Protect
plugins/	All installed plugins as .jpi files	Optional
users/	Jenkins user accounts and API tokens	✓ Important
nodes/	Agent (node) configuration files	✓ Important
workspace/	Build workspaces — temporary files from running builds	Skip

The secrets/ directory — handle with care

The secrets/ folder contains the encryption key (hudson.util.Secret and master.key) that Jenkins uses to encrypt stored credentials. Back up credentials.xml without the secrets/ folder and the credentials are permanently unreadable — even to you. Back up secrets/ to a location with strict access control — it is as sensitive as the passwords it protects.

Manual Backup with tar

The scenario:

You're a DevOps engineer about to perform a major Jenkins upgrade — moving from Java 11 to Java 21 and upgrading Jenkins LTS by two versions. Before touching anything, you need a full backup you can restore from in under 5 minutes if something goes wrong.

Tools used:

tar — the standard Linux archive tool. Creates a compressed archive of a directory. -czf means: create, gzip-compress, write to file.
systemctl stop jenkins — stops the Jenkins service before backup. Backing up a live Jenkins while builds are running can produce inconsistent backups — partially written files may be in an inconsistent state.
rsync — a file sync tool used here to copy the backup to a remote server. The -az flags mean archive mode (preserves permissions and timestamps) and compress in transit.
sha256sum — generates a checksum of the backup file. Store this alongside the backup — use it to verify the file hasn't been corrupted before you restore from it.

# Step 1: Stop Jenkins to ensure a consistent backup
# If builds are running, wait for them to complete first (safe-restart then stop)
sudo systemctl stop jenkins

# Step 2: Create a timestamped backup archive
# Excludes workspace/ — these are temporary build files, not worth backing up
# The timestamp in the filename makes it easy to identify when the backup was taken
BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="/backup/jenkins-backup-${BACKUP_DATE}.tar.gz"

sudo tar \
  --exclude=/var/lib/jenkins/workspace \
  --exclude=/var/lib/jenkins/caches \
  -czf ${BACKUP_FILE} \
  /var/lib/jenkins/

echo "Backup created: ${BACKUP_FILE}"
echo "Size: $(du -sh ${BACKUP_FILE} | cut -f1)"

# Step 3: Generate a checksum so you can verify the backup later
sha256sum ${BACKUP_FILE} > ${BACKUP_FILE}.sha256
echo "Checksum saved: ${BACKUP_FILE}.sha256"

# Step 4: Copy the backup to a remote server or S3-compatible storage
# rsync preserves file attributes and compresses during transfer
rsync -az ${BACKUP_FILE} ${BACKUP_FILE}.sha256 \
  backup-server-01:/backups/jenkins/

# Step 5: Restart Jenkins
sudo systemctl start jenkins
sudo systemctl status jenkins

echo "Backup complete. Jenkins restarted."

Where to practice: Run this on your local Docker Jenkins instance. Replace /var/lib/jenkins with /var/jenkins_home inside the container. To back up the Docker volume: docker run --rm -v jenkins-data:/source -v $(pwd):/backup alpine tar -czf /backup/jenkins-backup.tar.gz /source. Store the backup on a separate machine — a backup on the same disk as Jenkins is not a backup.

Stopping jenkins (via systemctl)...
jenkins.service: Succeeded.

Backup created: /backup/jenkins-backup-20240314-091422.tar.gz
Size: 847M

Checksum saved: /backup/jenkins-backup-20240314-091422.tar.gz.sha256

sending incremental file list
jenkins-backup-20240314-091422.tar.gz
jenkins-backup-20240314-091422.tar.gz.sha256
sent 847.23M bytes  received 58 bytes  14.02M bytes/sec
total size is 847.07M  speedup is 1.00

● jenkins.service - Jenkins Continuous Integration Server
     Active: active (running) since Thu 2024-03-14 09:16:55 UTC; 2s ago

What just happened?

Jenkins was stopped before backup — this ensures the backup is consistent. A live Jenkins can have builds writing files mid-backup, producing a partially captured state. Stopping first guarantees a clean snapshot.
Workspace and caches excluded — these directories contain temporary build files that can be reconstructed by running a build. Excluding them reduces backup size significantly without losing anything important.
847M backup size — a typical Jenkins with moderate use. Most of that size is build logs in jobs/*/builds/. If this is too large, configure log rotation on your jobs (Lesson 10 Practice 3) to reduce the history kept per job.
Checksum file created — before restoring, run sha256sum -c jenkins-backup.tar.gz.sha256 to verify the file hasn't been corrupted in storage or transit. A corrupt backup is the same as no backup.
Jenkins restarted and running in 2 seconds — the total downtime for this backup was about 5 minutes. For teams that can't afford even that, the ThinBackup plugin can back up while Jenkins is running (with some caveats).

Automated Backups with ThinBackup

Manual backups require someone to remember to run them. ThinBackup is a Jenkins plugin that automates scheduled backups without stopping Jenkins. It's not as complete as a tar backup of the full JENKINS_HOME, but it covers all the critical configuration files automatically.

Install it from Manage Jenkins → Plugin Manager → Available → ThinBackup. Then configure it at Manage Jenkins → ThinBackup:

Jenkins — Manage Jenkins → ThinBackup → Settings

Backup directory

/mnt/jenkins-backups

Full backup schedule

H 2 * * 0

Every Sunday at ~2 AM

Differential backup

H 2 * * 1-6

Mon–Sat at ~2 AM — only changed files

Options

Back up build results Back up plugins archives Move old backups to ZIP Wait for idle Jenkins (no builds running)

Max stored full backups

4 keeps 4 weeks of Sunday backups

Restoring From Backup

When disaster strikes — a failed upgrade, a corrupted config, a lost server — restoration is straightforward if your backup is complete and tested. Here's the procedure for a full restore from a tar backup:

# Step 1: Verify the backup file is intact before touching anything
sha256sum -c jenkins-backup-20240314-091422.tar.gz.sha256
# Expected output: jenkins-backup-20240314-091422.tar.gz: OK
# If it says FAILED — the backup is corrupted. Do NOT restore from it.

# Step 2: Stop Jenkins on the new or existing server
sudo systemctl stop jenkins

# Step 3: Clear the current JENKINS_HOME (if restoring over an existing install)
# WARNING: This is destructive — make sure you want to replace the current state
sudo rm -rf /var/lib/jenkins/

# Step 4: Extract the backup archive into place
sudo tar -xzf jenkins-backup-20240314-091422.tar.gz \
  -C /                        # extract relative to filesystem root
                               # the archive contains the full path /var/lib/jenkins/...

# Step 5: Restore correct ownership — Jenkins runs as the 'jenkins' user
sudo chown -R jenkins:jenkins /var/lib/jenkins/

# Step 6: Start Jenkins
sudo systemctl start jenkins

# Step 7: Verify Jenkins started cleanly
sudo systemctl status jenkins
sudo tail -50 /var/log/jenkins/jenkins.log

jenkins-backup-20240314-091422.tar.gz: OK

Stopping jenkins.service...

/var/lib/jenkins/config.xml
/var/lib/jenkins/credentials.xml
/var/lib/jenkins/secrets/master.key
/var/lib/jenkins/jobs/payment-service-build/config.xml
/var/lib/jenkins/jobs/frontend-test/config.xml
/var/lib/jenkins/users/admin/config.xml
...
Restore complete — 14,823 files extracted.

● jenkins.service - Jenkins Continuous Integration Server
     Active: active (running) since Thu 2024-03-14 10:02:11 UTC; 3s ago

Jenkins initial setup is NOT required — existing configuration loaded.
INFO: Jenkins is fully up and running

What just happened?

Checksum verified first — the backup file was confirmed intact before anything was touched. This step takes 5 seconds and has saved many engineers from restoring a corrupt backup onto a broken system and ending up with two broken systems.
14,823 files extracted — every job config, every credential, every user account, every plugin config, and the encryption keys — all restored in one command.
Jenkins initial setup is NOT required — because the backup included the full configuration, Jenkins recognised it was already configured and skipped the setup wizard entirely. Jobs, credentials, and users are all immediately available.
Secrets directory restored with keys — because secrets/master.key and the surrounding files were in the backup, credentials are decryptable. All stored passwords and tokens are available immediately after restore.
Ownership restored with chown — tar extracts files with the UID/GID from the backup. If the jenkins user has a different UID on the new server, files may be unreadable. The chown -R jenkins:jenkins step ensures Jenkins has access regardless.

The Most Ignored Step — Testing Your Backup

A backup you've never tested is a backup you don't actually have. Backup testing should be a scheduled activity — not something you do for the first time during an incident at 2 AM.

Monthly backup test checklist

Restore the backup to a staging Jenkins server
Verify the UI loads and shows all jobs
Run one critical pipeline end to end
Confirm credentials are decryptable (test a Docker login or SSH)
Check the user list and permissions are intact
Verify the checksum matches before and after transfer

Signs your backup is incomplete

Jenkins shows the setup wizard after restore
Credentials throw decryption errors
Jobs appear but have no build history
Users exist but API tokens don't work
Plugins appear installed but won't activate

Teacher's Note

The backup is not done until you've successfully restored from it once. Everything before that is just hope.

Practice Questions

1. Which subdirectory of JENKINS_HOME contains the encryption keys needed to decrypt stored credentials — and must always be backed up alongside credentials.xml?

2. Which JENKINS_HOME subdirectory contains only temporary build files and should be excluded from backups to reduce size?

3. Which command generates a checksum of the backup file so you can verify it hasn't been corrupted before restoring?

Quiz

Up Next · Lesson 32

Logging and Monitoring

Jenkins system logs, build log management, integrating with Prometheus and Grafana, and the metrics that actually tell you whether your Jenkins is healthy.

← Previous Course Index Next →

Jenkins Course