Linux Administration
System Services and systemd
In this lesson
systemd is the init system and service manager used by virtually every major Linux distribution. It is the first process launched by the kernel at boot (PID 1), and it is responsible for starting, stopping, and supervising every service and daemon on the system. Understanding systemd means understanding how Linux itself comes to life — and how to control every running service with precision and confidence.
How systemd Works — Units and the Dependency Graph
systemd organises everything it manages into units — discrete, declarative configuration files that describe a service, a socket, a mount point, a timer, or a boot target. When the system starts, systemd reads all unit files, builds a dependency graph, and starts units in parallel wherever it can, making it significantly faster than older sequential init systems.
Fig 1 — systemd as PID 1 activating a target which starts units in parallel
A background process — daemon. The most common unit type. Covers nginx, sshd, cron, docker, and almost every service you will manage.
A synchronisation point that groups units together. Replaces the concept of SysV runlevels. Examples: multi-user.target, graphical.target.
A scheduled activation unit — the systemd replacement for cron. Activates a paired .service unit on a calendar or monotonic schedule.
Manages a filesystem mount point. Allows other units to declare dependencies on a mount being active before they start.
Activates a service on-demand when a connection arrives on a socket. The service only starts when actually needed — saving resources at idle.
systemctl — Controlling Services Day to Day
systemctl is the command-line interface to systemd. It is the single tool you use to start, stop, restart, enable, disable, and inspect every service on the system. Mastering its core subcommands covers the vast majority of real-world service management tasks.
# The most complete view of a service — always start here when troubleshooting
sudo systemctl status nginx
# Enable nginx at boot and start it right now (single command)
sudo systemctl enable --now nginx
# Reload config without dropping connections (nginx supports this)
sudo nginx -t && sudo systemctl reload nginx
# Restart a service (drops connections — use only when reload is not enough)
sudo systemctl restart nginx
# Check whether a service will start at boot
systemctl is-enabled nginx
# List all failed units — your first stop after a reboot with problems
systemctl list-units --state=failed
# List all active service units
systemctl list-units --type=service --state=activeWhat just happened? systemctl status showed the full picture in one view: the unit file location, whether it is enabled at boot (enabled), the current runtime state (active (running)), the main PID and process tree, memory and CPU usage, and the most recent log lines — all without needing to open a separate log file.
Analogy: enable and start are completely separate concerns. enable sets the alarm clock — it determines whether the service wakes up when the system boots. start wakes it up right now regardless of the alarm. You need both: a service that is started but not enabled will disappear after the next reboot.
Understanding and Writing Unit Files
Every service systemd manages is described by a unit file — a plain text INI-style file with three sections. Distribution-provided unit files live in /lib/systemd/system/. Administrator overrides and custom services go in /etc/systemd/system/, which takes precedence. Never edit files in /lib/systemd/system/ directly — they will be overwritten by package upgrades.
# /etc/systemd/system/myapp.service
# A minimal but complete custom service unit file
[Unit]
Description=My Application Server
Documentation=https://myapp.example.com/docs
# Start only after the network is fully up
After=network-online.target
Wants=network-online.target
[Service]
# Type=simple means the ExecStart process IS the main service process
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
# The command that starts the service
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yml
# Automatically restart if the process exits unexpectedly
Restart=on-failure
RestartSec=5s
# Security hardening — limit what the service can access
NoNewPrivileges=true
PrivateTmp=true
# Output goes to the journal
StandardOutput=journal
StandardError=journal
[Install]
# This unit is "wanted by" multi-user.target — i.e. starts at normal boot
WantedBy=multi-user.target# After creating or modifying a unit file, always reload the systemd daemon
sudo systemctl daemon-reload
# Then enable and start the new service
sudo systemctl enable --now myapp
# View the unit file systemd is actually using for a service
systemctl cat nginx
# Edit a unit file safely — creates a drop-in override, never touches the original
sudo systemctl edit nginx
# View all drop-in overrides for a unit
systemctl cat nginx
ls /etc/systemd/system/nginx.service.d/
# Verify a unit file has no syntax errors
systemd-analyze verify /etc/systemd/system/myapp.serviceWhat just happened? systemctl cat printed the full unit file as systemd sees it, including any drop-in overrides stacked on top. This is the correct way to inspect a unit — reading the raw file in /lib/systemd/system/ directly may miss overrides that change the service's behaviour.
journalctl — Reading the systemd Journal
systemd captures all service output in a centralised structured log called the journal. journalctl is the tool for querying it. Because the journal is structured and indexed, it is far faster and more flexible to query than grepping through plain text log files.
The most frequently used flag. Shows only journal entries from the named service: journalctl -u nginx
Like tail -f but for the journal. Combine with -u to watch a specific service live: journalctl -fu nginx
Accepts human-readable time: --since "1 hour ago", --since "2025-03-12 09:00", --since today
Levels: emerg, alert, crit, err, warning, notice, info, debug. Shows specified level and above.
-b 0 = current boot, -b -1 = previous boot. Essential for diagnosing crash and reboot scenarios.
# View logs for nginx — most recent entries first
journalctl -u nginx -r
# Follow nginx logs in real time (like tail -f)
journalctl -fu nginx
# Show nginx logs from the last 30 minutes
journalctl -u nginx --since "30 minutes ago"
# Show only errors and above across the entire system
journalctl -p err
# Show all logs from the current boot — useful after an unexpected reboot
journalctl -b 0
# Show logs from the previous boot — diagnose a crash
journalctl -b -1
# Show logs between two specific times
journalctl -u nginx --since "2025-03-12 08:00" --until "2025-03-12 10:00"
# Check how much disk space the journal is using
journalctl --disk-usageWhat just happened? journalctl -p err -b 0 surfaced two important errors from the current boot — a filesystem error on sda1 and a missing SSH host key. Both would have been invisible in a standard systemctl status view. Filtering by priority level is one of the fastest ways to triage a system after an incident.
Targets — systemd's Replacement for Runlevels
Older Linux systems used SysV runlevels (numbers 0–6) to define system states. systemd replaces these with targets — named units that group other units together and define what the system should be doing. Targets are more descriptive and composable than runlevels.
SysV Runlevels (legacy)
0
Halt
1
Single-user mode
3
Multi-user, no GUI
5
Multi-user with GUI
6
Reboot
systemd Targets (modern)
poweroff.target
Halt the system
rescue.target
Single-user recovery
multi-user.target
Multi-user, no GUI
graphical.target
Multi-user with GUI
reboot.target
Reboot the system
# Check the current default boot target
systemctl get-default
# Change the default boot target (e.g. remove GUI on a server)
sudo systemctl set-default multi-user.target
# Switch to a target immediately without rebooting
sudo systemctl isolate rescue.target
# View all available targets
systemctl list-units --type=target
# Power management via systemctl
sudo systemctl poweroff
sudo systemctl reboot
sudo systemctl suspendWhat just happened? set-default works by creating a symlink at /etc/systemd/system/default.target pointing to the chosen target. This file in /etc/systemd/system/ takes precedence over the distribution default — it is the admin override mechanism at work.
Troubleshooting Failed Services — A Systematic Approach
When a service fails to start, a consistent sequence of checks resolves the vast majority of problems within minutes. Following this order prevents the common trap of changing multiple things at once and losing track of what fixed the problem.
Step 1 — Check the service status
The status output shows the last few log lines and the exit code. Often this is enough to identify the problem immediately.
sudo systemctl status myapp.service
Step 2 — Read the full journal for this service
The status view is truncated. The full journal shows the complete error output including stack traces and config parse errors.
journalctl -u myapp.service --since "5 minutes ago"
Step 3 — Verify the unit file and run the binary manually
Check the unit file for typos with systemctl cat, then try running the ExecStart command directly in a terminal to see raw output.
systemctl cat myapp.service
sudo -u myapp /opt/myapp/bin/server --config /etc/myapp/config.yml
Step 4 — Fix, reload daemon, restart, confirm
After making a fix, always reload the daemon before restarting the service, then confirm it reached the running state.
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
sudo systemctl status myapp.service
Always Run systemctl daemon-reload After Editing a Unit File
systemd caches unit file contents in memory. If you edit a unit file in /etc/systemd/system/ and then run systemctl restart without first running systemctl daemon-reload, systemd will restart the service using the old cached unit definition — your changes will have no effect. This is the source of much confusion when debugging custom services.
Lesson Checklist
start/stop (runtime) and enable/disable (boot behaviour), and I use enable --now to do both at once
[Unit], [Service], and [Install] sections and always run daemon-reload after any change
journalctl using unit filters, time ranges, and priority levels to find relevant log entries quickly
Teacher's Note
The single most underused journalctl invocation is journalctl -p err -b 0 — errors from the current boot across the whole system. Run this immediately after any unexpected reboot or service outage and it will almost always surface the root cause within the first few lines.
Practice Questions
1. You deploy a custom application and create a unit file at /etc/systemd/system/myapp.service. Write every command you would run, in order, to: load the unit, start the service, confirm it is running, and ensure it survives a reboot.
2. A service called webapp.service shows as failed in systemctl list-units. Describe your step-by-step troubleshooting approach, naming the specific commands you would run and what information you expect to find at each step.
3. Explain the difference between systemctl reload nginx and systemctl restart nginx. In a production environment serving live traffic, which would you prefer after editing nginx.conf, and what command would you run first to make it safe?
Lesson Quiz
1. You run sudo systemctl enable nginx. What is the immediate effect on the running system?
2. You edit /etc/systemd/system/myapp.service to change the Restart= value, then immediately run sudo systemctl restart myapp. Why might your change have no effect?
3. Which journalctl command would you run to see only critical errors and above from the previous boot session?
Up Next
Lesson 15 — Process Management
Viewing, controlling, and prioritising running processes with ps, top, kill, and nice