Overview
systemd-journald centralizes logging from every unit, the kernel, and early boot into one structured, queryable journal, replacing the older model of scattered plain-text log files. journalctl queries it, -u scopes output to a specific unit (including systemd's own messages about it and any coredump metadata, not just what the service printed itself), -p filters by severity, and -f follows new entries live. What actually determines whether any of this survives a reboot is journald's Storage= setting, and the default, auto, is easy to misjudge: it behaves like persistent only if /var/log/journal already exists, and like volatile (memory-only, wiped on reboot) otherwise, so two systems that never explicitly configured this setting can behave completely differently depending on directory history alone.
Quick Reference
Storage= value | Location | Survives reboot? |
|---|---|---|
volatile | /run/log/journal (memory-backed) | No |
persistent | /var/log/journal (disk) | Yes |
auto (default) | Either, depending on whether /var/log/journal exists | Depends on directory history |
journalctl option | Does |
|---|---|
-u <unit> | Scope to one systemd unit, plus systemd's own messages about it |
-p <level> | Filter to that severity or more severe |
-f | Follow the journal live, like tail -f |
Syntax
journalctl -u nginx.service --since "1 hour ago"
journalctl -u nginx.service -p err -fExamples
# Make persistence an explicit choice instead of leaving it to
# whether /var/log/journal happens to already exist.
mkdir -p /var/log/journal
systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journaldAuto storage mode can silently discard every log on reboot
Storage=auto, journald's default, behaves like persistent only if /var/log/journal already exists; otherwise every log entry lives only in memory and is gone on the next reboot. A freshly provisioned system with no one having created that directory is quietly running volatile, even though nobody set it explicitly.
Visual Diagram
Common Mistakes
- Assuming logs are persisted by default and discovering only after an incident-driven reboot that
Storage=autowas actually running volatile. - Grepping a combined log file for a service name instead of using
journalctl -u, missing systemd's own start/stop/failure messages and coredump metadata. - Running
journalctl -funfiltered during an incident and having to visually scan every unit's routine output instead of scoping to the relevant unit and severity. - Forgetting that creating
/var/log/journalmid-session doesn't retroactively persist journal entries already written only to the volatile, memory-backed journal.
Performance
- Filtering with
-uand-pat query time is efficient since the journal is indexed by fields including unit and priority, not a linear text scan of a flat file. - Persistent, disk-backed journals grow over time;
SystemMaxUse=/RuntimeMaxUse=injournald.confbound that growth, unbounded journals can otherwise consume meaningful disk space on a long-running system.
Best Practices
- Explicitly set
Storage=persistent(or create/var/log/journal) rather than leaving persistence to depend on whether that directory happens to already exist. - Use
journalctl -u <unit>over grepping a shared log file, it's scoped by the journal's own unit metadata, not string matching. - Combine
-u,-p, and-ftogether when live-debugging a specific service's failures, rather than following the full, unfiltered journal. - Set explicit size/retention limits (
SystemMaxUse=,MaxRetentionSec=) so a persistent journal doesn't grow unbounded on disk.