Linux Log Files and journalctl:

Logs Tell You Everything

When something breaks at 2 AM, logs are your first and best source of truth. Before you start guessing, before you restart anything, before you panic—check the logs.

Every process, every service, every kernel event gets logged somewhere. Knowing where to look and how to read what you find is the difference between solving problems in minutes versus hours.

Where Logs Live

Traditional Log Locations

Log File	Contains	Look Here When
/var/log/syslog	General system messages	Generic troubleshooting
/var/log/auth.log	Authentication events	Login issues, sudo usage, SSH
/var/log/kern.log	Kernel messages	Hardware issues, drivers
/var/log/dmesg	Boot messages	Hardware detection, early boot
/var/log/apt/	Package manager logs	What was installed/updated
/var/log/nginx/	Web server logs	Web traffic, errors
/var/log/mysql/	Database logs	Query issues, connections

Reading Traditional Logs

# View last 50 lines
tail -50 /var/log/syslog

# Follow log in real-time
tail -f /var/log/syslog

# Search for specific term
grep "error" /var/log/syslog

# Case-insensitive search
grep -i "fail" /var/log/auth.log

# Show context around matches
grep -C 3 "error" /var/log/syslog

# Multiple files
grep "error" /var/log/*.log

journalctl: The Modern Way

systemd’s journal centralises logs from all services. One tool to query everything.

Basic Usage

# All logs (paginated)
journalctl

# Last 50 entries
journalctl -n 50

# Follow in real-time
journalctl -f

# Since last boot
journalctl -b

# Previous boot
journalctl -b -1

Filtering by Service

# Specific service
journalctl -u nginx

# Multiple services
journalctl -u nginx -u php-fpm

# Service with recent entries
journalctl -u nginx -n 100

# Follow service logs
journalctl -u nginx -f

Filtering by Time

# Last hour
journalctl --since "1 hour ago"

# Since specific time
journalctl --since "2024-01-15 09:00:00"

# Time range
journalctl --since "2024-01-15 09:00" --until "2024-01-15 12:00"

# Today only
journalctl --since today

# Yesterday
journalctl --since yesterday --until today

Filtering by Priority

# Errors and worse only
journalctl -p err

# Warnings and worse
journalctl -p warning

# Emergency to error
journalctl -p emerg..err

Priority levels (0-7): emerg, alert, crit, err, warning, notice, info, debug

Output Formats

# Short (default)
journalctl -u nginx

# Verbose (all fields)
journalctl -u nginx -o verbose

# JSON (for parsing)
journalctl -u nginx -o json

# One line per entry
journalctl -u nginx -o short-precise

Common Troubleshooting Patterns

Service Won’t Start

# Check service status (shows recent logs)
systemctl status nginx

# Get more log context
journalctl -u nginx -n 50

# Follow while restarting
journalctl -u nginx -f &
sudo systemctl restart nginx

What Happened at a Specific Time?

# Around the incident time
journalctl --since "2024-01-15 14:30" --until "2024-01-15 14:45"

# Add priority filter
journalctl --since "2024-01-15 14:30" -p err

# Specific service at that time
journalctl -u myapp --since "2024-01-15 14:30"

Authentication Issues

# SSH login failures
journalctl -u ssh | grep -i fail

# Traditional log
grep "Failed password" /var/log/auth.log

# All sudo activity
journalctl | grep sudo

Disk Space Issues

# Journal space usage
journalctl --disk-usage

# Clean old journals
sudo journalctl --vacuum-time=7d

# Limit journal size permanently
# Edit /etc/systemd/journald.conf
# SystemMaxUse=500M

Application-Specific Logs

nginx

# Access log (who's visiting)
tail -f /var/log/nginx/access.log

# Error log (what's broken)
tail -f /var/log/nginx/error.log

# Common patterns
grep " 500 " /var/log/nginx/access.log  # Server errors
grep " 404 " /var/log/nginx/access.log  # Not found

Docker

# Container logs
docker logs container_name

# Follow container logs
docker logs -f container_name

# Last 100 lines
docker logs --tail 100 container_name

# With timestamps
docker logs -t container_name

PostgreSQL

# Main log location varies
tail -f /var/log/postgresql/postgresql-*-main.log

# Through journalctl
journalctl -u postgresql

Log Analysis Techniques

Counting Occurrences

# Count error types
grep "error" /var/log/syslog | wc -l

# Count by IP in access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Count HTTP status codes
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

Finding Patterns

# Errors in the last hour
journalctl --since "1 hour ago" -p err

# What services are erroring
journalctl -p err --since today | grep -oP '(?<=]: ).*?(?=:)' | sort | uniq -c | sort -rn

# Failed login attempts by IP
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

Timeline Correlation

# What happened just before the crash?
journalctl --until "2024-01-15 14:30:00" | tail -100

# All logs around incident
journalctl --since "2024-01-15 14:29" --until "2024-01-15 14:31"

Log Rotation

Logs rotate to prevent disk filling. Understand the rotation:

# View rotation config
cat /etc/logrotate.d/nginx

# Manual rotation
sudo logrotate -f /etc/logrotate.d/nginx

# Check rotation status
cat /var/lib/logrotate/status

Rotated logs are usually compressed: access.log.1, access.log.2.gz, etc.

# Read compressed logs
zcat /var/log/nginx/access.log.2.gz | tail

# Or
zless /var/log/nginx/access.log.2.gz

Interview Questions

Quick Reference

# Traditional logs
tail -f /var/log/syslog           # Follow system log
tail -f /var/log/auth.log         # Follow auth log
grep "error" /var/log/syslog      # Search logs

# journalctl basics
journalctl -n 50                   # Last 50 entries
journalctl -f                      # Follow all logs
journalctl -b                      # Since boot

# Service logs
journalctl -u nginx                # Specific service
journalctl -u nginx -f             # Follow service

# Time filtering
journalctl --since "1 hour ago"    # Recent
journalctl --since today           # Today
journalctl --since "2024-01-15"    # Specific date

# Priority filtering
journalctl -p err                  # Errors only
journalctl -p warning              # Warnings and worse

# Maintenance
journalctl --disk-usage            # Check space
sudo journalctl --vacuum-time=7d   # Clean old logs

The Career Translation

Next Steps

• Centralised logging - ELK stack, Loki, Graylog
• Log parsing - Advanced awk, jq for JSON logs
• Alerting - Trigger alerts on log patterns
• Compliance - Log retention requirements

Logs are your post-mortem and your early warning system. Master reading them and you'll solve problems faster than anyone who skips this step.

---

Part 9 of the Linux Fundamentals series. Next: SSH essentials for remote administration—connecting to and managing remote servers securely.

---