Log Management Visualizer - House of Script

← Back to Catalog

Log File Explorer

Explore common Linux and Windows log files to understand their structure and content.

Log Files

/var/log/syslog

/var/log/auth.log

/var/log/kern.log

/var/log/apache2/access.log

/var/log/apache2/error.log

journalctl

Windows: System

Windows: Security

Log Entry Anatomy

Understanding the components of a log entry is crucial for effective log analysis.

Sample Log Entry

Dec 11 14:23:45 webserver01 sshd[12345]: Accepted publickey for admin from 192.168.1.100 port 52234 ssh2

Timestamp

Dec 11 14:23:45

When the event occurred. Format varies (syslog, ISO 8601, Unix timestamp).

Hostname

webserver01

The system that generated the log entry. Essential for distributed systems.

Process Name

sshd

The program or service that generated the log. Helps categorize events.

Process ID (PID)

12345

Unique identifier for the process instance. Useful for tracking specific sessions.

Message

Accepted publickey for admin from 192.168.1.100 port 52234 ssh2

The actual event description. Contains the details of what happened.

Severity Level

Informational (6)

Indicates the importance of the event (0=Emergency to 7=Debug).

Logrotate Simulator

Configure and visualize how logrotate manages log files to prevent disk space exhaustion.

Rotation Settings

Rotation Interval:

Keep Last N Files:

Max Size Before Rotation (MB):

Compress rotated logs

Delay compression (delaycompress)

Use date extension (dateext)

Rotation Visualization

application.log (Current - 45 MB)

application.log.1 (72 MB)

application.log.2.gz (18 MB compressed)

application.log.3.gz (19 MB compressed)

application.log.4.gz (21 MB compressed)

Log Parsing Tools

Master essential command-line tools for log analysis and extraction.

Common grep Patterns

$ grep "error" /var/log/syslog

Search for lines containing "error"

$ grep -i "failed" /var/log/auth.log

Case-insensitive search for "failed"

$ grep -E "error|warning|critical" /var/log/syslog

Search for multiple patterns using extended regex

$ grep -A 3 -B 3 "kernel panic" /var/log/kern.log

Show 3 lines before and after matches (context)

awk for Field Extraction

$ awk '{print $1, $2, $3}' /var/log/syslog

Extract first three fields (timestamp)

$ awk '/error/ {print $0}' /var/log/syslog

Print lines matching "error"

$ awk '{count[$5]++} END {for (proc in count) print proc, count[proc]}' /var/log/syslog

Count occurrences by process name (field 5)

Common One-Liners

$ tail -f /var/log/syslog | grep --line-buffered "error"

Real-time monitoring for errors

$ awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head -10

Top 10 IP addresses in Apache access log

$ grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn

Count failed login attempts by IP

Interactive Parser

Enter a grep/awk pattern:

Results will appear here...

Log Severity Levels (RFC 5424)

Understanding severity levels helps prioritize log analysis and alerting.

Emergency

System is unusable. Immediate action required.

kernel: Kernel panic - not syncing

Alert

Action must be taken immediately.

database: Disk failure imminent

Critical

Critical conditions requiring attention.

apache: Cannot allocate memory

Error

Error conditions that need resolution.

sshd: Failed password for user

Warning

Warning conditions that may require attention.

disk: 90% capacity reached

Notice

Normal but significant conditions.

systemd: Service restarted

Informational

Informational messages about normal operations.

sshd: User logged in successfully

Debug

Debug-level messages for development.

app: Function called with params...

Centralized Logging Concepts

Learn about collecting and aggregating logs from multiple systems.

Syslog Protocol Architecture

Log Sources
Applications, Services, Devices

↓

Local Syslog Daemon
rsyslog / syslog-ng

↓ (UDP 514 / TCP 514 / TLS 6514)

Centralized Log Server
Aggregation & Storage

↓

Analysis & Visualization
ELK Stack, Splunk, Graylog

rsyslog Configuration

                    # /etc/rsyslog.conf - Basic configuration

# Load modules
module(load="imuxsock")    # Local system logging
module(load="imklog")      # Kernel logging

# Global directives
$WorkDirectory /var/spool/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Rules
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*                                   /var/log/secure
mail.*                                       /var/log/maillog
cron.*                                       /var/log/cron
*.emerg                                      :omusrmsg:*

# Forward logs to central server
*.* @@logserver.example.com:514

# Alternative: Save to file with specific format
$template CustomFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\n"
*.* /var/log/custom.log;CustomFormat
                

Key Concepts

Facility

Category of the log source: kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, authpriv, ftp, local0-local7

Priority

Combination of facility and severity level. Example: mail.info, auth.warning, kern.emerg

UDP vs TCP

UDP (514): Fast but unreliable. TCP (514/6514): Reliable with guaranteed delivery. TLS: Encrypted transmission.

Log Aggregation Benefits

Centralized storage, correlation across systems, easier compliance, reduced storage on endpoints, unified search and analysis.

Sample Remote Logging Setup

                    # On the client (sender)
# /etc/rsyslog.d/50-remote.conf
*.* @@192.168.1.100:514   # TCP
# *.* @192.168.1.100:514  # UDP (@ = UDP, @@ = TCP)

# On the server (receiver)
# /etc/rsyslog.d/50-server.conf
module(load="imtcp")
input(type="imtcp" port="514")

# Template for remote logs
$template RemoteHost,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteHost
                

Log Management Quiz

Test your knowledge of log management concepts and best practices.