Chapter 17. Log File Management

Table of Contents

Introduction
System Logger
/dev/log
Log Event Meta Information
System Logger Configuration
Non-Syslog Log Files
Xorg Logging
Gentoo Related Logs
Maintaining Log Files with Logrotate
Installing Logrotate
Configuring Logrotate
Maintaining Log Files with Cron

Introduction

If the previous chapters are all read through, you'll most likely have successfully installed Gentoo Linux. However, managing your Linux system is only starting. Within this chapter, we go through how the various Linux services log events on your system and how you can properly manage these log files.

System Logger

The system logger is a service that allows for various software tools to log their events through a unified, standardized interface. For locally running software, these tools can log through the /dev/log socket. Services that run remotely can send their events through the network. For a general understanding of logging daemons, having to deal with remotely running software might bring us too far.

/dev/log

The /dev/log socket is created by the system logger (which we have very briefly discussed previously) and is made writable for everyone. That is to be expected, as there are many tools that will log through this socket. Every tool that wants to log through the system logger logs a single line at a time inside this socket. The system logger is listening "on the other side" of the socket and will process these log events.

Log Event Meta Information

You will most likely never need to write your own tool that logs through the system logger, yet understanding how a log event is sent is vital if you want to manage your log files properly. The reason for this is because you will want to filter based on criticality and origin of the events, and these are passed on as part of the log event itself.

When a tool wants to log an event, he needs to provide two additional fields:

  • the facility where the event is about, and

  • the importance level of the event

The facility defines the type of the program that is sending out the event. This allows the system logger to properly filter messages, possibly sending them into different log files. Example facilities are authpriv (security/authorization messages), cron (scheduling messages) and kern (kernel messages). A full list of facilities can be obtained through the syslog man page

~$ man 3 syslog

The importance level defines the criticality of the event. The set of supported importance levels is:

Table 17.1. System logger importance levels

DEBUGMessages used for debugging purposes
INFOInformational messages
NOTICEA normal, yet somewhat more important message than informational
WARNINGThis message needs your attention
ERRORSomething wicked has occurred and will need further investigation
CRITA critical error has occurred, regular operations might be interrupted or run in a degraded mode
ALERTImmediate action needs to be taken
EMERGThe system is unusable (no, this has nothing to do with emerge, Gentoo Portage' installation tool)

Based on these two fields, log messages are then filtered by the system logger into one or more log files.

System Logger Configuration

How a system logger is configured depends on the system logger you use. In this book, I'll focus on the syslog-ng logger.

The configuration file for syslog-ng is /etc/syslog-ng/syslog-ng.conf. An example configuration is displayed below.

@version: 3.0
options {
  stats_freq(43200);
};

source src {
  unix-stream("/dev/log" max-connections(256));
  internal();
  file("/proc/kmsg");
};

destination messages { file("/var/log/messages"); };
destination cron { file("/var/log/cron.log"); };
destination auth { file("/var/log/auth.log"); };

filter f_messages { not facility(cron, auth, authpriv); };
filter f_cron { facility(cron); };
filter f_auth { facility(auth, authpriv); };

filter f_warnplus { level(warn, err, crit, emerg); };

log { source(src); filter(f_cron); filter(f_warnplus); destination(cron); };
log { source(src); filter(f_auth); destination(auth); };
log { source(src); filter(f_messages); destination(messages); };

It might be easy to read the configuration file from the bottom up.

  • The log entries define where messages come from (source), which filters the system logger applies (filter) and where the resulting messages are stored in (destination)

  • The filter entries define what the filters actually do. For instance, the filter f_warnplus only accepts events with an importance level of warn or higher.

  • The destination entries define where the events are stored in (the log files)

  • The source entry defines where the system logger gets its messages from (which, in this case, is the /dev/log socket, the kernel message interface kmsg and its own internal logging)

This fairly simple example immediately shows how flexible the logs can work. There are many more interesting filters you can apply, such as match() to match regular expressions within the logged event and program() to match log events of a particular tool.

Non-Syslog Log Files

Many tools log through the system logger, but it is not a huge majority. Lots and lots of tools, server software and others have their own logging system. This makes it a bit more difficult to fully manage the log files properly. However, if you know where the log files are, then that's a start.

Xorg Logging

The Xorg server stores it log file at /var/log/Xorg.0.log. The trailing 0 denotes that this is of the current/last start. The log file of the start before that is called Xorg.1.log, and so on.

Xorg uses the following notations to identify the various criticality levels:

Markers: (--) probed, (**) from config file, (==) default setting,
   (++) from command line, (!!) notice, (II) informational,
   (WW) warning, (EE) error, (NI) not implemented, (??) unknown.

The Xorg server will automatically rotate the log files, by default 3 times, after which it will erase the oldest log file.

Gentoo Related Logs

Gentoo Portage (and other Gentoo tools) have their own set of logs as well.

Package Installation Logs

The package logs (known as elogs) as defined in the chapter on software management are stored in /var/log/portage/elog. The variables in /etc/portage/make.conf that define the location and the logging aspects are:

# Base directory for Portage logging
PORT_LOGDIR="/var/log/portage"
# Type of logging (save = separate log per installation)
PORTAGE_ELOG_SYSTEM="save"
# Log filter (what to log)
PORTAGE_ELOG_CLASSES="info warn error log"

Portage does not clean up old log files though, so we'll need to implement something for ourselves.

Maintaining Log Files with Logrotate

Most tools do not offer log rotation or clean up by default. It is therefore recommended to implement some sort of log rotation for your files. An interesting tool to install for this purpose is app-admin/logrotate. The tool is triggered by your system scheduler (cron) and is best configured by creating separate configuration files for your log files.

Installing Logrotate

First, install logrotate:

~# emerge logrotate

Next, make sure that it is scheduled to be ran every day. For instance, you can have a script called logrotate.cron inside /etc/cron.daily with the following content (Gentoo users will have this as part of the installation):

#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf

Configuring Logrotate

To configure logrotate, create configuration files inside /etc/logrotate.d. For instance, the following /etc/logrotate.d/syslog-ng helps us rotate the log files that we identified as part of the system logger configuration:

/var/log/messages /var/log/cron.log /var/log/auth.log {
  rotate 6
  monthly
  # It is ok if a log file is missing
  missingok
  # Only execute postrotate after all 3 log files are rotated
  # instead of after each log file
  sharedscripts
  # Commands to run after the rotation has occurred
  # This will instruct syslog-ng to reload its configuration
  # and log files
  postrotate 
    /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
  endscript
}

The file informs logrotate that the mentioned log files are rotated 6 times on a monthly basis (so you keep 7 months of history). The tool will automatically rename the "old" log files by adding a date stamp to the filename. You can also instruct logrotate to compress the rotated log files or even move them to a different location.

Maintaining Log Files with Cron

Not all log files can be easily managed with logrotate. For instance, you might want to only remove log files that are older than 1 month. This can be easily accomplished with a simple cron entry. For instance, /etc/cron.weekly/elog-cleanup:

#!/bin/sh

find /var/log/portage/elog -type f -mtime +30 -exec rm '{}' \;