= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

In This Chapter

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

This chapter will show you how to troubleshoot problems not only on your Linux box but also on remote networking devices using syslog.

Syslog is a utility for tracking and logging all manner of system messages from the merely informational to the extremely critical. Each system message sent to the syslog server has two descriptive labels associated with it that makes the message easier to handle.

o The first describes the function (facility) of the application that generated it. For example, applications such as mail and cron generate messages with easily identifiable facilities named "mail" and "cron".

o The second describes the degree of severity of the message. There are eight in all and they are listed below:

Syslog Facilities

Severity Level	Keyword	Description
0	emergencies	System unusable
1	alerts	Immediate action required
2	critical	Critical condition
3	errors	Error conditions
4	warnings	Warning conditions
5	notifications	Normal but significant conditions
6	informational	Informational messages
7	debugging	Debugging messages

The files to which syslog will write each type of message received is set in the /etc/syslog.conf configuration file. This file consists of two columns, the first lists the facilities and severities of messages to expect and the second lists the files to which they should be logged. By default, RedHat's /etc/syslog.conf file is configured to put most of the messages the file /var/log/messages. Here is a sample:

.info;mail.none;authpriv.none;cron.none /var/log/messages

In this case, all messages of severity "info" and above are logged, but none from the mail, cron or authentication facilities/subsystems. You can make this logging even more sensitive by replacing the line above with one that captures all messages from debug severity and above in the /var/log/messages file. This may be more suitable for troubleshooting.

*.debug /var/log/messages

Certain applications will additionally log to their own application specific log files and directories independent of the syslog.conf file. Here are some common examples:

Files:

/var/log/maillog : Mail

/var/log/httpd/access_log : Apache web server page access logs

Directories:

/var/log

/var/log/samba : Samba messages

/var/log/mrtg : MRTG messages

/var/log/httpd : Apache webserver messages

NOTE: The /etc/syslog.conf file is very sensitive to spaces. Only use tabs on lines that don't start with the "#" comment character. Spaces in the file will cause unpredictable results.

Activating Changes To The syslog Configuration File

Changes to /etc/syslog.conf will not take effect until you restart syslog. Issue this command to do so:

[root@bigboy tmp]# /etc/init.d/syslog restart

How To View New Log Entries As They Happen

If you want to get new log entries to scroll on the screen as they occur, then you can use this command:

[root@bigboy tmp]# tail -f /var/log/messages

Similar commands can be applied to all log files. This is probably one of the best troubleshooting tools available in Linux. Another good command to use apart from "tail" is "grep". Grep will help you search for all occurrences of a string in a log file, you can pipe it through the "more" command so that you only get one screen at a time. Here is an example.

[root@bigboy tmp]# grep string /var/log/messages | more

You can also just use the plain old "more" command to see one screen at a time of the entire log file without filtering with "grep". Here is an example:

[root@bigboy tmp]# more /var/log/messages

Logging Syslog Messages To A Remote Linux Server

Configuring the Linux Syslog Server

By default syslog doesn’t expect to receive messages from remote clients. Here’s how to configure your Linux server to start listening for these messages.

As we saw previously, syslog checks its /etc/syslog.conf file to determine the expected names and locations of the log files it should create. It also checks the file /etc/sysconfig/syslog to determine the various modes in which it should operate. Syslog will not listen for remote messages unless the SYSLOGD_OPTIONS variable in this file has a “–r” included in it as shown below.

# Options to syslogd

# -m 0 disables 'MARK' messages.

# -r enables logging from remote machines

# -x disables DNS lookups on messages recieved with -r

# See syslogd(8) for more details

SYSLOGD_OPTIONS="-m 0 -r"

# Options to klogd

# -2 prints all kernel oops messages twice; once for klogd to decode, and

# once for processing with 'ksymoops'

# -x disables all klogd processing of oops messages entirely

# See klogd(8) for more details

KLOGD_OPTIONS="-2"

You will have to restart syslog on the server for the changes to take effect. The server will now start to listen on UDP port 514 which you can verify using either one of the following netstat command variations.

[root@bigboy tmp]# netstat -a | grep syslog

udp 0 0 *:syslog *:*

[root@bigboy tmp]# netstat -an | grep 514

udp 0 0 0.0.0.0:514 0.0.0.0:*

[root@bigboy tmp]#

Configuring the Linux Client

The syslog server is now expecting to receive syslog messages. You now have to configure your remote Linux client to send messages to it. This is done by editing the /etc/hosts file on the Linux client named smallfry. Here are the steps:

· Determine the IP address and fully qualified hostname of your remote logging host

· Add an entry in the /etc/hosts file in the format

IP-address fully-qualified-domain-name hostname "loghost"

Example:

192.168.1.100 bigboy.my-site.com bigboy loghost

Server "bigboy" has now become the remote logging server as its /etc/hosts entry has an alias "loghost" which indicates to syslog that this is a remote syslog server. Remember to restart syslog to make these changes take effect.

You can now test to make sure that the syslog server is receiving the messages with a simple test such as restarting the lpd printer daemon and making sure the remote server sees the messages.

Linux Client

[root@smallfry tmp]# /etc/init.d/lpd restart

Stopping lpd: [ OK ]

Starting lpd: [ OK ]

[root@smallfry tmp]#

Linux Server

[root@bigboy tmp]# tail /var/log/messages

...

Apr 11 22:09:35 smallfry lpd: lpd shutdown succeeded

Apr 11 22:09:39 smallfry lpd: lpd startup succeeded

...

[root@bigboy tmp]#

Syslog Configuration and Cisco Network Devices

Syslog reserves facilities "local0" through "local7" for log messages received from remote servers and network devices. Routers, switches, firewalls and load balancers each logging with a different facility can each have their own log files for easy troubleshooting. The Cisco Home Networking companion guide has examples of how to configure syslog to do this with Cisco devices using separate log files for the routers, switches, PIX firewalls, CSS arrowpoints and LocalDirectors.

Syslog and Firewalls

Syslog listens by default on UDP port 514. If you are logging to a remote syslog server via a firewall, you'll have to allow traffic on this port to pass through the security device.

Logrotate

Logrotate is a Linux utility that renames and reuses system error log files on a periodic basis so that they don't occupy excessive disk space.

The /etc/logrotate.conf File

This is logrotate’s general configuration file in which you can specify the frequency with which the files are reused.

o You can specify either “weekly” or “daily” rotation parameter. In the case below the weekly option is "commented out" with a "#", allowing for daily updates.

o The "rotate" parameter specifies the number of copies of log files logrotate will maintain. In the case below the 4 copy option is "commented out" with a "#", while allowing 7 copies.

o The "create" parameter creates a new log file after each rotation

Therefore our sample configuration file will create daily archives of ALL the logfiles and store them for seven days. The files will have the following names with "logfile" being current active version:

logfile

logfile.0

logfile.1

logfile.2

logfile.3

logfile.4

logfile.5

logfile.6

Sample contents of /etc/logrotate.conf

# rotate log files weekly
#weekly

# rotate log files daily
daily

# keep 4 weeks worth of backlogs
#rotate 4

# keep 7 days worth of backlogs
rotate 7

# create new (empty) log files after rotating old ones
create

The /etc/logrotate.d Directory

Most Linux applications that use syslog will put an additional configuration file in this directory to specify the names of the log files to be rotated. It is a good practice to verify that all new applications that you want to use the syslog log have configuration files in this directory. Here are some sample files which define the specific files to be rotated for each application.

The /etc/logrotate.d/syslog File (For General System Logging)

/var/log/cisco/* /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron {

    sharedscripts
    postrotate
    /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript

}

The /etc/logrotate.d/apache File (For Apache)

/var/log/httpd/access_log /var/log/httpd/agent_log /var/log/httpd/error_log /var/log/httpd/referer_log {

    missingok
    sharedscripts
    postrotate
    /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript

}

The /etc/logrotate.d/samba File (for SAMBA)

/var/log/samba/*.log {

    notifempty
    missingok
    sharedscripts
    copytruncate
    postrotate
    /bin/kill -HUP `cat /var/lock/samba/*.pid 2> /dev/null` 2> /dev/null || true
    endscript

}

Activating logrotate

The above logrotate settings will not take effect until you issue the following command to do so:

[root@bigboy tmp]# logrotate –f

If you want logrotate to reload only a specifc configuration file, and not all of them, then issue the logrotate command with just that filename as the argument like this:

[root@bigboy tmp]# logrotate -f /etc/logrotate.d/syslog

Chapter 7

Troubleshooting Linux With Syslog

Syslog

About syslog