Red Hat Knowledgebase

Release Found: Red Hat Enterprise Linux 3

Symptom:
The audit facility is enabled with the default configuration. When the filesystem containing the audit logs exceeds 80% utilization, things start randomly hanging on the system.

Solution:
The audit facility in Red Hat Enterprise Linux 3 is a means to monitor system calls. The default configuration shipped with Red Hat Enterprise Linux 3 logs these system calls to a set of binary logfiles that live in /var/log/audit.d. The configuration files for auditd live in: /etc/audit, with the main configuration file being /etc/audit/audit.conf.

This file, by default, contains the following line in the output section:

notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";

When the free space in the filesystem holding the audit logs is less than 20%, the above notify command will error out and auditd will enter suspend mode. This causes all system calls to block and will produce the symptoms described previously in this article.

When this occurs, you will also see errors similar to the following in /var/log/messages:

Jun 26 06:50:00 foobar audbin[23924]: saving binary audit log /var/log/audit.d/bin.2
Jun 26 06:50:00 foobar auditd[1946]: Notify command /usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% terminated abnormally
Jun 26 06:50:00 foobar auditd[1946]: output error
Jun 26 06:50:00 foobar auditd[1946]: output error
Jun 26 06:50:00 foobar auditd[1946]: output error; suspending execution

There are several ways to deal with this. One way is to simply reduce the 20% value of the -T option to a percentage that is closer to the actual size of the logfiles. For example, if a /var filesystem is 5GB in size, then 20MB (the size of one of the binary logfiles) is less than 1% of this, so reducing this value to 2-3% should be reasonable.

Another way is to provide a -N option to the notify audbin command that it can use to take corrective action when this occurs. Some suggestions are in the comments in the default audit.conf file:

#  AUDBIN THRESHOLDS:
#       The above notify will cause auditd to enter 'suspend' mode when
#       free space on the /var/ filesystem falls below 20%.
#       To take remedial action, eg. moving the oldest save file to /backup, use:
#       notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'mv -f %f /backup'";
#       or even
#       notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'rm -f %f'";
#       This will free space by removing the oldest "save." files first from /var,
#       returning 0 to auditd and allowing it to continue.

More information on the audit facility is available starting with the laus(7) manpage:

# man 7 laus

Detailed information is also available in the eal3-certification-doc-1.1 package. You can install this package with up2date:

# up2date eal3-certification-doc

The documentation will then be installed under /usr/share/doc/eal3-certification-doc-1.1.