Protokol auditu a soubor zpráv se na CentOS/RHEL neotáčejí

Problém

Protokol auditu a soubor zpráv se neotáčejí.
Otáčení souborů /var/log/messages, /var/log/audit.log a /var/log/asmaudit.log neprobíhá na úrovni OS.

$ ls -ltr /var/log/audit*
-rw------- 1 root root 4436099 Mar 22 03:07 audit.log.10.gz
-rw------- 1 root root 3873269 Mar 23 03:07 audit.log.9.gz
-rw------- 1 root root 4036499 Mar 25 03:07 audit.log.8.gz
-rw------- 1 root root 3454722 Mar 26 03:08 audit.log.7.gz
-rw------- 1 root root 6145628 Mar 27 03:07 audit.log.6.gz
-rw------- 1 root root 2979527 Mar 28 03:07 audit.log.5.gz
-rw------- 1 root root 3797834 Mar 29 03:07 audit.log.4.gz
-rw------- 1 root root 4283296 Mar 30 03:08 audit.log.3.gz   ----->>> No logs rotated for 20 days
-rw------- 1 root root 96190 Apr 20 11:20 audit.log.2.gz       ----->>> No logs rotated for 20 days
-rw------- 1 root root 387 Apr 22 12:16 audit.log.1.gz
-rw------- 1 root root 507157 May 3 10:26 audit.log

$ ls -ltr /var/log/mess*
-rw-r----- 1 root orarom 2996230 Mar 28 03:07 messages.6.gz
-rw-r----- 1 root orarom 3821592 Mar 29 03:07 messages.5.gz
-rw-r----- 1 root orarom 4324393 Mar 30 03:08 messages.4.gz
-rw-r----- 1 root orarom 5240535 Mar 31 03:08 messages.3.gz   ----->>> No logs rotated for 20 days
-rw-r----- 1 root orarom 96226018 Apr 20 11:20 messages.2.gz  ----->>> No logs rotated for 20 days
-rw-r----- 1 root orarom 10463366 Apr 22 12:16 messages.1.gz
-rw-r----- 1 root orarom 1555913366 May 3 10:26 messages

$ ls -ltr /var/log/asmau*
-rw------- 1 root root 4005971 Mar 26 03:07 asmaudit.log.4.gz  
-rw------- 1 root root 4128224 Apr 20 11:19 asmaudit.log.3.gz
-rw------- 1 root root 16248312 Apr 22 12:16 asmaudit.log.2.gz
-rw------- 1 root root 74179752 Apr 22 12:16 asmaudit.log.1
-rw------- 1 root root 396961019 May 3 10:27 asmaudit.log

$ cat /var/lib/logrotate.status | grep mess
"/var/log/messages" 2017-4-22                               ----->>> Logs rotated manually

Kořenová příčina

cron.daily úloha uzamčena jinou úlohou anacronu. Na výstupu ps vidíme, že anacron stále běží a blokuje ostatní procesy.

$ ps -ef|grep cron
root 17592 1 0 Mar11 ? 00:00:16 crond
root 132079 232781 0 Mar31 ? 00:00:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 232781 1 0 Mar31 ? 00:00:00 /usr/sbin/anacron -s
oracle 264332 233472 0 06:37 pts/6 00:00:00 grep cron
root 392965 132079 0 Mar31 ? 00:00:00 /bin/sh /etc/cron.daily/mlocate.cron
root 392966 132079 0 Mar31 ? 00:00:00 awk -v progname=/etc/cron.daily/mlocate.cron progname {????? print progname ":\n"????? progname="";???? }???? { print; }

Řešení

Zabijte zaseknuté procesy cronu a restartujte proces anacronu. Například:

$ ps -ef|grep cron
root 17592 1 0 Mar11 ? 00:00:16 crond
root 132079 232781 0 Mar31 ? 00:00:00 /bin/bash /usr/bin/run-parts /etc/cron.daily
root 1320791 0 Mar31 ? 00:00:00 /usr/sbin/anacron -s
oracle 264332 233472 0 06:37 pts/6 00:00:00 grep cron
root 392965 132079 0 Mar31 ? 00:00:00 /bin/sh /etc/cron.daily/mlocate.cron
root 392966 132079 0 Mar31 ? 00:00:00 awk -v progname=/etc/cron.daily/mlocate.cron progname {????? print progname ":\n"????? progname="";???? }???? { print; }

a mělo by to být jako níže

$ ps -ef|grep cron
oracle 75374 61598 0 00:37 pts/2 00:00:00 grep cron
root 387494 1 0 Jul07 ? 00:00:00 crond

Poté prosím sledujte server po dobu jednoho dne, zda bude střídat protokoly.

$ egrep "messages|audit" /var/lib/logrotate.status
"/var/log/asmaudit.log" 2017-7-9
"/var/log/messages" 2017-7-9
"/var/log/audit.log" 2017-7-8

$ ls -ltr /var/log/audit*
-rw------- 1 root root 4036499 Mar 25 03:07 /var/log/audit.log.10.gz
-rw------- 1 root root 3454722 Mar 26 03:08 /var/log/audit.log.9.gz
-rw------- 1 root root 6145628 Mar 27 03:07 /var/log/audit.log.8.gz
-rw------- 1 root root 2979527 Mar 28 03:07 /var/log/audit.log.7.gz
-rw------- 1 root root 3797834 Mar 29 03:07 /var/log/audit.log.6.gz
-rw------- 1 root root 4283296 Mar 30 03:08 /var/log/audit.log.5.gz
-rw------- 1 root root 96190 Apr 20 11:20 /var/log/audit.log.4.gz
-rw------- 1 root root 387 Apr 22 12:16 /var/log/audit.log.3.gz
-rw----r-- 1 root root 109466 Jun 27 10:39 /var/log/audit.log.2.gz
-rw----r-- 1 root root 3530666 Jul 8 03:08 /var/log/audit.log.1.gz
-rw----r-- 1 root root 180505494 Jul 10 00:38 /var/log/audit.log

$ ls -ltr /var/log/mess*
-rw-r----- 1 root orarom 96226018 Apr 20 11:20 /var/log/messages.6.gz
-rw-r----- 1 root orarom 10463366 Apr 22 12:16 /var/log/messages.5.gz
-rw-r----- 1 root orarom 206584051 Jun 27 10:39 /var/log/messages.4.gz
-rw-r----- 1 root orarom 58913698 Jul 7 11:09 /var/log/messages.3.gz
-rw-r----- 1 root orarom 3541858 Jul 8 03:08 /var/log/messages.2.gz
-rw-r----- 1 root orarom 6033888 Jul 9 03:08 /var/log/messages.1.gz
-rw-r----- 1 root orarom 180613996 Jul 10 00:38 /var/log/messages

$ ls -ltr /var/log/asmau*
-rw------- 1 root root 1542952 Jun 27 10:37 /var/log/asmaudit.log.4.gz
-rw----r-- 1 root root 22711243 Jul 7 11:08 /var/log/asmaudit.log.3.gz
-rw----r-- 1 root root 3626457 Jul 9 03:08 /var/log/asmaudit.log.2.gz
-rw----r-- 1 root root 42116158 Jul 9 03:08 /var/log/asmaudit.log.1
-rw----r-- 1 root root 22786400 Jul 10 00:39 /var/log/asmaudit.log