Použití žurnálů systemd k řešení přechodných problémů

Další možnosti dostupné pomocí -o možnost umožňuje export dat v různých formátech, jako je binární nebo JSON. Také jsem našel -x možnost svítí, protože může zobrazit další vysvětlující zprávy pro některé položky deníku. Pokud tuto možnost vyzkoušíte, uvědomte si, že může výrazně zvýšit výstupní datový tok. Například dodatečné informace pro položku jako:

[    4.503737] testvm1.both.org systemd[1]:Spouštění kontroly systému souborů na /dev/mapper/VG01-root...
[    4.691555] testvm1.both.org systemd-fsck[548] :root:čistý, 1813/327680 souborů, 48555/1310720 bloků
[    4.933065] testvm1.both.org systemd[1]:Dokončená kontrola souborového systému na /dev/mapper/VG01-root.

expanduje na toto:

[    4.503737] testvm1.both.org systemd[1]:Spouštění kontroly systému souborů na /dev/mapper/VG01-root...
-- Předmět:Spouštěcí úloha pro jednotku systemd-fsck-root Služba .service byla zahájena
-- Defined-By:systemd
-- Podpora:https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Spouštěcí úloha pro jednotku systemd-fsck-root.service byla zahájena.
-- 
-- Identifikátor úlohy je 36.
[    4.691555] testvm1.both.org systemd -fsck[548]:root:čistý, 1813/327680 souborů, 48555/1310720 bloků
[    4.933065] testvm1.both.org systemd[1]:Dokončená kontrola souborového systému na /dev/mapper/VG01-root. 
-- Předmět:Spouštěcí úloha pro jednotku systemd-fsck-root.service byla úspěšně dokončena
-- Defined-By:systemd
-- Podpora:https://lists.freedesktop. org/mailman/listinfo/systemd-devel
-- 
-- Spouštěcí úloha pro jednotku systemd-fsck-root.service byla úspěšně dokončena.
-- 
-- The identifikátor práce je 36

Jsou zde nějaké nové informace, ale myslím, že hlavní výhodou je, že informace jsou kontextualizovány, aby do určité míry objasnily původní stručné zprávy.

Zúžení vyhledávání

Většinou není nutné a ani žádoucí vypisovat všechny deníkové záznamy a ručně v nich vyhledávat. Někdy hledám záznamy související s konkrétní službou a jindy hledám záznamy, které se staly v konkrétních časech. journalctl poskytuje výkonné možnosti, které vám umožní zobrazit pouze data, která vás zajímají.

Začněte s --list-boots volba, která uvádí všechna spouštění během časového období, kdy existují záznamy žurnálu. Všimněte si, že journalctl.conf soubor může specifikovat, že záznamy žurnálu jsou vyřazeny poté, co dosáhnou určitého stáří nebo poté, co místo na úložném zařízení (HDD/SSD) zabrané žurnálem dosáhne určené maximální velikosti:

[root@testvm1 ~]# journalctl --list-boots 
-10 dcb6dcc0658e4a8d8c781c21a2c6360d Po 2020-06-08 07:47:20 EDT:Po 2020-076-08 EDT:Po 2020-076-08 /> -9 7D61951A85F445C5946774AAAAE8BC2A4 PÁN 2020-07-03 15:50:06 EDT-FRI 2020-07-03 18:21:23 EDT 
 -8 1b3A847577E544B4A2679FE576B62206 PRI 2020-07-03 18:21:58 EDT- Pá 2020-07-03 22:10:54 EDT
 -7 5fef01a3568743af99118107ca6f61ae Pá 2020-07-03 22:18:41 EDT-So 2020<0-01-06:5604 6E944F999EBD9405984090F829A927FA4 SAT 2020-07-04 07:33:25:56 EDT-SAT 2020-07-04 07:58:59 EDT 
 -5 -5 -5 EC8B0C81CA4744B1BAD071BDEF432959 So 2020-07-04 08:12:06 EDT-SAT 2020-07 -04 09:12:47 EDT
 -4 cb173ec716824e21b87ccf6cb43a9a99 So 2020-07-04 10:19:53 EDT-So 2020-07-04 2020-07-04 E25431>21b87ccf /83938 07-04 07:59:58 EDT–Ne 2020-07-05 09:39:30 EDT
 -2 06fb81f1b29e4f68af9860844668446c Po 2020-07-07-05-05 02:02 50:06 EDT
 -1 33dbf8e6b9de4113a591c4f487d0ac37 Po 2020-07-13 04:50:33 EDT— Čt 2020-07-16 13:49:32 EDT
  0 75c9b70913934748b5396b3b172bee50 Po 2020-07-20 08:43:01 EDT-Pá 20240:02:02 

 Nejnovější ID spouštění se zobrazí ve spodní části; je to dlouhé náhodné hexadecimální číslo. Tato data můžete použít k zobrazení žurnálů pro konkrétní spouštění. To lze zadat pomocí čísla posunu spouštění ve sloupci zcela vlevo nebo pomocí UUID ve druhém sloupci. Tento příkaz zobrazí žurnál spouštěcí instance s offsetem -2 —druhé předchozí spuštění z aktuálního:
 
[root@testvm1 ~]# journalctl -b -2
-- Záznamy začínají v Po 2020-06-08 07:47:20 EDT, končí v Pá 2020-07-24 12:44:06 EDT. --
Jul 06 06:27:05 testvm1.both.org jádro:Linux verze 5.7.6-201.fc32.x86_64 ([email protected]) (gcc verze 10.1>
Jul 06 06:27:05 testvm1.both.org kernel:Command line:BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.7.6-201.fc32.x86_64 root=/dev/mapper/VG01-root ro>
Jul 06 06:27:05 testvm1.both.org kernel:x86/fpu:Supporting XSAVE feature 0x001:'x87 floating point registers'
Jul 06 06:27:05 testvm1.both.org kernel:x86/fpu:Supporting XSAVE feature 0x002:'SSE registers'
 

 Or you could use the UUID for the desired boot. The offset numbers change after each boot, but the UUID does not:
 
[root@testvm1 ~]# journalctl -b 06fb81f1b29e4f68af9860844668446c 
 

 The -u option allows you to select specific units to examine. You can use a unit name or a pattern for matching, and you can use this option multiple times to match multiple units or patterns. In this example, I used it in combination with -b to show chronyd journal entries for the current boot:
 
[root@testvm1 ~]# journalctl -u chronyd -b
-- Logs begin at Mon 2020-06-08 07:47:20 EDT, end at Sun 2020-07-26 09:10:47 EDT. --
Jul 20 12:43:31 testvm1.both.org systemd[1]:Starting NTP client/server...
Jul 20 12:43:31 testvm1.both.org chronyd[811]:chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCD>
Jul 20 12:43:31 testvm1.both.org chronyd[811]:Frequency -0.021 +/- 0.560 ppm read from /var/lib/chrony/drift
Jul 20 12:43:31 testvm1.both.org chronyd[811]:Using right/UTC timezone to obtain leap second data
Jul 20 12:43:31 testvm1.both.org systemd[1]:Started NTP client/server.
Jul 20 12:44:00 testvm1.both.org chronyd[811]:Selected source 192.168.0.52
Jul 20 12:44:00 testvm1.both.org chronyd[811]:System clock TAI offset set to 37 seconds
Jul 20 12:44:00 testvm1.both.org chronyd[811]:System clock wrong by 1.412227 seconds, adjustment started
Jul 20 12:44:01 testvm1.both.org chronyd[811]:System clock was stepped by 1.412227 seconds
[root@testvm1 ~]#
 

 Suppose you want to look at events that were recorded between two arbitrary times. You can also use -S (--since ) and -U (--until ) to specify the beginning and ending times. The following command displays journal entries starting at 15:36:00 on July 24, 2020, through the current time:
 
[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00" 
 

 And this command displays all journal entries starting at 15:36:00 on July 24, 2020, until 16:00:00 on July 25:
 
[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00" -U "2020-07-25 16:00:00" 
 

 This command combines -S , -U , and -u to give journal entries for the NetworkManager service unit starting at 15:36:00 on July 24, 2020, until 16:00:00 on July 25:
 
[root@testvm1 ~]# journalctl -S "2020-07-24 15:36:00" -U "2020-07-25 16:00:00" -u NetworkManager 
 

 Some syslog facilities, such as cron, auth, mail, daemon, user, and more, can be viewed with the --facility volba. You can use --facility=help to list the available facilities. In this example, the mail facility is not the Sendmail service that would be used for an email service, but the local client used by Linux to send email to root as event notifications. Sendmail actually has two parts, the server, which (for Fedora and related distributions) is not installed by default, and the client, which is always installed so that it can be used to deliver system emails to local recipients, especially root:
 
[root@testvm1 ~]# journalctl --facility=mail 
 

 The journalctl man page lists all the options that can be used to narrow searches. The table below summarizes some of the options I use most frequently. Most of these options can be used in various combinations to further narrow a search. Refer to my previous article Analyzing systemd calendar and timespans for details on creating and testing timestamps, as well as important tips like using quotes around timestamps.
 
Options to narrow searches of the journal
 
Option
 
Description
 
 
 
--list-boots 
 
This displays a list of boots. The information can be used to show journal entries only for a particular boot.
 
 
-b [offset|boot ID] 
 
This specifies which boot to display information for. It includes all journal entries from that boot through shutdown or reboot.
 
 
--facility=[facility name] 
 
This specifies the facility names as they're known to syslog. Use --facility=help to list the valid facility names.
 
 
-k , --dmesg 
 
These display only kernel messages and are equivalent to using the dmesg command.
 
 
-S , --since [timestamp]  
 
These show all journal entries since (after) the specified time. They can be used with --until  to display an arbitrary range of time. Fuzzy times such as "yesterday" and "2 hours ago"—with quotes—are also allowed.
 
 
-u [unit name] 
 
The -u option allows you to select specific units to examine. You can use a unit name or a pattern for matching. This option can be used multiple times to match multiple units or patterns. 
 
 
-U , --until [timestamp] 
 
These show all journal entries until (prior to) the specified time. They can be used with --since  to display an arbitrary range of time. Fuzzy times such as "yesterday" and "2 hours ago"—with quotes—are also allowed.
 
 
 
 
Other interesting options
 

 The journalctl program offers some other interesting options, as well. These are useful for refining the data search, specifying how the journal data is displayed, and managing the journal files.
 
Additional interesting journalctl options
 
Option 
 
Description 
 
 
-f , --follow 
 
This journalctl option is similar to using the tail -f příkaz. It shows the most recent entries in the journal that match whatever other options have been specified and also displays new entries as they occur. This can be useful when watching for events and when testing changes.
 
 
-e , --pager-end 
 
The -e option displays the end of the data stream instead of the beginning. This does not reverse the order of the data stream, rather it causes the pager to jump to the end.
 
 
--file [journal filename] 
 
This names a specific journal file in /var/log/journal/ .
 
 
-r , --reverse 
 
This option reverses the order of the journal entries in the pager so that the newest are at the top rather than the bottom.
 
 
-n , --lines=[X] 
 
This shows the most recent X number of lines from the journal.
 
 
--utc 
 
This displays times in UTC rather than local time.
 
 
-g , --grep=[REGEX] 
 
I like the -g option because it enables me to search for specific patterns in the journal data stream. This is just like piping a text data stream through the grep příkaz. This option uses Perl-compatible regular expressions.
 
 
--disk-usage 
 
This option displays the amount of disk storage used by the current and archived journals. It might not be as much as you think.
 
 
--flush 
 
Journal data stored in the virtual filesystem /run/log/journal/ , which is volatile storage, is written to /var/log/journal/ which is persistent storage. This option ensures that all data is flushed to /run/log/journal/ at the time it returns.
 
 
--sync 
 
This writes all unwritten journal entries (still in RAM but not in /run/log/journal ) to the persistent filesystem. All journal entries known to the journaling system at the time the command is entered are moved to persistent storage.
 
 
--vacuum-size= --vacuum-time= --vacuum-files= 
 
These can be used singly or in combination to remove the oldest archived journal files until the specified condition is met. These options only consider archived files, and not active files, so the result might not be exactly what was specified.
 
 
 
 

 I'll explore some of these entries below. More options can be found in the journalctl man page.
 
Journal files
 

 If you have not already, be sure to list the files in the journal directory on your host. Remember that the name of the directory containing the journal files is a long, random number. This directory contains multiple active and archived journal files, including some for users:
 
[root@david ~]# cd /var/log/journal/ad8f29ed15044f8ba0458c846300c2a4/
[root@david ad8f29ed15044f8ba0458c846300c2a4]# ll
total 352308
-rw-r-----+ 1 root systemd-journal  33554432 May 28 13:07 system@0c91aaef57c441859ea5e421edff6528-0000000000000001-0005a6703120d448.journal
-rw-r-----+ 1 root systemd-journal 109051904 Jun 23 21:24 system@0c91aaef57c441859ea5e421edff6528-0000000000008238-0005a6b85e4e03c6.journal
-rw-r-----+ 1 root systemd-journal 100663296 Jul 21 18:39 system@0c91aaef57c441859ea5e421edff6528-0000000000021f3e-0005a8ca55efa66a.journal
-rw-r-----+ 1 root systemd-journal  41943040 Jul 30 09:34 system.journal
-rw-r-----+ 1 root systemd-journal   8388608 May 28 13:07 user-1000@037bcc7935234a5ea243b3af304fd08a-0000000000000c45-0005a6705ac48a3c.journal
-rw-r-----+ 1 root systemd-journal  16777216 Jun 23 21:24 user-1000@bc90cea5294447fba2c867dfe40530db-0000000000008266-0005a6b85e910761.journal
-rw-r-----+ 1 root systemd-journal  41943040 Jul 21 16:08 user-1000@bc90cea5294447fba2c867dfe40530db-0000000000021f4b-0005a8ca68c83ab7.journal
-rw-r-----+ 1 root systemd-journal   8388608 Jul 30 09:34 user-1000.journal
[root@david ad8f29ed15044f8ba0458c846300c2a4]#
 

 You can see the user files in this listing for the user ID (UID) 1000, which is my Linux account. The --files option allows me to see the content of specified files, including the user files:
 
[root@david ad8f29ed15044f8ba0458c846300c2a4]# journalctl --file user-1000.journal

Jul 29 14:13:32 david.both.org tumblerd[145509]:Registered thumbnailer /usr/bin/gdk-pixbuf-thumbnailer -s %s %u %o
Jul 29 14:13:32 david.both.org Thunar[2788]:ThunarThumbnailer:got 0 handle (Queue)
Jul 29 14:13:32 david.both.org Thunar[2788]:ThunarThumbnailer:got 0 handle (Error or Ready)
Jul 29 14:13:32 david.both.org Thunar[2788]:ThunarThumbnailer:got 0 handle (Finished)
Jul 29 14:15:33 david.both.org tumblerd[145552]:error:Broken zip file structure
Jul 29 14:20:34 david.both.org systemd[2466]:dbus-:[email protected]:Succeeded.
Jul 29 14:34:17 david.both.org systemd[2466]:Starting Cleanup of User's Temporary Files and Directories...
Jul 29 14:34:17 david.both.org systemd[2466]:systemd-tmpfiles-clean.service:Succeeded.
Jul 29 14:34:17 david.both.org systemd[2466]:Finished Cleanup of User's Temporary Files and Directories.
Jul 29 14:48:26 david.both.org systemd[2466]:Started dbus-:[email protected].
Jul 29 14:48:26 david.both.org tumblerd[145875]:Registered thumbnailer gsf-office-thumbnailer -i %i -o %o -s %s
 

 This output shows, among other things, temporary file cleanup for the UID1000 user. Data relating to individual users may be helpful in locating the root cause of problems originating in user space. I found a number of interesting entries in this output. Try it on your host and see what you find.
 
Adding journal entries
 

 It can be useful to add your own entries to the journal. This is accomplished with the systemd-cat program that allows piping the STDOUT of a command or program to the journal. This command can be used as part of a pipeline on the command line or in a script:
 
[root@testvm1 ~]# echo "Hello world" | systemd-cat -p info -t myprog
[root@testvm1 ~]# journalctl -n 10
Jul 27 09:01:01 testvm1.both.org CROND[976442]:(root) CMD (run-parts /etc/cron.hourly)
Jul 27 09:01:01 testvm1.both.org run-parts[976445]:(/etc/cron.hourly) starting 0anacron
Jul 27 09:01:01 testvm1.both.org run-parts[976451]:(/etc/cron.hourly) finished 0anacron
Jul 27 09:07:53 testvm1.both.org unknown[976501]:Hello world
Jul 27 09:10:47 testvm1.both.org systemd[1]:Starting system activity accounting tool...
Jul 27 09:10:47 testvm1.both.org systemd[1]:sysstat-collect.service:Succeeded.
Jul 27 09:10:47 testvm1.both.org systemd[1]:Finished system activity accounting tool.
Jul 27 09:10:47 testvm1.both.org audit[1]:SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/syst>
Jul 27 09:10:47 testvm1.both.org audit[1]:SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/syste>
Jul 27 09:17:10 testvm1.both.org myprog[976516]:Hello world
[root@testvm1 ~]#
 

 The -p option specifies a priority, emerg, alert, crit, err, warning, notice, info, debug, or a value between 0 and 7 that represents each of those named levels. These priority values are the same as those defined by syslog(3) . The default is info. The -t option is an identifier, which can be any arbitrary short string, such as a program or script name. This string can be used for searches by the journalctl příkaz:
 
[root@testvm1 ~]# journalctl -t myprog
-- Logs begin at Mon 2020-06-08 07:47:20 EDT, end at Mon 2020-07-27 09:21:57 EDT. --
Jul 27 09:17:10 testvm1.both.org myprog[976516]:Hello world
[root@testvm1 ~]#
 
Journal management
 

 I use the --disk-usage option to check on journal sizes, along with other commands relating to disk usage, to ensure that my /var filesystem is not filling up:
 
[root@testvm1 ~]# journalctl --disk-usage 
Archived and active journals take up 136.0M in the file system.
[root@testvm1 ~]#
 

 The disk usage for the journals on the testvm1 host is about 136MB. The result on my primary workstation is 328MB, and the host I use for my firewall and router uses 2.8GB for the journals. Journal sizes depend greatly on the host's use and daily run time. My physical hosts all run 24x7.
 

 The /etc/systemd/journald.conf file can be used to configure the journal file sizes, rotation, and retention times to meet any needs not met by the default settings. You can also configure the journal storage location—you can specify a directory on the storage device or whether to store everything in RAM, which is volatile storage. If the journals are stored in RAM, they will not be persistent between boots.
 

 The default time unit in the journald.conf file is seconds, but it can be overridden using the suffixes year , month , week , day , h , or m .
 

 Suppose you want to limit the total amount of storage space allocated to journal files to 1GB, store all journal entries in persistent storage, keep a maximum of 10 files, and delete any journal archive files that are more than a month old. You can configure this in /etc/systemd/journald.conf using:
 
SystemMaxUse=1G
Storage=persistent 
SystemMaxFiles=10
MaxRetentionSec=1month
 

 By default, the SystemMaxUse is 10% of available disk space. The default settings have been fine for the systems I work with, and I have not needed to change any of them. The journald.conf man page states that the time-based settings for specifying how long to store journal entries in a single file or to retain older files are normally not necessary. This is because file number and size configurations usually force rotation and deletion of old files before any time settings might be needed.
 

 The SystemKeepFree option ensures a specific amount of space is kept free for other data. Many databases and application programs use the /var filesystem to store data, so ensuring enough available storage space may be critical in systems with smaller hard drives and minimum space allocated to /var .
 

 If you make changes to this configuration, be sure to monitor the results carefully for an appropriate period of time to ensure they are performing as expected.
 
Journal file rotation
 

 The journal files are typically rotated automatically based upon the configuration in the /etc/systemd/journald.conf soubor. Files are rotated whenever one of the specified conditions is met. So if, for example, the space allocated to journal files is exceeded, the oldest file(s) is deleted, the active file is made into an archive, and a new active file is created.
 

 Journal files can also be rotated manually. I suggest using the --flush option to ensure current data is moved to persistent storage before you run the command:
 
[root@testvm1 ~]# journalctl --rotate 
 

 There is another way to purge old journal files without performing a file rotation. The vacuum-size= , vacuum-files= , and vacuum-time= commands can be used to delete old archive files down to a specified total size, number of files, or prior time. The option values consider only the archive files and not the active ones, so the resulting reduction in total file size might be somewhat less than expected.
 

 The following command purges old archive files so that only ones that are less than one month old are left. You can use the s , m , h , days , months , weeks , and years suffixes:
 
[root@testvm1 ~]# journalctl --vacuum-time=1month  
 

 This command deletes all archive files except for the four most recent ones. If there are fewer than four archive files, nothing will happen, and the original number of files remains:
 
[root@testvm1 ~]# journalctl --vacuum-files=4 
 

 This last vacuum command deletes archive files until 200MB or less of archived files are left:
 
[root@testvm1 ~]# journalctl --vacuum-size=200M 
 

 Only complete files are deleted. The vacuum commands do not truncate archive files to meet the specification. They also work only on archive files, not active ones.
 
Final thoughts
 

 This article looked at using the journalctl command to extract various types of data from the systemd journal in different formats. It also explored managing journal files and how to add entries to the log from commands and scripts.
 

 The systemd journal system provides a significant amount of metadata and context for entries compared to the old syslogd program. This additional data and the context available from the other journal entries around the time of an incident can help the sysadmin locate and resolve problems much faster than having to search multiple syslog files.
 

 In my opinion, the journalctl command meets the Unix philosophy that programs should do one thing and do it well. The only thing journalctl does is extract data from the journal and provide many options for selecting and formatting that data. At about 85K, it is not very big. Of course, that does not include shared libraries, but those are, by definition, shared with other programs.
 

 You should now have enough information to use the systemd journal more effectively. If you would like to know more than what I covered here, look in the man pages for journalctl and systemd-cat .
 
Resources
 

 There is a great deal of information about systemd available on the internet, but much is terse, obtuse, or even misleading. In addition to the resources mentioned in this article, the following webpages offer more detailed and reliable information about systemd startup. This list has grown since I started this series of articles to reflect the research I have done.
 
 
DigitalOcean has a very good article about journalctl and how to view and manipulate systemd logs.
 
The Fedora Project has a good, practical guide to systemd. It has pretty much everything you need to know in order to configure, manage, and maintain a Fedora computer using systemd.
 
The Fedora Project also has a good cheat sheet that cross-references the old SystemV commands to comparable systemd ones.
 
Red Hat documentation contains a good description of the Unit file structure as well as other important information. 
 
For detailed technical information about systemd and the reasons for creating it, check out Freedesktop.org's description of systemd.
 
Linux.com's "More systemd fun" offers more advanced systemd information and tips.
 
 

 There is also a series of deeply technical articles for Linux sysadmins by Lennart Poettering, the designer and primary developer of systemd. These articles were written between April 2010 and September 2011, but they are just as relevant now as they were then. Much of everything else good that has been written about systemd and its ecosystem is based on these papers.
 
 
Rethinking PID 1
 
systemd for Administrators, Part I
 
systemd for Administrators, Part II
 
systemd for Administrators, Part III
 
systemd for Administrators, Part IV
 
systemd for Administrators, Part V
 
systemd for Administrators, Part VI
 
systemd for Administrators, Part VII
 
systemd for Administrators, Part VIII
 
systemd for Administrators, Part IX
 
systemd for Administrators, Part X
 
systemd for Administrators, Part XI