Linux Admin - Log Management

Systemd has changed the way system logging is managed for CentOS Linux. Instead of every daemon on the system placing logs into inspanidual locations than using tools such as tail or grep as the primary way of sorting and filtering log entries, journald has brought a single point of administration to analyzing system logs.

The main components behind systemd logging are: journal, jounralctl, and journald.conf

journald is the main logging daemon and is configured by editing journald.conf while journalctl is used to analyze events logged by journald.

Events logged by journald include − kernel events, user processes, and daemon services.

Set the Correct System Time Zone

Before using journalctl, we need to make sure our system time is set to the correct time. To do this, we want to use timedatectl.

Let s check the current system time.

[root@centos rdc]# timedatectl status 
Local time: Mon 2017-03-20 00:14:49 MDT 
Universal time: Mon 2017-03-20 06:14:49 UTC 
RTC time: Mon 2017-03-20 06:14:49 
Time zone: America/Denver (MDT, -0600) 
NTP enabled: yes 
NTP synchronized: yes 
RTC in local TZ: no 
DST active: yes 
Last DST change: DST began at 
              Sun 2017-03-12 01:59:59 MST 
              Sun 2017-03-12 03:00:00 MDT 
Next DST change: DST ends (the clock jumps one hour backwards) at 
              Sun 2017-11-05 01:59:59 MDT 
              Sun 2017-11-05 01:00:00 MST
              
[root@centos rdc]#

Currently, the system is correct to the local time zone. If your system is not, let s set the correct time zone. After changing the settings, CentOS will automatically calculate the time zone offset from the current time zone, adjusting the system clock right away.

Let s pst all the time zones with timedatectl −

[root@centos rdc]# timedatectl pst-timezones 
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau

That is the contended output from timedatectl pst-timezones. To find a specific local time-zone, the grep command can be used −

[root@centos rdc]# timedatectl pst-timezones | grep -i "america/New_York" 
America/New_York
[root@centos rdc]#

The label used by CentOS is usually Country/Region with an underscore instead of space (New_York versus "New York").

Now let s set our time zone −

[root@centos rdc]# timedatectl set-timezone "America/New_York"

[root@centos rdc]# date 
Mon Mar 20 02:28:44 EDT 2017

[root@centos rdc]#

Your system clock should automatically adjust the time.

Use journalctl to Analyze Logs

Common command pne switches when using journalctl −

Examine Boot Logs

First, we will examine and configure the boot logs in CentOS Linux. The first thing you will notice is that CentOS, by default, doesn t store boot logging that is persistent across reboots.

To check boot logs per reboot instance, we can issue the following command −

[root@centos rdc]# journalctl --pst-boots 
-4 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-3 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-2 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-1 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
 0 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:07:36 MDT

[root@centos rdc]#  

After rebooting the system, we can see another entry.

[root@centos rdc]# journalctl --pst-boots 
-5 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-4 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-3 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-2 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
-1 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:09:57 MDT
 0 aa6aaf0f0f0d4fcf924e17849593d972 Mon 2017-03-20 01:10:07 MDT—Mon 2017-03-20 01:12:44 MDT
 
[root@centos rdc]#

Now, let s examine the last boot logging instance −

root@centos rdc]# journalctl -b -5 
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Mon 2017-03-20 01:20:27 MDT. --
Mar 19 22:01:57 localhost.localdomain systemd-journal[97]: Runtime journal is using 8.0M 
(max allowed 108.4M
Mar 19 22:01:57 localhost.localdomain kernel: Initiapzing cgroup subsys cpuset
Mar 19 22:01:57 localhost.localdomain kernel: Initiapzing cgroup subsys cpu
Mar 19 22:01:57 localhost.localdomain kernel: Initiapzing cgroup subsys cpuacct
Mar 19 22:01:57 localhost.localdomain kernel: Linux version 3.10.0514.6.2.el7.x86_64 
(builder@kbuilder.dev.
Mar 19 22:01:57 localhost.localdomain kernel: Command pne: 
BOOT_IMAGE=/vmpnuz-3.10.0-514.6.2.el7.x86_64 ro
Mar 19 22:01:57 localhost.localdomain kernel: Disabled fast string operations
Mar 19 22:01:57 localhost.localdomain kernel: e820: BIOS-provided physical RAM map:

Above is the condensed output from our last boot. We could also refer back to a boot log from hours, days, weeks, months, and even years. However, by default CentOS doesn t store persistent boot logs. To enable persistently storing boot logs, we need to make a few configuration changes −

Make central storage points for boot logs

Give proper permissions to a new log folder

Configure journald.conf for persistent logging

Configure Boot Location for Persistent Boot Logs

The initial place journald will want to store persistent boot logs is /var/log/journal. Since this doesn t exist by default, let s create it −

[root@centos rdc]# mkdir /var/log/journal

Now, let s give the directory proper permissions journald daemon access −

systemd-tmpfiles --create --prefix /var/log/journal

Finally, let s tell journald it should store persistent boot logs. In vim or your favorite text editor, open /etc/systemd/jounrald.conf".

# See journald.conf(5) for details.  
[Journal]=Storage=peristent

The pne we are concerned with is, Storage=. First remove the comment #, then change to Storage = persistent as depicted above. Save and reboot your CentOS system and take care that there should be multiple entries when running journalctl pst-boots.

Note − A constantly changing machine-id pke that from a VPS provider can cause journald to fail at storing persistent boot logs. There are many workarounds for such a scenario. It is best to peruse the current fixes posted to CentOS Admin forums, than follow the trusted advice from those who have found plausible VPS workarounds.

To examine a specific boot log, we simply need to get each offset using journald --pst-boots the offset with the -b switch. So to check the second boot log we d use −

journalctl -b -2

The default for -b with no boot log offset specified will always be the current boot log after the last reboot.

Analyze Logs by Log Type

Events from journald are numbered and categorized into 7 separate types −

0 - emerg   :: System is unusable 
1 - alert   :: Action must be taken immediatly    
2 - crit    :: Action is advised to be taken immediatly 
3 - err     :: Error effecting functionapty of apppcation 
4 - warning :: Usually means a common issue that can affect security or usilbity 
5 - info    :: logged informtation for common operations 
6 - debug   :: usually disabled by default to troubleshoot functionapty

Hence, if we want to see all warnings the following command can be issued via journalctl −

[root@centos rdc]# journalctl -p 4
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Wed 2017-03-22 22:33:42 MDT. --
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: RSDP 00000000000f6a10 00024
(v02 PTLTD )
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: XSDT 0000000095eea65b 0005C
(v01 INTEL  440BX    06040000 VMW  01
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACP 0000000095efee73 000F4
(v04 INTEL  440BX    06040000 PTL  00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: DSDT 0000000095eec749 1272A
(v01 PTLTD  Custom   06040000 MSFT 03
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACS 0000000095efffc0 00040
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: BOOT 0000000095eec721 00028
(v01 PTLTD  $SBFTBL$ 06040000  LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: APIC 0000000095eeb8bd 00742
(v01 PTLTD  ? APIC   06040000  LTP 00 
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: MCFG 0000000095eeb881 0003C
(v01 PTLTD  $PCITBL$ 06040000  LTP 00 
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: SRAT 0000000095eea757 008A8
(v02 VMWARE MEMPLUG  06040000 VMW  00 
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: HPET 0000000095eea71f 00038
(v01 VMWARE VMW HPET 06040000 VMW  00 
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: WAET 0000000095eea6f7 00028
(v01 VMWARE VMW WAET 06040000 VMW  00 
Mar 19 22:01:57 localhost.localdomain kernel: Zone ranges: 
Mar 19 22:01:57 localhost.localdomain kernel:   DMA      [mem 0x000010000x00ffffff] 
Mar 19 22:01:57 localhost.localdomain kernel:   DMA32    [mem 0x010000000xffffffff] 
Mar 19 22:01:57 localhost.localdomain kernel:   Normal   empty 
Mar 19 22:01:57 localhost.localdomain kernel: Movable zone start for each node 
Mar 19 22:01:57 localhost.localdomain kernel: Early memory node ranges 
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x000010000x0009dfff] 
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x001000000x95edffff] 
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x95f000000x95ffffff] 
Mar 19 22:01:57 localhost.localdomain kernel: Built 1 zonepsts in Node order,
mobipty grouping on.  Total pages: 60 
Mar 19 22:01:57 localhost.localdomain kernel: Popcy zone: DMA32 
Mar 19 22:01:57 localhost.localdomain kernel: ENERGY_PERF_BIAS: Set to
 normal , was  performance 

The above shows all warnings for the past 4 days on the system.

The new way of viewing and perusing logs with systemd does take pttle practice and research to become famipar with. However, with different output formats and particular notice to making all packaged daemon logs universal, it is worth embracing. journald offers great flexibipty and efficiency over traditional log analysis methods.