Logs

- [Instructor] Computers are always doing stuff, and luckily for us, they keep a record of what's going on. The system's log files keep detailed information about individual actions, the operating system, and other services running within it are taking. These records are kept in various log files, usually in the /var/log directory. The system log, located at /var/log/syslog on Ubuntu and other Debian systems and /var/log/messages on Red Hat systems, keep track of the actions that the system takes and provides a place where software can also write information. Some software maintains its own log files which is convenient for isolating chatty programs and to keep output related to one particular app separate from the rest of the logging information.

First, let's take a look at the system log. Here on Ubuntu, as I mentioned, it's at /var/log/syslog, and we can browse through it, page by page, with less. A log entry will have the date and time that the item was added to the log, the host name of the system, the program name and process ID, and a description from that program of what's being logged. Items in the log are kept in chronological order.

Let's take a look at an application log now. Maybe the log for the package manager. You can see that this looks a little different. It only has the date and time, a status, and a package name and version. We need less context here because we know where the information is coming from, as opposed to messages coming in from all kinds of places, as is the case with a system log. In the log folder, there's a file called auth.log, which contains information about activities that required authorization, instances when users tried to use the superuser privileges, and when there are changes to the user and groups files, and so on.

Log files are stored as text files, so we can use the usual text tools to work with them. Right now, we're using less to look at this log, and we can move up and down with the arrows and move up and down by a whole screen with F and B, for forward and backward. We can search within this file by typing slash and then a search term. Pressing Enter will find the next match of the term, and we can search again by typing slash and Enter. I can isolate lines of the file that contain a particular term by typing ampersand and a search term.

This will show me only the lines containing that term. To exit this mode, I'll type ampersand again, or we can grep these files, because they're text. Let's look for the term, dhcp, in syslog. All right, grep "dhcp" /var/log/syslog. It's often easier to search these files for something you know rather than scrolling through and hunting for it, though it can be interesting to read through these files sequentially, too, in order to get a sense of what the system's doing.

Logging on the system is handled by a program called rsyslogd or rsyslog. Software passes messages to it, and then it puts them down in the log file. Rsyslog is configured in the rsyslog.conf file inside etc. There, we can describe how we want it to work. We can write information into the syslog ourselves with the logger command. This is useful for making troubleshooting notes, and they can be called from a script to write information to the file. To write a message, I'll write, logger "hello from the command line" and then I'll tail the syslog with tail /var/log/syslog, and there's my message.

Rsyslog can also be configured to send logs to a remote machine or to accept them from another machine, which is helpful to centralize your logs. I won't configure that in this episode, but if you'd like to see how to do it, check out our courses on remote logging here at LinkedIn Learning. Every so often, rsyslog rotates the logs, or it chops them and archives the old parts. These files appear with a log file name and a number for the most recent log and with a number and the extension .gz, for gzip, for even older ones. Text generally compresses very well, so these archived logs don't take up a whole lot of space, and they can be kept around for a long time.

The retention and rollover policies for these log archives can be configured in the logrotate.conf file in the /etc/logrotate.d/ folder. To read the archived log files, we can use programs like a zless and zcat. So, to peek at the first few lines of an old log, we can just run zcat on the log file and pipe it into head. We don't need to manually decompress these files to work with them. So even though they're compressed, we can use them just like regular, plain text logs. Logs can look cryptic, but they follow a particular pattern, and once you get used to reading them, you'll gain an understand of what your system is doing.