Skip to main content

Services and automating tasks

Lab Setup​

  • We will be using a virtual machine in the faculty's cloud.
  • When creating a virtual machine in the Launch Instance window:
    • Name your VM using the following convention: scgc_lab<no>_<username>, where <no> is the lab number and <username> is your institutional account.
    • Select Boot from image in Instance Boot Source section
    • Select SCGC Template in Image Name section
    • Select the m1.medium flavor.
  • The username for connecting to the VM is student.

systemd services​

Most common Linux distribution now ship with the systemd software suite to manage a services and other system components. The use of systemd provides a uniform configuration interface across multiple Linux distributions, and allows configuring service prerequisites and various environment parameters.

We can inspect the status of a service using systemctl status servicename.

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 20XX-02-30 00:00:00 UTC; 33min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 674 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 677 (sshd)
      Tasks: 1 (limit: 1696)
     Memory: 4.9M
     CGroup: /system.slice/ssh.service
             └─677 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Feb 30 00:00:00 scgc-services systemd[1]: Starting OpenBSD Secure Shell server...
Feb 30 00:00:00 scgc-services sshd[677]: Server listening on 0.0.0.0 port 22.
Feb 30 00:00:00 scgc-services sshd[677]: Server listening on :: port 22.
Feb 30 00:00:00 scgc-services systemd[1]: Started OpenBSD Secure Shell server.

In the service's status command's output we can see a number of useful stats about the service:

  • whether the service is enabled (i.e., it is automatically started when the system boots) - the enabled; parameter in the line starting with Loaded;
  • the time the service has started and how long it has been running;
  • what processes and how many are in the service's control group;
  • the last few lines output by the service to its output or standard error.

Changing the state of services​

You can start and stop a service using systemctl start servicename and systemctl stop servicename, respectively.

Stop the SSH service

Try stopping the ssh service. What do you observe? Try connecting to the virtual machine again. What happens if you reboot the virtual machine?

Services can be enabled or disabled using systemctl enable servicename and systemctl disable servicename.

Install nginx and toggle enabled state

Install the nginx service using apt and observe its status. Toggle its enabled state (if disabled, enable it; if enabled, disable it). How does the toggling affect its current running state?

warning

Be careful when disabling services. Some services, like ssh are critical. If disabled completely, you may become unable to connect to the system even if you reboot it.

tip

Look for enable UNIT in man systemctl to find out how you can immediately affect the service's running state when enabling or disabling it.

Tuning service files​

systemd services are configured using files under /lib/systemd/system or /etc/systemd/system. We can use systemctl cat servicename to see the configuration of a particular systemd unit.

$ sudo systemctl cat nginx
# /lib/systemd/system/nginx.service
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

The service configuration file contains various parameters that define generic service information (e.g., the Description and Documentation fields in the [Unit] section), dependencies on other services (e.g., the After and WantedBy fields in the [Unit] and [Install] sections), as well as service-specific parameters (e.g., where the service's PID should be stored, how systemd should keep track of processes, execution steps, and so on).

Override service configuration options

systemd can configure a number of parameters for the process. For example, the maximum number of open file can be configured by setting the LimitNOFILE (see man systemd.exec for more details) parameter in the [Service] section of the process.

Use systemctl edit nginx to edit nginx's configuration parameters and set the maximum number of open files to a really low number (e.g., 5).

You can define just the section name and the parameters you want to override; you don't need to copy all the configuration in the original service file.

tip

After configuring the parameter, you should see the following when running systemctl cat nginx (note that an overrides file has been created under /etc/systemd for the service).

# /etc/systemd/system/nginx.service.d/override.conf
[Service]
LimitNOFILE=5
Restart nginx with overridden settings

Try restarting the nginx service and observe what happens.

Note that this parameter is usually used to increase the maximum number of open files, not reduce it - this is just an example to demonstrate the parameter.

journald​

Logs captured by systemd from the services are kept in the system journal. The logs kept in the journal can be easily queried using the journalctl command.

In its most basic form, the journalctl command displays logs from the beginning of this system boot. We can use the -b flag followed by a number to get the logs associated with a specific previous boot. For example, -b -1 for the logs of the previous boot.

info

Logs are not persisted by the default configuration. We must edit the /etc/systemd/journald.conf file and set the Storage parameter to persistent in order to force the journal to create the appropriate permanent files to store the logs.

Edit the file and restart the systemd-journald service. Reboot the system a couple of times before inspecting the logs for each boot cycle.

tip

Persistent logs are especially important to inspect what may have caused a system crash in a previous boot cycle.

When trying to monitor service errors, the following parameters are particularly useful:

  • -e moves to the end of the log immediately (we do not usually need to inspect old logs, and this option can be used to skip to the latest logs);
  • -x adds explanations for some messages from a message catalog;
  • -u unitname only shows messages associated with a specific systemd unit (e.g., service);
  • -t syslogtag displays messages with a specific syslog tag associated with them (i.e., the text after the hostname in the logs, next to a number - for example nginx in this line: Feb 30 00:00:00 scgc-services nginx[XXXXX];
  • -n number only shows the most recent number logs;
  • -f makes the query follow updates to the logs, so new lines are displayed immediately.
Follow logs to debug errors in real time

Use the correct flags to show the logs for the nginx service and automatically update the displayed information.

From a different terminal update the /etc/nginx/nginx.conf file and break the configuration (e.g., remove a ; character somewhere), and then restart the service. Observe the logged information in the first terminal.

Task automation using cron​

Some tasks must be performed periodically for various purposes - creating backups of configuration files, sending emails, cleaning up redundant information, etc..

There are multiple ways to configure repeating tasks, but the most common is using cron. Cron has a simple syntax for configuring tasks, with five fields describing when the task should run and the remaining parameters define the command that should run. To edit the list of tasks for the current user, use crontab -e. The command has the same format as it would in a terminal; for example the following line could be used to backup the important directory every minute:

* * * * * umask 0077; tar zcvf "/home/student/backups/$(date +"\%F-\%H-\%M-\%S").tar.gz" /home/student/important
info

The percent (%) sign is a special character and must be escaped using backslashes.

Create a task that runs every two minutes

Create a task that counts how many regular files there are in the /usr/share directory every two minutes. Use the find command to list the regular files and the wc command to count how many lines are printed by find (i.e., this will give you the number of files).

Now, you may be wondering what happens to the output. If you look in the journal for the syslog tag CRON you may see some lines that show this warning:

(CRON) info (No MTA installed, discarding output)

By default cron gathers the process' standard output and error and emails them to the user. However, since the system does not have any mail services installed, the output is discarded automatically.

Configure postfix with local storage

Install the postfix package and configure it to only store the messages locally.

After the scheduled tasks execute, check the /var/log/student file to see the logged information.

info

A mail is only created if the scheduled tasks output the data to the standard output and error. You can explicitly redirect the output and error streams to a file using 2>&1 >>/some/file or completely discard them using 2>&1 >/dev/null, which will disable the creation of notification emails.

Automating tasks using systemd timers​

systemd timers are another option to automate recurring tasks, as well as triggering tasks relative to other events (e.g., system boot time).

By default a timer triggers a unit with the same name as them - for example, a service named servicename.service is automatically triggered by servicename.timer.

The list of timers available on the system can be inspected using systemctl list-timers. The output of the command displays the name of the timer, when the last activation of the timer happened and when the next activation is scheduled, as well as the name of the service that the timer activates.

In this section we will write a service that will periodically update the system's message of the day (the text that is displayed when connecting to the system).

note

By default the system displays a dynamic MOTD that is automatically generated when connecting. To make the output more clear we will disable it.

Edit the /etc/pam.d/sshd file and comment the following line by prepending a pound sign (#) before it, or removing it entirely:

session    optional     pam_motd.so  motd=/run/motd.dynamic

After updating the PAM file, connect to the virtual machine from a different terminal and observe how the output has changed.

Now that we have a clean canvas, let's implement a task that runs the following script every 2 minutes.

Place the following script in /usr/local/sbin/update-motd.sh. When it runs it updates the /etc/motd file, which will be displayed when connecting to the system.

#!/bin/sh

set -eu -o pipefail

fetch_info() {
        echo "System information at $(date)"
        echo

        echo "Network interfaces:"
        ip -br a s | grep -E '^(eth|enp|virbr)'
        echo

        echo "Disk usage:"
        df -h /
        echo
}

echo "Fetching system information"
fetch_info > /etc/motd
echo "Finished successfully"
Create a service file and timer that run the script

Using the motd-news.service and motd-news.timer service and timer files as reference, create a new service and timer unit files in /etc/systemd/system, for a service named motd-update.

The script executed by the service does not have to have the executable bit set on it. To run it, we can explicitly invoke the bash interpretor using the following execution parameter:

ExecStart=/bin/bash /usr/local/sbin/update-motd.sh

The timer should trigger the service to run every two minutes. Use man systemd.time and read the CALENDAR EVENTS section to find how the OnCalendar field should be set.

Make sure to remove the RandomizedDelaySec=12h parameter from the timer to not add random delays in the execution.

Enable only the timer. The service does not have to be enabled, as it will be automatically started by the timer periodically.

tip

Confirm that you have the correctly set up the timer by listing the registered timers and checking the timer's status or the associated journal entry.

If you have misconfigured the timer, you can edit it using the following command (the timer must be restarted after being edited):

$ sudo systemctl edit --full motd-update.timer

If you have configured the timer and service correctly, you should be able to see lines in the service's logs that look like this:

Feb 30 00:00:00 scgc-services bash[XXXXX]: Fetching system information
Feb 30 00:00:00 scgc-services bash[XXXXX]: Finished successfully

This is because by default systemd uses the name of the executable that is invoked by ExecStart as the system log tag.

Set an expected syslog identifier for the service

Change the service's system log tag to motd-update using the SyslogIdentifier parameter in the [Service] section.

Use the --full option when editing the service file, since this change is relevant to the service as a whole, and not an override.

If the change is applied correctly, the logs from the next service run should look like this:

Feb 30 00:00:00 scgc-services update-motd[XXXXX]: Fetching system information
Feb 30 00:00:00 scgc-services update-motd[XXXXX]: Finished successfully