Skip to main content

Configuration management

Setup

We will be using a virtual machine in the faculty's cloud.

When creating a virtual machine in the Launch Instance window:

  • Name your VM using the following convention: scgc_lab<no>_<username>, where <no> is the lab number and <username> is your institutional account.
  • Select Boot from image in Instance Boot Source section
  • Select SCGC Template in Image Name section
  • Select the m1.large flavor.

In the base virtual machine:

  • Download the laboratory archive from here in the work directory. Use: wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-conf-manage.zip to download the archive.

  • Extract the archive. The .qcow2 files will be used to start virtual machines using the runvm.sh script.

  • Start the virtual machines using bash runvm.sh.

  • The username for connecting to the nested VMs is student and the password is student.

$ # change the working dir
$ cd ~/work
$ # download the archive
$ wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-conf-manage.zip
$ unzip lab-conf-manage.zip
$ # start VMs; it may take a while
$ bash runvm.sh
$ # check if the VMs booted
$ virsh net-dhcp-leases labvms

Puppet resources

Puppet is a configuration management tool. In order to describe the required configurations, Puppet uses its own declarative language. Puppet can manage both Linux and Windows systems.

All Puppet tasks will be done inside the virtual machine that the runvm.sh scripts starts.

To use Puppet you must first install the puppet package on your system.

[student@lab-conf-manage ~]$ sudo dnf install puppet

A resource is an abstraction for most entities and operations that can be performed on a system. As an example, the state of a service (running / stopped) is defined in Puppet as a resource.

Use the puppet resource service command to see the system services from Puppet's perspective.

[student@lab-conf-manage ~]$ puppet resource service
service { 'NetworkManager-dispatcher.service':
ensure => 'stopped',
enable => 'true',
provider => 'systemd',
}
service { 'NetworkManager-wait-online.service':
ensure => 'running',
enable => 'true',
provider => 'systemd',
}
service { 'NetworkManager.service':
ensure => 'running',
enable => 'true',
provider => 'systemd',
}
service { 'auditd.service':
ensure => 'running',
enable => 'true',
provider => 'systemd',
}
service { 'autovt@.service':
ensure => 'stopped',
enable => 'true',
provider => 'systemd',
}
service { 'chrony-dnssrv@.service':
ensure => 'stopped',
enable => 'true',
provider => 'systemd',
}
[...]
Command syntax

The previous command's syntax is:

  • puppet command - used to access most of Puppet's features;
  • resource subcommand - interact with available Puppet resources;
  • service parameter - type of resources to show.

Resource structure

To show the resource representing the root user account, we can use the following command:

[student@lab-conf-manage ~]$ puppet resource user root
user { 'root':
ensure => 'present',
comment => 'root',
gid => 0,
home => '/root',
provider => 'useradd',
shell => '/bin/bash',
uid => 0,
}
Resource command output

The output contains the following information:

  • the type of the resource - in this example user;
  • the name of the resource - root;
  • attributes of the resource - ensure, comment, gid, and so on;
  • the values of each attribute.

The output above represents a resource declaration of the user resource.

Types of resources

Besides services and users, Puppet implements other types of resources. To see a list of the available resources, you can use the following command:

[student@lab-conf-manage ~]$ puppet describe --list
These are the types known to puppet:
augeas - Apply a change or an array of changes to the ...
cron - Installs and manages cron jobs
exec - Executes external commands
file - Manages files, including their content, owner ...
filebucket - A repository for storing and retrieving file ...
group - Manage groups
[...]

Managing resources

We can create new resources using the puppet resource command. The generic syntax is:

puppet resource type name attr1=val1 attr2=val2

We want to create the user worker so that:

  • the user's home directory is /home/worker;
  • the user's default shell is /bin/sh.

We can create the user with the following command:

[student@lab-conf-manage ~]$ sudo puppet resource user worker ensure=present shell=/bin/sh home=/home/worker
Notice: /User[worker]/ensure: created
user { 'worker':
ensure => 'present',
home => '/home/worker',
provider => 'useradd',
shell => '/bin/sh',
}
Verify resource creation

Open the /etc/passwd file and check the entry that was created for the user.

In order to remove a resource, the ensure parameter must be set to absent.

As an example, to remove the worker user that we have previously created, we can use the following command:

[student@lab-conf-manage ~]$ sudo puppet resource user worker ensure=absent
Notice: /User[worker]/ensure: removed
user { 'worker':
ensure => 'absent',
provider => 'useradd',
}
Verify resource removal

Check the /etc/passwd file again and confirm that the user has been removed.

Puppet manifests

Even though we can create, modify and remove resources from the command line, using the puppet resource command, this is not a scalable approach and not appropriate for complex scenarios.

As mentioned above, Puppet is a configuration management tool, which means that we describe what resources must (or must not) exist on a system. Consequently, the preferred approach would be:

  • creating a resource file that describes the resources on the system;
  • applying the described modifications using Puppet.

Files containing Puppet resource declarations are called manifests and usually have the .pp file extension.

Puppet is a pull-type configuration system

The examples presented here are meant for educational purposes, and are better suited for testing configurations. Puppet usually has a client-server architecture, where the server keeps track of all manifest files, and the clients connect to the server to retrieve (pull) them. This enables Puppet to scale up and configure an entire cluster of servers.

Creating a manifest

We are going to write a manifest file that describe a file resource. The file is going to have the following properties:

  • path: /tmp/my_file;
  • access mode: 0640;
  • content: This file was created using Puppet.

We will save the following configuration in a file called my_file_manif.pp:

file {'my_file':
path => '/tmp/my_file',
ensure => present,
mode => '0640',
content => 'This file was created using Puppet',
}

Applying a manifest

To apply a manifest we can use the puppet apply command:

[student@lab-conf-manage ~]$ puppet apply my_file_manif.pp
Notice: Compiled catalog for lab-conf-manage in environment production in 0.03 seconds
Notice: /Stage[main]/Main/File[my_file]/ensure: defined content as '{md5}73a33bf2b9a33847ac15cca5bf18c1c7'
Notice: Applied catalog in 0.05 seconds
Verify resource creation

Check that the file has been created and the content and access permissions are correct.

Try applying the manifest one more time without modifying the file.

[student@lab-conf-manage ~]$ puppet apply my_file_manif.pp
Notice: Compiled catalog for lab-conf-manage in environment production in 0.02 seconds
Notice: Applied catalog in 0.04 seconds
File resource state

Notice that if the file is already in the required state, Puppet will avoid performing any action. This is because repeated application of the manifest should be idempotent, meaning that it should not create any changes in the system unless required.

Manifest application after file changes

Change the file's permissions and contents. Apply the manifest after each change and notice the output given by Puppet. How does it keep track of content changes?

States (ensure)

The ensure attribute usually specifies if the resource:

  • must exist (ensure => present);
  • must not exist (ensure => absent).

Some types of resources define additional states for the attribute. For example, file resources can also have the following values for ensure:

  • directory;
  • link;
  • file.

Define a manifest that creates a symbolic link to /tmp/my_file. Use the Puppet documentation for the file type resource for more details.

tip

The target attribute must also be set when the ensure => link attribute is set.

Authorized SSH key

Create a new manifest file and define a ssh_authorized_key type resource. The resource must allow the student user on the host to authenticate as the student user on the virtual machine using an SSH key.

tip

You must generate an SSH key on the host system, and copy its public key to the manifest file. We recommend using ED25519 keys for their shorter length.

Resource dependency

A puppet manifest can contain declarations for multiple resources, but the order in which they are applied is not strictly enforced.

There are some situations when we have to make sure that a resource is applied before another (e.g. a package is installed before starting a service). In such cases, we must define resource dependencies.

The Puppet relationships docs provide a more detailed overview of how resource relationships work. We will provide some examples in the following sections.

Before / require

We can modify the previously created manifest and add an exec resource:

exec {'echoer':
command => ['/bin/echo', 'Test hello!'],
logoutput => true,
require => File['my_file'],
}

file {'my_file':
path => '/tmp/my_file',
ensure => present,
mode => '0640',
content => 'This file was created using Puppet',
}

The exec type resource defines a shell command that must be executed. This type of resource is not usually used since other resource types provide built-in checks to verify if resource application is required. The logoutput attribute specifies that the output of the command must also be passed to Puppet's notification output.

Observe the execution order

Apply the manifest above and observe the order in which the resources are evaluated. Change the managed file and see how this affects the execution.

The before attribute can be used to create an equivalent syntax:

exec {'echoer':
command => ['/bin/echo', 'Test hello!'],
logoutput => true,
}

file {'my_file':
path => '/tmp/my_file',
ensure => present,
mode => '0640',
content => 'This file was created using Puppet',
before => Exec['echoer'],
}

Notify / subscribe

Some resources require running an action that has the effect of a "refresh" (e.g. a service that needs to reload its configuration). The behaviour of some resources (e.g. service type resource) may differ depending on whether the resource is refreshed or not.

If in addition to resource dependencies we need to refresh a second resource when the first one is changed, we can either:

  • use notify instead of before;
  • use subscribe instead of require.

As the example above, we can update the configuration above to use the notify attribute instead before.

tip

By default, the exec resource will always run. To adapt it to work with the notify system, add the refresh or refreshonly attributes accordingly.

Behaviour change

Notice how the behaviour changes between before and notify.

Equivalent require syntax

Instead of explicitly using before / require or notify / subscribe, we can use the -> or ~> operators:

file {'my_file':
path => '/tmp/my_file',
ensure => present,
mode => '0640',
content => 'This file was created using Puppet',
}
->
exec {'echoer':
command => ['/bin/echo', 'Test hello!'],
logoutput => true,
}

caution

Be careful when typing ~> after adding a new line (pressing Enter). The sequence <Enter>~. (i.e. pressing Enter and then typing tilde (~) and then period (.)) is used to kill the SSH session.

Design patterns: package / file / service

In many situations, Puppet is used to make sure that a system service is installed, has the appropriate configuration and started. Such a scenario can be implemented using three resources:

  • package;
  • file;
  • service.

The first two components have a before / require relation, while the last two have a notify / subscribe relation.

Create the following manifest which implements this design pattern for the SSH service, and the apply the manifest:

package {'openssh-server':
ensure => present,
}
->
file {'/etc/ssh/sshd_config':
ensure => file,
mode => '600',
source => '/home/student/config-samples/sshd_config',
}
~>
service {'sshd':
ensure => running,
enable => true,
}

Modify various parts of the package / file / service triplet and reapply the manifest. For example:

  • uninstall the package;
  • change the configuration file;
  • stop the service.
Create a configuration manifest to configure nginx

Create a package / file / service manifest for the nginx service. You can use the example configuration file from /home/student/config-samples.

On Alma Linux, the package is called nginx and the configuration file must is placed in /etc/nginx/nginx.conf.

Also, add a custom index page that you place under /usr/share/nginx/html/index.html. The contents of the file must be set using the content attribute of the file type resource.

Variables and conditional statements

For an easier configuration parametrization, Puppet allows using variables and conditionals.

Variables

Similar to PHP, variables in Puppet start with a $ symbol - i.e. to access a variable we use the $variable syntax, both for assignment and referencing.

We can change the manifest file for my_file, defining the contents as a variable:

$content = 'This file was created using Puppet'

file {'my_file':
path => '/tmp/my_file',
ensure => present,
mode => '0640',
content => $content,
}

Facts

In addition to user-defined variables, Puppet uses a powerful data gathering engine, called facter to retrieve information about the system. The variables defined by the facter are called facts and can be accessed as system variables in manifests.

We can use the facter command to see a list of all variables that are defined by the data engine.

[student@lab-conf-manage ~]$ facter
augeas => {
version => "1.12.0"
}
disks => {
sda => {
model => "QEMU HARDDISK",
size => "8.00 GiB",
size_bytes => 8589934592,
vendor => "ATA"
}
}
dmi => {
bios => {
release_date => "04/01/2014",
vendor => "SeaBIOS",
version => "1.13.0-1ubuntu1.1"
},
[...]

Conditionals

An example of when using system variables (facts) is useful is when we need to take decisions based on the value of some of them.

The following manifest ensures that the nginx service:

  • is stopped if the system is a physical machine;
  • is started if the system is a virtual machine.

The decision is taken based on the value of the $is_virtual variable.

if $is_virtual {
service {'nginx':
ensure => running,
enable => true,
require => Package['nginx'],
}
} else {
service {'nginx':
ensure => stopped,
enable => false
}
}

Manifest to install and configure nginx

First, uninstall the nginx server from the virtual machine. Write a manifest that will:

  • install the nginx server;
  • configure nginx to serve a custom index page.

Use the case conditional statement to create the index files in the appropriate directories.

Facts dictionary

Depending on the version of Puppet that you use, the facts may change their position in the facts dictionary. Review the output of the facter command to locate the variables that you need.

Configuration parameters

You are not expected to change the service's configuration files this time around. You only need to define a custom index.html file in the appropriate location and access permissions.

Depending on the operating system's family (e.g., RedHat or Debian), the default nginx configuration files differ as follows:

  • on RedHat systems, the root HTML directory is /usr/share/nginx/html and the service runs as the nginx user;
  • on Debian systems, the root HTML directory is /var/www/html and the service runs as the www-data user.

Ansible

Ansible is a configuration management and provisioning tool, similar to Puppet. It uses SSH to connect to servers and run the configuration modules. As opposed to Puppet, which normally uses a pull configuration (the client systems connect to the Puppet server to fetch the configurations), Ansible works in a push configuration (the configuration server has remote (e.g. SSH) access to all the systems that it configures).

An advantage of Ansible is that it does not require a specific service daemon to be installed on the target systems before the provisioning is performed. It operates using various Python scripts for remote Linux systems, or Powershell scripts for Windows systems.

We are going to install and configure Ansible on the host system:

student@lab-conf-manage-host:~$ sudo apt update
student@lab-conf-manage-host:~$ sudo apt install ansible

Make sure to check the version of Ansible that you have installed. Later versions of Ansible may rename modules and add functionality, so you must ensure that you use the correct documentation for your version.

student@lab-conf-manage-host:~$ ansible --version
ansible 2.9.6
config file = /etc/ansible/ansible.cfg
[...]

Configuring the inventory

Ansible uses inventory files where we can define lists of hosts and host groups that are possible targets for our configurations.

Inventory files can also contain variable definitions on a per-host or per-group basis. An example inventory file is created as /etc/ansible/hosts, initially containing only comments.

The inventory file can be in any of many formats; the example inventory's format is INI, but we will use the YAML format instead. Create the inventory.yml file with the following contents:

all:
hosts:
localhost:
ansible_connection: local
children:
remotes:
hosts:
vm:
ansible_host: 192.168.100.91
ansible_user: student
Inventory file structure

In the inventory file above we have defined two hosts:

  • localhost - the local system. Since the ansible_connection is set to local, Ansible will run the configuration modules (scripts) locally, without attempting to connect to the system through SSH;
  • vm - the remote system. The ansible_host variable must be set to define the IP or hostname Ansible should connect to if it differs from the entry's name (vm). The ansible_user defines the user Ansible will connect as. Note that Ansible will also parse the SSH config file for custom hosts and SSH parameters; as such, you can have an entry for vm that defines both parameters in the SSH configuration file, and they could be omitted here.

Testing host availability

We can test our connection to the hosts in the inventory file. To run the ping module (tests that we can connect to the system) against localhost we can run the following command:

student@lab-conf-manage-host:~$ ansible -i inventory.yml -m ping localhost
localhost | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}

When running against the remotes host group, make sure that you can connect to all hosts in the group (in this case, just the 192.168.100.91 virtual machine) using SSH keys. The following command runs the ping module on all servers in the remotes group.

student@lab-conf-manage-host:~$ ansible -i inventory.yml -m ping remotes
vm | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"ping": "pong"
}

In both cases, we can see that Ansible's output is a JSON that reports some information about the task that was executed.

Ansible facter

Ansible has a fact gathering engine similar to Puppet. To gather facts about the remote hosts we can use the setup module.

student@lab-conf-manage-host:~$ ansible -i inventory.yml -m setup vm
vm | SUCCESS => {
"ansible_facts": {
"ansible_all_ipv4_addresses": [
"192.168.100.91"
],
[...]
"ansible_fips": false,
"ansible_form_factor": "Other",
"ansible_fqdn": "localhost",
"ansible_hostname": "lab-conf-manage",
"ansible_hostnqn": "",
"ansible_interfaces": [
"lo",
"eth0"
],
"ansible_is_chroot": false,
"ansible_iscsi_iqn": "",
[...]

Unless explicitly disabled, this module always runs at the beginning of playbooks to gather facts about the system.

Two-factor authentication for SSH

We plan to enable the use of two-factor authentication for SSH using the Google authenticator PAM plugin. To do this, we need to first create a Google authenticator configuration file on the host. To install the required packages and create a Google authenticator configuration file with sensible defaults we can use the following commands:

student@lab-conf-manage-host:~$ sudo apt install libpam-google-authenticator qrencode
student@lab-conf-manage-host:~$ google-authenticator -t -d -Q UTF8 -r 3 -R 30 -w 3 -e 5
Authenticator parameters

The parameters given to the google-authenticator command have the following meanings:

  • -t - create a time-based (TOTP) validation token;
  • -d - disallow code reuse (i.e., if the code has been used for authentication, it cannot be used again);
  • -Q UTF print the QR code using UTF8 characters. If it is not displayed properly, you can use ANSI or NONE (in the latter case you will need to manually copy the secret code);
  • -r 3 -R 30 - limit login attempts to 3 every 30 seconds;
  • -w 3 - use a window of 3 codes (the previous code, current one, next one) for authentication; since codes are time-based, this allows a maximum of 30 seconds of time skew between the code generator (phone) and server;
  • -e 5 - generate 5 emergency codes that can be used if the generator is not available.
Scan the authenticator code

When the configuration file is created, the secret code should be displayed both as a QR code and a secret key. Scan the QR code or manually enter the secret key into your preferred TOTP generator application (e.g. Google Authenticator).

Prepare configuration files

We will begin by copying the existing configuration files from the virtual machine to the host. The files must be updated to enable the Google authenticator PAM plugin.

student@lab-conf-manage-host:~$ mkdir files
# Copy the `/etc/pam.d/sshd` and `/etc/ssh/sshd_config` files from the VM to the `files` directory.
# The files should be named `pam_sshd` and `sshd_config`.
Update the configuration files

After the files have been copied, the following configuration lines must be added / updated:

  • in sshd_config:
AuthorizedKeysFile      .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication yes
AuthenticationMethods publickey keyboard-interactive
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
  • in pam_sshd:
#%PAM-1.0
auth substack password-auth
auth required pam_google_authenticator.so
auth include postlogin

The changes above enable SSH keyboard interactive authentication (e.g., password, OTP codes) as an alternative to the public key authentication, and enable the challenge responses required for multiple authentication steps. In the PAM configuration the Google authenticator plugin is added after password authentication, so the user's password is required before the OTP code.

If the public key should have been requested instead of a password (so users need to present a valid SSH key, and the OTP code afterwards), the password-auth subtrack would instead be completely disabled, and the AuthenticationMethods in the SSH configuration would be set to publickey,keyboard-interactive.

Alternative approaches

We could also use a module like lineinfile to update parts of the configuration files directly on the remote system, but this approach does not guarantee that the content that we do not update is correct.

Create configuration playbook

An Ansible playbook is a collection of tasks that will be executed on the systems that we want to configure. We will configure the system to use the configuration files that we have prepared earlier.

We can use variables to set configuration parameters inside playbooks. The syntax used to expand all variables - including the ones created by the setup module is {{variable}}. For example, {{ansible_hostname}} expands to the hostname of the system Ansible currently configures (e.g. lab-conf-manage for the virtual machine).

Secret information

Ansible will sometimes need to use or configure confidential information. Such information must be added to encrypted vaults that are kept safe. Ansible can interact with various vault formats, but for this example we will use the default Ansible vault tool.

The Google authenticator information is confidential information, since anyone with access to the file would be able to extract the secret code from it, removing any benefit of implementing multi-factor authentication.

We first need to create a vault for the secret values:

  • the sudo password required to run privileged commands (when connecting as a non-root account);
  • the Google authenticator configuration file.

In preparation, we will convert the Google authenticator configuration file we have created earlier as a template to base64. This step is not required, since you can also add the entire file as-is to the vault. However, since its contents do not have to be human-readable in the configuration, converting it makes the configuration simpler. Run the command below and copy the output.

student@lab-conf-manage-host:~$ base64 ~/.google_authenticator | paste -s -d ''

To create the vault file we will use the ansible-vault command as shown below. The command will ask for an encryption password for the vault file; this password will be requested when the vault is read or edited, so make sure to use a password that you will remember (you can also keep it in a separate file that is usually not accessible to the Ansible server).

student@lab-conf-manage-host:~$ ansible-vault create files/vault
New Vault password:
Confirm New Vault password:

After running the command, an editor process will open. Enter the configuration parameters and then close the editor:

Authenticator file variable

Set the authenticator_file variable to the output of the base64 command above. Do not use the base64_encoded_google_... string.

ansible_become_password: student
authenticator_file: base64_encoded_google_authenticator_configuration_file
Confirm the vault is encrypted

After closing the editor, the configuration file is automatically encrypted. Confirm that the file is encrypted and you can view it using the ansible-vault command again.

student@lab-conf-manage-host:~$ cat files/vault
$ANSIBLE_VAULT;1.1;AES256
[...]
student@lab-conf-manage-host:~$ ansible-vault view files/vault
Vault password:
ansible_become_password: student
authenticator_file: base64_encoded_google_authenticator_configuration_file

After configuring the vault, we have completed all prerequisites and can now create the playbook file (as sshd.yml):

---
- hosts: remotes

tasks:
# Include secret variables defined in the vault file
- name: Include vault file
include_vars: "files/vault"

# Install the SSH server and make sure it is at the latest version using the package
# manager identified by ansible
- name: Ensure sshd is at the latest version
package:
name: openssh-server
state: latest
become: yes

# Install the Google authenticator PAM plugin and make sure it is at the latest version
- name: Ensure google-authenticator is at the latest version
package:
name: google-authenticator
state: latest
become: yes

# Copy the Google authenticator configuration file
# The file must have '0600' permissions. The default location is inside the user's home directory
# Ansible can run filters to parse variables;
# in this case, we use the `b64decode` filter to decode the file from base64
- name: Copy Google Authenticator config file
copy:
content: "{{ authenticator_file | b64decode }}"
dest: /home/student/.google_authenticator
mode: 0600
owner: student
group: student

# Overwrite the sshd configuration file. Make sure the challenge response setting
# is enabled, and keyboard-interactive is a valid authentication method
- name: Write the sshd configuration file
template:
src: files/sshd_config
dest: /etc/ssh/sshd_config
owner: root
group: root
mode: 0600
become: yes
notify:
- restart sshd

# Overwrite the PAM configuration file. Make sure that authentication through
# google authenticator is required
- name: Write the PAM configuration file
template:
src: files/pam_sshd
dest: /etc/pam.d/sshd
owner: root
group: root
mode: 0644
become: yes
notify:
- restart sshd

# Handlers that are invoked when the configuration files change.
# Handlers are run at the end of the playbook, regardless of how many times they are triggered.
handlers:
# Restart the SSH daemon
- name: restart sshd
service:
name: sshd
state: restarted
become: yes

To execute the playbook, use the command below. It will automatically run all tasks in the tasks section on the servers defined in the remotes group. You should see a similar output.

student@lab-conf-manage-host:~$ ansible-playbook -i inventory.yml --ask-vault-pass sshd.yml
Vault password:

PLAY [remotes] ***************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************
ok: [vm]

TASK [Include vault file] ****************************************************************************************
ok: [vm]

TASK [Ensure sshd is at the latest version] **********************************************************************
ok: [vm]

TASK [Ensure google-authenticator is at the latest version] ******************************************************
changed: [vm]

TASK [Copy Google Authenticator config file] *********************************************************************
changed: [vm]

TASK [Write the sshd configuration file] *************************************************************************
changed: [vm]

TASK [Write the PAM configuration file] **************************************************************************
changed: [vm]

RUNNING HANDLER [restart sshd] ***********************************************************************************
changed: [vm]

PLAY RECAP *******************************************************************************************************
vm : ok=8 changed=5 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

You can test the connection to the virtual machine using the following command to force using the keyboard-interactive authentication method:

student@lab-conf-manage-host:~$ ssh 192.168.100.91 -o PreferredAuthentications=keyboard-interactive
Unable to login

You may notice that you are not able to login on the remote system if it is running a RedHat operating system (e.g., Alma Linux). Inspect the journal logs on the system to identify the problem:

[student@lab-conf-manage ~]$ sudo journalctl -xe -u sshd

If the error message looks like Failed to create tempfile "/home/student/.google_authenticator~XXXXXX": Permission denied, it is because SELinux does not allow the SSH service to create a temporary file in the user's home directory (check the error using audit2allow -aw).

In this case, you must fix the issue:

  • change the location of the Google authenticator secrets file to /home/student/.ssh/.google_authenticator in the Ansible playbook;
  • check the Google authenticator man page and find how you can set an alternate location for the secrets file;
  • update the location of the secrets file in the PAM configuration;
  • run the Ansible playbook again;
  • test the connection.