Skip to main content

Load balancing

Setup​

We will be using a virtual machine in the faculty's cloud.

When creating a virtual machine in the Launch Instance window:

  • Name your VM using the following convention: scgc_lab<no>_<username>, where <no> is the lab number and <username> is your institutional account.
  • Select Boot from image in Instance Boot Source section
  • Select SCGC Template in Image Name section
  • Select the m1.large flavor.

In the base virtual machine:

  • Download the laboratory archive from here in the work directory. Use: wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-lb.zip to download the archive.

  • Extract the archive. The .qcow2 files will be used to start virtual machines using the runvm.sh script.

  • Start the virtual machines using bash runvm.sh.

  • The username for connecting to the nested VMs is student and the password is student.

$ # change the working dir
$ cd ~/work
$ # download the archive
$ wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-lb.zip
$ unzip lab-lb.zip
$ # start VMs; it may take a while
$ bash runvm.sh
$ # check if the VMs booted
$ virsh net-dhcp-leases labvms

Load balance topology​

TopologyTopology

The machines in the topology (the three KVM machines and the host) have the following roles:

  • load-balancer is the director. It handles the load balancing for the other two virtual machines (the real servers);
  • real-server-1 and real-server-2 are the real servers;
  • the host is the client that requests data from the servers.

Linux Virtual Server (LVS)​

Linux Virtual Server (LVS) is an advanced, open source load balancing solution. It is also integrated in the Linux kernel. The IP virtual server that we will present in this section is an extension of LVS and is a transport-layer load balancer. This type of load balancer is also known as a Layer 4 LAN switch, since it does not use application layer information.

In LVS terminology, the load balancing server is called the Director, whereas the machines that respond to requests are named Real Servers (RS). A client will access the service exclusively using the address of the director.

LVS has three operation modes:

  • LVS-NAT - the director performs NAT for the real servers. It is useful when the RSs do not have public IP addresses and are part of the same network. It does not scale well because the entire traffic must go through the director;
  • LVS-TUN - the director tunnels the client packets and the real servers communicate directly with the client. It scales better than LVS-NAT because only the client traffic goes through the director, but it requires implementing a tunneling configuration on the real servers;
  • LVS-DR - the director routes the packets towards the real servers without tunneling and the real servers communicate with the client directly. It removes the tunneling prerequisite, but both the director and the real servers must have their network interfaces in the same LAN. Additionally, the real servers must be able to answer requests sent to the director, since the request destination addresses are not overwritten.

LVS-DR (direct routing)​

An HTTP service will be load balanced. The nginx web server is already installed on the real servers. The director will split the client requests to the two real servers.

To manage the IP virtual server, you must first install the ipvsadm package on the director system.

student@load-balancer:~$ sudo apt update
student@load-balancer:~$ sudo apt install ipvsadm

We will first configure a virtual address on the director machine. We will add the 192.168.100.251/24 address on the eth0:1 sub-interface on the load-balancer machine.

student@load-balancer:~$ sudo ip addr add dev eth0 192.168.100.251/24 label eth0:1

Configure IPVS in DR mode​

We will configure the HTTP service as a virtual service. To do this, we need to specify the virtual address, port and transport protocol used (TCP, in our case). The following command creates a service (the -A parameter) that handles requests coming to the 192.168.100.251 IP address on TCP port 80:

student@load-balancer:~$ sudo ipvsadm -A -t 192.168.100.251:80

Once the virtual service has been configured, we can also add the real servers:

student@load-balancer:~$ sudo ipvsadm -a -t 192.168.100.251:80 -r 192.168.100.72:80 -g
student@load-balancer:~$ sudo ipvsadm -a -t 192.168.100.251:80 -r 192.168.100.73:80 -g

In the commands above we add the real servers (with IP addresses 192.168.100.72 and 192.168.100.73) to the service that was created on 192.168.100.251:80. Routing will be done in direct route (DR) mode (set using the -g parameter).

We also need to make the real servers answer to requests that are meant for the virtual address. There are two ways of achieving this:

  • configure the virtual address on a loopback interface on the real servers. A disadvantage of doing this is that the real servers might respond to ARP messages that are actually meant for the director. This issue is known as the ARP problem;
  • configure an iptables rule that causes the real server to accept packets even though the virtual address is not configured on any interface. We will use this approach.
student@real-server-1:~$ sudo iptables -t nat -A PREROUTING -d 192.168.100.251 -j REDIRECT
student@real-server-2:~$ sudo iptables -t nat -A PREROUTING -d 192.168.100.251 -j REDIRECT

The virtual service is now completely configured and we can use it.

Test the service

To test the functionality, fetch the HTTP page from 192.168.100.251. Download the page repeatedly and notice how the service behaves.

We can use tcpdump to start a capture of the packets that traverse the virbr-labs bridge on the host, with link-level header inspection (the -e parameter). You could optionally add the -A parameter to also print the packet bodies.

student@lab-lb-host:~/work$ sudo tcpdump -i virbr-labs -e src port 80 or dst port 80
Note IP and MAC addresses

Check the IP and MAC addresses of the packets:

  • sent from the client towards the director;
  • sent from the director towards the real servers;
  • replied by the real servers.

The configuration of the virtual server can be inspected using the -l parameter:

student@load-balancer:~$ sudo ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.100.251:http wlc
  -> lab-lb-2:http                Route   1      0          0
  -> lab-lb-3:http                Route   1      0          0

A list of connections that are managed by the virtual server can be obtained using the -c parameter:

student@load-balancer:~$ sudo ipvsadm -l -c

Adding a connection threshold​

We can update the basic configuration to change the server's behaviour.

Edit the configuration to set a maximum number of connections redirected towards each real server. Set the limit to three for each server.

tip

The edit command is similar to the one that added the real servers, but the -a parameter is changed to -e. You must add the proper parameter to set the connection threshold.

Check how connections are handled

Start a large number of connections (e.g., using a for loop) and see how the servers respond to the requests.

For real servers with different hardware configurations, a different number of maximum connections can be configured. Alternatively, we can use different types of schedulers (e.g., weighted round-robin with different weights for each server).

Cleanup​

To delete the service, use the -D parameter:

student@load-balancer:~$ sudo ipvsadm -D -t 192.168.100.251:80

You must also delete the iptables rules on the real servers:

student@real-server-1:~$ sudo iptables -t nat -F
student@real-server-2:~$ sudo iptables -t nat -F

LVS-TUN (tunneling)​

For this task we will configure the IP virtual server in tunneling mode.

Configure IPVS in TUN mode​

Similar to the previous task, configure the virtual service on the director and add the real servers in tunneling mode.

tip

You must replace the -g parameter.

As mentioned before, when using the tunneling mode we must configure a tunnel interface on both real servers. The tunnel interface's IP address must be the same as the IP address of the director.

To create an IP-IP tunnel interface on the first real server you can use the following command:

student@real-server-1:~$ sudo ip tunnel add tun0 mode ipip local 192.168.100.72

Create a tunnel interface on the second real server. Make sure to set the correct local IP address. After creating the interfaces, add the director's IP address (192.168.100.251/24) on the tunnel interfaces and activate them on both servers.

warning

When adding the IP address to the IP-IP tunnel, make sure to set a metric of 32768 (or larger than the 100 used for the eth0 interface). Otherwise, the server will prefer using the tunnel for the 192.168.100.0/24 network and will become inaccessible.

If you have set the wrong metric, you can reboot the virtual machine using virsh reboot lab-vm-X. Alternatively, you can access its console using VNC and removing the tunnel.

Check how connections are handled

Check how connections are handled using tcpdump. Compare the connections handling to how connections were handled in direct routing mode.

Cleanup​

Remove the service on the director and the tunnel interfaces on the real servers.

student@real-server-1:~$ sudo ip tunnel del tun0
student@real-server-2:~$ sudo ip tunnel del tun0

HAProxy​

HAProxy is a service that can function as a high-availability load balancer and proxy. HAProxy can function as a TCP load balancer, meaning that it can proxy random TCP streams, such as SSH connections, but for the purposes of this lab, we are interested in configuring it as an HTTP proxy.

HAProxy as load balancer​

To use HAProxy, it must first be installed on the load balancer system:

student@load-balancer:~$ sudo apt update
student@load-balancer:~$ sudo apt install haproxy

Make sure to enable and start the HAProxy.

student@load-balancer:~$ sudo systemctl enable --now haproxy

We can configure HAProxy as a load balancer and reverse proxy for the two HTTP servers, real-server-1 and real-server-2. To do this, edit the /etc/haproxy/haproxy.cfg file and configure the server to listen for HTTP connections and use the two servers as the backends (append the following configuration at the end of the file):

frontend www
        bind *:80
        default_backend realservers

backend realservers
        mode http
        balance roundrobin
        server realserver-1 192.168.100.72:80
        server realserver-2 192.168.100.73:80
tip

Restart the HAProxy whenever you edit the settings file to reload the changes.

Check how connections are handled

Check how connections are handled using tcpdump. Compare the connections handling to how they were handled when using IP virtual server.

Health checks​

HAProxy can automatically detect when the backend servers are down and remove them from the pool of available servers to avoid sending requests to the servers when they are unavailable.

To add health checks, edit the configuration files and add the httpchk option in the backend section, and then enable checks on the backend servers. Set the check interval to 3 seconds.

tip

To enable health checks on the server, you must use the check setting, and configure the interval using the inter keyword.

caution

Since the backend servers only have a basic configuration that serves static pages (an index.html file), you cannot use the default health check method (OPTIONS). You will have to use the GET method instead. To change the used method, use the http-check send directive and configure it to use the GET.

A fully functional application written using a programming language / framework like PHP or NodeJS will likely handle the OPTIONS HTTP method without issues.

After adding the health checks, see how HAProxy behaves when stopping the nginx service on one of the real servers. Verify the service's logs and how new requests are directed through the load balancer.

HAProxy response caching​

HAProxy can cache server responses to reduce the load on the web servers. This is an important functionality since some files (e.g., CSS style pages, JavaScript script, static images) are the same for all users. Consequently such pages can be cached by the load balancer and served without requesting them from the backend.

Configure HAProxy caching​

The cache must first be created at the top level of the configuration file, and then used in the backend section. You can find more details in the cache section of the documentation.

We will configure a 256MB cache that will be used to store objects of a maximum of 20MB. Only responses that are smaller than the maximum size (and are cacheable according to the conditions defined in the documentation) will be added to the cache. Additionally, we configure the cache such that objects that are older than five seconds will expire. When an object's cache entry has expired, it will be requested again from the backend servers.

backend realservers
        [...]
        http-request cache-use main
        http-response cache-store main
        server [...]

cache main
        total-max-size 256
        max-object-size 20971520
        max-age 5
Observe responses when using a cache

Observe the behaviour of the load balancer after the cache has been enabled. Send multiple requests to the server and see how the responses change when using, or not using the cache. Keep sending requests for more than 10 seconds.

We can extract information about the cached data using the administration socket. To do this, we will use socat to connect to the socket and enter commands:

student@load-balancer:~$ echo 'show cache' | sudo socat - /run/haproxy/admin.sock

Blocking response caching​

The web server can decide that some responses should not be cached, or that the response must be re-validated by the client before reusing. We can use the Cache-Control HTTP header to change how the response is be cached.

For example, we can set the no-cache policy on the responses coming from the second web server. Edit the /etc/nginx/sites-available/default page on the real-server-2 server and add the following configuration in the server section.

location /index.html {
        add_header Cache-Control no-cache;
}

After editing the configuration, restart the nginx service. Observe what responses the client receives and the stats about HAProxy caches.

warning

On production servers it usually does not make sense to set the no-cache parameter on plain HTML files. However, some response categories, such as transitory messages used in authentication protocols should not be cached, since they should not be reusable, and caching them may leak confidential information.

Performance improvements using caching​

To get a better view of the load on the systems, we must first configure HAProxy to only use one of the backend servers. Remove the realserver-2 from the configuration and restart HAProxy. Afterwards, confirm that you only get replies from the first server.

We will use the httperf tool to test performance. Install the tool on the host system:

student@lab-lb-host:~$ sudo apt install httperf

To start testing using the index file, use the following command:

student@lab-lb-host:~$ httperf --server 192.168.100.251 --port 80 --num-conns 10000 --rate 1000 --timeout 5
Check the load on the system

Observe the load on the load balancer and the real server using htop. When the test is finished, see the numbers printed by httperf for Connection rate, Request rate, Net I/O.

Disable response caching and repeat the tests. See how the numbers change.

Securing connections with HTTPS​

HTTPS is a protocol that encrypts communication between the client and the server. This makes using HTTPS a requirement on all sites that require authentication, or handle any sort of confidential information. Anyone with access to the network can insert sniffing tools or implement various types of attacks that force traffic to go through their system; without the encryption provided by HTTPS they can view the raw packets and extract any transmitted information.

Currently, almost all sites implement HTTPS, by either using a paid certificate from a certification authority, or free certificates from authorities like Let's Encrypt.

For the purposes of this lab, since we do not have a (public) domain for our services, we will use locally signed certificates - either a self-signed certificate, or certificates signed by an internal certification authority. Review the generating a certificate section for more details.

Using HTTPS with HAProxy​

Before enabling HTTPS, create a self-signed certificate in the /etc/ssl/private directory (see the crt-base directive in the HAProxy configuration). For this example, we will assume that you will create the load-balancer.pem file as the certificate.

To enable HTTPS you must use the ssl setting and specify the certificate file using the crt setting on the bind directive. The default port for HTTPS is 443.

frontend www
        bind *:80
        bind *:443 ssl crt load-balancer.pem
        [...]
Using the proper key file

The crt directive expects all certificate information to be placed in the same file (i.e., the certificate, private key, issuer certificate chain, etc., must be concatenated in a larger *.pem file) and HAProxy will automatically extract the data from that file.

If the certificate file does not contain some of the expected information, it will attempt to use some pre-defined file extensions to look for it. For example, you could have the key in a separate file called load-balancer.pem.key.

Connecting to an HTTPS server

Both wget and curl support connecting to an HTTPS server and expect the server to use a certificate that has been signed by a trusted authority. Since we do not have a trusted certificate, you can pass the -k parameter to curl, or the --no-check-certificate parameter to wget.

Automatic HTTPS redirects​

A good practice is redirecting requests coming to the HTTP port (80) to use HTTPS. To do this, we can add the following line to the frontend section of the HAProxy configuration file:

frontend www
      [...]
      redirect scheme https if !{ ssl_fc }
Following redirects

To follow redirects, you can use the -L flag for curl. wget should automatically follow redirects and download the page.

Internal HTTPS servers​

Even when using HTTPS for client connections, establishing connections to the backend servers using plain HTTP implies that we trust our internal network. We can reduce the risk of having internal traffic intercepted (at the cost of computational resources required for encryption) by adding TLS encryption to all internal servers.

Begin by enabling HTTPS on the two backend servers. You will have to create certificates for both backend servers. You can choose to either create a self-signed certificate for each server, or create a CA on the host system that you will use to sign the certificates.

We will assume that the certificates are created as /etc/ssl/private/real-server.pem and the key is /etc/ssl/private/real-server.key. Add the following configuration in the /etc/nginx/sites-enables/ssl file:

server {
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;

        server_name _;

        ssl_certificate /etc/ssl/private/real-server.pem;
        ssl_certificate_key /etc/ssl/private/real-server.key;

        root /var/www/html;
        index index.html;

        location / {
                try_files $uri $uri/ =404;
        }
}

After the configuration has been added, restart the nginx service on both servers and check that you can send HTTPS requests to them.

At this point all that is left is configuring HAProxy to connect to the backend servers using HTTPS. You must set the correct port and enable the use of SSL when connecting.

tip

Since the backend servers do not use certificates that are signed by a trusted authority, you will have to let HAProxy know that it can trust them. Use the ca-file setting on the server directives to specify the certificate of the CA (or the certificate files in case of self-signed certificates) that HAProxy should trust.