Routing
Lab Setup
- We will be using a virtual machine in the faculty's cloud.
- When creating a virtual machine in the Launch Instance window:
- Name your VM using the following convention:
scgc_lab<no>_<username>, where<no>is the lab number and<username>is your institutional account. - Select Boot from image in Instance Boot Source section
- Select SCGC Template in Image Name section
- Select a flavor that is at least m1.large.
- Name your VM using the following convention:
- The username for connecting to the VM is
student. - For the following exercises, the resources can be found in the laboratory archive:
$ cd work/
$ wget https://repository.grid.pub.ro/cs/scgc/laboratoare/lab-routing.zip
$ unzip lab-cert.zip
$ bash runvm.sh
Also run the following command in you current shell, or close and reopen the shell.
$ source ~/.bashrc
Topology
For all the exercises we will use the above topology.
Configuring and deleting IP addresses
We first want to ensure connectivity between the host and red stations. In this tutorial we will use the iproute suite on Linux to perform common layer 3 (IP addressing) configurations.
We will configure one IP address from the 10.10.10.0/24 class on the link interfaces between the host station and the red station. That is, between host(usernet) (the usernet interface on the host station) and red(red-eth0) (the red-eth0 interface on the red station).
On the usernet interface on the host station we will configure the IP address 10.10.10.1 with the mask 255.255.255.0 (/24 in the prefixed form):
root@host:~# ip address add 10.10.10.1/24 dev usernet
Note that the iproute2 suite (ie the ip utility) uses the mask in prefixed format: /24.
Immediately after a network configuration run a command to validate the configuration. In our case it is the command to display the level 3 (Network) configuration, that is, the IP address:
root@host:~# ip address show dev usernet
47: usernet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 4e:1b:b8:d9:14:bb brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 scope global usernet
On the red-eth0 interface on the red station we will configure the IP address 10.10.10.2 with the mask 255.255.255.0 (/24 in the prefixed form):
root@host:~# go red
[...]
root@red:~# ip address add 10.10.10.2/24 dev red-eth0
root@red:~# ip address show dev red-eth0
46: red-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:16:3e:8e:84:21 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.2/24 scope global eth0
inet6 fe80::216:3eff:fe8e:8421/64 scope link
valid_lft forever preferred_lft forever
Likewise, after a network configuration, I ran the validation command, in this case ip address.
To test the connectivity between the host and red stations we use the ping command:
root@host:~# ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
^C
--- 10.10.10.2 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1007ms
After a few seconds, stop the ping command using the Ctrl+c key combination.
Notice that there is no connectivity between the two stations: packets are completely lost (100% packet loss). The reason is that we have not enabled the interfaces, we have only done layer 3 configurations.
Trace the layer 2 configuration of the interfaces using the ip link command:
root@host:~# ip link show dev usernet
10: usernet: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 3e:03:f0:76:76:ab brd ff:ff:ff:ff:ff:ff
Notice that the interface is not active at layer 2 (Data Link). To activate the interface use the command:
root@host:~# ip link set dev usernet up
Look again at the Layer 2 (Data Link) configuration of the usernet interface and notice that it is now partially UP (UP and DOWN also appear in the command output):
root@host:~# ip link show dev usernet
10: usernet: <NO_CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 3e:03:f0:76:76:ab brd ff:ff:ff:ff:ff:ff
Test connectivity again using the ping command. Still no connectivity. This and the fact that DOWN also appeared in the output of the previous command is due to the fact that we did not enable the red-eth0 interface on the red station. The red-eth0 interface on the red station is the one connected to the usernet interface on the host station; both must be enabled to have an active connection.
On the red station check the layer 2 configuration of the red-eth0 interface on the red. Note that it is DOWN and enable the interface if applicable using the command:
root@red:~# ip link set dev red-eth0 up
Verify that the interface is now active using the command:
root@red:~# ip link show dev red-eth0
Use the ping command to retest connectivity between the host and red stations.
We want to go back to the original configuration. For this run a form command:
# ip address flush dev INTERFACE
where INTERFACE is the usernet interface on the host station, respectively red-eth0 on the red station. Ensure that no more IP addresses are configured on the interfaces using a command of the form:
# ip address show dev INTERFACE
where INTERFACE is the interface usernet interface on the host station, respectively red-eth0 on the red station.
Configuring IP addresses
We want to have connectivity between the red station and the host station, respectively between the green station and the host station. For this, we will configure IP addresses on each. For the conenctivity between host and red, you already configured it in the previous exercise.
Configure one 10.10.20.0/24 class IP address each on the link between the green station and the host station (ie link green(green-eth0) ↔ host(usernet)) and test the connectivity.
Consider checking the Data Link level using the ip link command and enabling the interfaces as needed.
IP Addressing and Routing
We want to achieve connectivity between the red and green stations as well. Since the two stations are on different local networks, we will need to configure the host station as the default gateway on each station.
To add the default gateway on the red station, use the commands:
root@host:~# go red
[...]
root@red:~# ip route add default via 10.10.10.1
After configuration (adding the route), we validate the configuration with a specific command. In this case we trace the routing table using the command:
root@red:~# ip route show
default via 10.10.10.1 dev red-eth0
10.10.10.0/24 dev red-eth0 proto kernel scope link src 10.10.10.2
The IP address 10.10.10.1 represents the IP address of the usernet interface on the host station.
Enter the green station and execute the commands similarly.
Test connectivity by pinging between the green and red stations. Notice that it doesn't work. The reason why there is no connectivity is represented by the fact that the host station does not have routing enabled (it does not send the packets coming from one interface to another interface).
To enable routing on the host station, run the command:
root@host:~# sysctl -w net.ipv4.ip_forward=1
To validate the routing enable configuration, we run the command:
root@host:~# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
Test the connectivity between red and green again and see if it works.
Start the ping command from the red station to the green station.
Open a new terminal and run on the host station the command:
root@host:~# tcpdump -n -i usernet
listening on Ethernet, link-type EN10MB (Ethernet), capture size 65535 bytes
18:46:48.783576 IP red.local > 10.10.20.2: ICMP echo request, id 434, seq 163, length 64
18:46:48.783622 IP 10.10.20.2 > red.local: ICMP echo reply, id 434, seq 163, length 64
Notice the ICMP echo request/reply packets that pass through the host station (or in other words the host station routes them).
Complete connectivity setup
We want to ensure full connectivity between all stations in the topology. The blue station must be configured accordingly.
Configure class 10.10.30.0/24 IP addresses on the link between the host station and the blue station (ie between host(usernet) and blue(blue-eth0)).
Be sure to check the layer 2 link using the ip link command.
Test the connectivity between the host station and the blue station.
On the blue station, configure the host station as the default gateway, to allow connectivity to the other stations.
On the blue station, use as default gateway the IP address on the usernet interface of the host station.
Test connectivity between any two stations.
ARP table
ARP (Address Resolution Protocol) is a protocol that makes each operating system internal the association between the IP addresses and the MAC addresses of the stations with which it communicates. Often the stations know the IP addresses of their neighbors but do not know the MAC addresses; the ARP protocol populates a system-local ARP table with the necessary entries. The ARP protocol is run by default by the operating system when communicating with a station whose MAC address is unknown.
We aim to trace the ARP table of a Linux system.
On the host station monitor the ARP table using the command:
root@host:~# ip neighbor show
[...]
The table may be empty (no recent communication) or have some entries (most recent communication) or entries marked STALE (unreliable entries).
To populate the ARP table initiate communication with the other stations using the ping command:
root@host:~# ping -c 1 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
64 bytes from 10.10.10.2: icmp_req=1 ttl=64 time=0.033 ms
[...]
root@host:~# ping -c 1 10.10.20.2
PING 10.10.20.2 (10.10.20.2) 56(84) bytes of data.
64 bytes from 10.10.20.2: icmp_req=1 ttl=64 time=0.036 ms
[...]
root@host:~# ping -c 1 10.10.30.2
PING 10.10.30.2 (10.10.30.2) 56(84) bytes of data.
64 bytes from 10.10.30.2: icmp_req=1 ttl=64 time=0.080 ms
[...]
Watch the ARP table again:
root@host:~# ip neighbor show
10.10.10.2 dev usernet laddr 00:16:3e:8e:84:21 REACHABLE
10.10.20.2 dev usernet laddr 00:16:3e:d1:b2:95 REACHABLE
10.10.30.2 dev usernet laddr 00:16:3e:32:0f:ae REACHABLE
10.9.0.1 dev eth0 laddr 0a:00:27:00:00:00 REACHABLE
Notice that each station (red, green and blue) has a corresponding entry in the ARP table marked REACHABLE (valid entry). The additional input is the communication of the virtual machine (host station) with the fep.grid.pub.ro system.
Perform the above steps for each of the red, green and blue stations:
- Follow the ARP table.
- Initiate communication with the other stations to populate the ARP table.
- Watch the ARP table again.
Notice that in the ARP table of each of the red, green and blue stations there is an ARP entry corresponding to the host station. This happens because the communications go through the default gateway (ie through the host station) and each station only needs to know the MAC address of the gateway.
Troubleshoot IP address configuration problem
Run the prepare script with the new ex6 argument:
root@host:~# start_lab ex6
After running the script the red station was restarted and the configurations were redone. You will need to reconnect to the red station using the command:
root@host:~# go red
The script configures the IP address 10.10.7.1 on the usernet interface of the host station and the IP address 10.10.7.2 on the red-eth0 interface of the red station. To display the IP configuration on the two interfaces use the commands:
root@host:~# ip address show usernet
47: usernet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 4e:1b:b8:d9:14:bb brd ff:ff:ff:ff:ff:ff
inet 10.10.7.1/32 scope global usernet
root@red:~# ip address show red-eth0
46: red-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:3e:8e:84:21 brd ff:ff:ff:ff:ff:ff
inet 10.10.7.2/24 scope global red-eth0
inet6 fe80::216:3eff:fe8e:8421/64 scope link
valid_lft forever preferred_lft forever
Use the ping command to test connectivity between the two IP addresses (10.10.7.1 and 10.10.7.2) on the two stations. Notice that there is no connectivity.
To troubleshoot this problem, we follow the routing table of each station:
root@red:~# ip r s
default via 10.10.7.1 dev red-eth0
10.10.7.0/24 dev red-eth0 proto kernel scope link src 10.10.7.2
root@host:~# ip r s
default via 10.9.0.1 dev eth0
10.9.0.0/16 dev eth0 proto kernel scope link src 10.9.3.210
169.254.169.254 via 10.9.0.100 dev eth0
192.168.2.0/24 dev usernet proto kernel scope link src 192.168.2.1
192.168.3.0/24 dev usernet proto kernel scope link src 192.168.3.1
Notice that the relevant route (10.10.7.0/24) does not appear on the host in the routing table. Either the interface is disabled or the configuration is wrong.
Look carefully at the level 3 information:
root@host:~# ip address show usernet
47: usernet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 4e:1b:b8:d9:14:bb brd ff:ff:ff:ff:ff:ff
inet 10.10.7.1/32 scope global usernet
root@red:~# ip address show red-eth0
46: red-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:16:3e:8e:84:21 brd ff:ff:ff:ff:ff:ff
inet 10.10.7.2/24 scope global red-eth0
inet6 fe80::216:3eff:fe8e:8421/64 scope link
valid_lft forever preferred_lft forever
We can see that the interfaces are up (UP). But one of these addresses (10.10.7.1) has the /32 mask. This means that they cannot network with each other and also explains the absence of the relevant route from the routing table.
Fixing this error is done by adding the IP address with the correct mask 10.10.7.1/24 on the usernet interface on the host. Verify that you have connectivity between host and red.
Remember to delete the wrong address using the command ip address delete 10.10.7.1/32 dev usernet. If you don't delete the wrong address you will have 2 IP addresses on the interface, one with a /24 mask and one with a /32 mask.
A relatively common mistake in IP address configuration in Linux is to omit the netmask. Be careful not to omit the mask when configuring IP addresses on interfaces in Linux.
Listing a station's routing table is among the first steps in troubleshooting a connectivity problem.
Troubleshooting connectivity problem
We aim to troubleshoot a connectivity issue. To "generate" the problem run the preparation script with the new argument ex7:
root@host:~# start_lab ex7
For troubleshooting, the recommended first step is to display the routing table. The routing table will help you troubleshoot if certain entries are missing or misconfigured.
Check connectivity between all stations in the topology. Notice that there is no connectivity between any station and the blue station. Troubleshoot connectivity issues to the blue station on the host station.
Identify and resolve issues.
IPv6
We want to provide IPv6 connectivity between host station and red. In this tutorial we will use the iproute suite from Linux to perform the necessary configurations. Use the -6 parameter to make IPv6 related settings.
We will configure one 2201::/64 class IP address on the link interfaces between the host station and the red station. That is, between host(usernet) (the usernet interface on the host station) and red(red-eth0) (the red-eth0 interface on the red station).
On the usernet interface on the host station we will configure the IP address 2201::1/64:
root@host:~# ip -6 address add 2201::1/64 dev usernet
Immediately after a network configuration run a command to validate the configuration. In our case it is the IPv6 address display command:
root@host:~# ip -6 address show dev usernet
47: usernet: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2201::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::215:5dff:fe5b:a38e/64 scope link
valid_lft forever preferred_lft forever
On the red-eth0 interface on the red station we will configure the IP address 2201::2/64:
Configure on the host-blue link IPv6 addresses from the 2202::/64 network and on the host-green link IPv6 addresses from the 2203::/64 network.
Enable routing for IPv6 on the host station:
root@host:~# sysctl -w net.ipv6.conf.all.forwarding=1
You also need to add default routes on red, green and blue to the host.
Check connectivity between containers using the ping command.
Persistent setup
We want that when resetting a station the level 3 configurations (IP addresses) are preserved. The configurations we have made so far are temporary and are lost when the station is restarted. In Linux, the persistence of configurations is achieved by placing them in specific text files, each distribution (eg: Debian, RedHat) having its own configuration mode.
To prepare the exercise, run the prepare script:
root@host:~# start_lab ex9
root@host:~# ip address flush dev usernet
Persistently perform the configuration from exercise IPv6 for the host. The Linux distribution used in the lab is Ubuntu.
For details on how to make persistent configurations on Ubuntu systems, see this page. You will make a static configuration.
After you have made the necessary configurations for red, you can test by running the reboot_vms command on the host:
root@host:~# reboot_vms
For information related to enabling routing, see this page.