General Remarks
In this guide, we will describe how to setup a Raspberry Pi Cluster using an Ansible playbook. In general, the setup will use one Raspberry Pi as a master node and the others as a compute node. The compute nodes and master node will be connected via a switch that should be plugged into the native Ethernet ports of the Pis. To ensure stability, one should stick to separate power supplies instead of PoE. The master node will also need a USB-Ethernet-Adapter for access to the general internet. This USB-Interface should be plugged into a USB 3.0 slot. The general internet traffic will also need a gateway, for which we will use another desktop or laptop machine. Your laptop will also need two ethernet interfaces, so probably you will need one-two additional Ethernet adapters (you could also route the outside traffic over your laptop wifi, but that's not as stable as ethernet). Thus, the chain of connection is:
Internet - {your local network} - Workstation/Laptop (10.0.1.1) - {Ethernet to Pi USB Interface, 10.0.1.0/24} - Pi Masternode (10.0.1.2, 10.0.0.1) - {Ethernet, 10.0.0.0/24} - Switch - Nodes
A more graphical depiction of the setup can be found in the paper. We will be using static IPs in this guide because we are using Ansible to configure them. You should be familiar with basic concepts of routing and Linux, i.e., you should know how to use `ssh`.
Setting up the Workstation Gateway
Please first note that these instructions are only tested on an Arch-based Linux distribution. Specifics for Debian-distributions may vary.
First of all, we need to find out the names of our network interfaces. On our laptop, we will have an outside interface (OINT for the rest of this guide) that is connected to the public internet. We also have an interface connected to the cluster (CINT for the rest of the guide). Please find out - e.g. using `ifconfig` - how your specific interfaces are called. You will probably need to enable the interface for the cluster first using `ip link set CINT up`.
First of all, we want to give us our IP adress in the `10.0.1.0/24` network. For that, run the command `ip addr add 10.0.1.1 dev CINT`. We also need to setup a route for this network using `ip route add 10.0.1.0/24 dev CINT proto kernel scope link src 10.0.1.1`. Lastly, we also will need a route into the cluster network. For that, run `ip route add 10.0.0.0/24 via 10.0.1.2 dev CINT`.
We now need to enable packet forwarding in the kernel. First, check your configuraton status using `sysctl -a | grep forward`. You can temporarily enable packet forwarding using the command `sysctl net.ipv4.ip_forward=1` or persistently enable it by editing the file `/etc/sysctl.d/30-ipforward.conf`:
net.ipv4.ip_forward=1
net.ipv6.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1
We now use `iptables` to enable NAT. Thus, run the following commands in your shell.
iptables -t nat -A POSTROUTING -o OINT -j MASQUERADE
iptables -A FORWARD -i OINT -o CINT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i CINT -o OINT -j ACCEPT
Checkout `iptables-persistent` if you want to make changes persistent. After saving the rules, it should somehow look like this:
# iptables -L -n -v
Chain INPUT (policy ACCEPT 1780 packets, 164K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
20 1520 ACCEPT all -- wlp4s0 enp0s20f0u1u3 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
20 1520 ACCEPT all -- enp0s20f0u1u3 wlp4s0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 871 packets, 104K bytes)
pkts bytes target prot opt in out source destination
Now do one final cross-check if all the routes are set up correctly using `ip route show`. It should show something like that:
default via 172.22.16.1 dev enp60s0u2 proto dhcp src 172.22.16.253 metric 204
10.0.0.0/24 via 10.0.1.2 dev enp0s20f0u1u3
10.0.1.0/24 dev enp0s20f0u1u3 proto kernel scope link src 10.0.1.1
Preparing the SD Cards
Now, we need to flash the SD cards for each Pi. First, download the latest Raspian Lite image. In order to enable ssh and set the static IP, we need to modify the `.img` image before flashing it. For each new compute node, please proceed according to the next steps and remember to change the IP.
First, we need to mount the image as a loopback device using `losetup -P /dev/loop0 raspbian.img`. Then, we can mount the boot partition using `mount /dev/loop0p1 /mnt/` to a mountpoint of your choice (`/mnt/` in this example). Now, we want to enable ssh using `touch /mnt/ssh` and unmount the boot partition using `umount /mnt/`.
We now mount the root partition using `mount /dev/loop0p2 /mnt/` and now edit the file `/mnt/etc/dhcpcd.conf`:
interface eth0
static ip_address=10.0.0.<compute node id>/24
static routers=10.0.0.1
static domain_name_servers=8.8.8.8
Please make sure that your `<compute node id>`s start with `2` instead of `1` because `1` will be our master node. We are done configuring the image, so now run `umount /mnt` and `losetup -d /dev/loop0` and flash the image to your SD card using ` dd if=raspbian.img of=/dev/<sd card device> bs=1M` (you can also use 4M as block size, but this is more error-prone). If you don't know what your sd card device is, try using `bashmount` from the AUR to find out (Debian users: Sorry, look for another tool).
Now, we need to setup our master pi SD card. For that, we proceed the same way as above, but this time, the dhcpcd.conf needs to be a little different as the master node is a member of both networks, as stated in the first section of this guide.
interface eth0
static ip_address=10.0.0.1/24
static domain_name_servers=8.8.8.8
nolink
interface eth1
static ip_address=10.0.1.2/24
static routers=10.0.1.1
static domain_name_servers=8.8.8.8
nolink
We assume your USB network interface on the master node will show up as `eth1` - which was the case for our ethernet adapter, but yours may be different. To continue the setup, try to ssh into your master node (both IPs `10.0.0.1` and `10.0.1.2` should work). From there, try to ping 8.8.8.8. If both of that works, congratulations! You did everything correct and can continue. Otherwise, cross-check the `iptables` and route settings.
Setting Up the Cluster
Install `ansible` on your workstation connected to the cluster. Now, we will make use of our Ansible playbook you can find here. Clone the repository in some directory of your choice. The basic idea of playbooks is idempotence. This means that no matter how often you will execute the playbook, the basic state of the nodes shouldn't change. You can e.g. run the playbook to keep the nodes in a updated state (it will run `apt full-upgrade` for you). All changes made to configuration files will be done again and not clash with their own results.
Now, we first need to make some configuration tweaks. In the `hosts.yml` file, you can configure the nodes of your cluster. Make sure to only place one node in the [master] section of the file. The `piname` option defines the hostname of the node. In the config.yml, you can configure several options:
- sshkey: Here, enter your personal SSH public key. Then, you'll be able to ssh pi@node0x into every node instead of using password authentication.
- usbinterface: If your USB network interface on the master Pi is not called eth1, you should change that here
- timezone: Change that to your local timezone
- ssh_key_filename: You'll need to generate a new public/private SSH key pair and place it in the playbook root directory. If these two files are not called id_rsa and id_rsa.pub, you'll need to enter the prefix here. Please do not forget to create this pair, otherwise Hadoop will not work.
Now, create a folder called native in the project root and copy your Hadoop native bindings into it. You can either compile these by yourself (see our guide here), download our ARM bindings from here or just use the offical bindings your download together with Hadoop here (these will just be ignored when running on ARM).
If you want to, you can modify the Hadoop settings (blocksize etc.) in `roles/raspberry_node/defaults/main.yml`. If you want to use Docker as well, make sure to change `raspberry_compute` to `raspberry_docker` in the `setup.yml`. For a structural overview of the different roles, read the README of the Github repository.
Now, just run the `run.sh` file and everything should be done automatically for you. Please note that especially on the first run, the setup will take some time (also depending on your internet speed). Please note that sometimes the playbook may either hang or fail. In this case, just run it again until everything works out (I don't really know why this happens - Ansible is just weird sometimes).
After that, you can visit `https://10.0.1.2/ganglia` and you should be able to see all nodes in the dashboard.
Cluster Usage
The monitoring is available under `https://node01/ganglia`. All hadoop commands need to be run under the `hadoop` user (`sudo su hadoop`). Please note that the NameNode is already formatted and a basic directory structure is already created. Checkout the hadoop Task in the raspberry_node role to see what is happening during setup.
You can reach the NameNode Web UI under `https://node01:9870`. The Yarn Webinterface can be reached via `https://node01:8088`.
Link Section
