426 lines
14 KiB
Org Mode
426 lines
14 KiB
Org Mode
# -*- ii: ii; -*-
|
|
#+TITLE: Raspberry pi kubernetes cluster guide
|
|
#+AUTHOR: James Blair
|
|
#+EMAIL: mail@jamesblair.net
|
|
#+DATE: 26th June 2021
|
|
|
|
|
|
This file serves as a complete step by step guide for creating a bare metal raspberry pi kubernetes cluster using [[https://k3s.io/][k3s]] from [[https://rancher.com/][Rancher]].
|
|
|
|
My goal for this build is to replace a server I currently run at home that hosts several workloads via Docker with a scalable k3s cluster.
|
|
|
|
Additionally in future I would like the cluster to be portable and operate via 3G-5G Cellular network and an array of batteries.
|
|
|
|
I chose k3s as it incredibly lightweight but still CNCF certified and production grade software that is optimised for resource constraints of raspberry pis.
|
|
|
|
|
|
|
|
|
|
* Hardware pre-requisites
|
|
|
|
** Cluster machines
|
|
|
|
For this guide I am using four [[https://www.pishop.us/product/raspberry-pi-4-model-b-4gb/][Raspberry Pi 4 4GB]] machines and one [[https://www.pbtech.co.nz/product/SEVRBP0267/Raspberry-Pi-4-Model-B-8GB-LPDDR4-Quad-Core-Cortex][Raspberry Pi 4 8GB]] for more memory hungry workloads.
|
|
|
|
|
|
** Boot media
|
|
|
|
This guide requires each Raspberry Pi to have a removable SD card or other removable boot media. I am using five 32GB SD Cards though any USB or SD card at least 8GB in size should work fine.
|
|
|
|
*Note:* Newer raspberry pi firmware can support USB or even NVME booting. If boot disk IO performance is the goal of your cluster you may want to explore this instead. Additionally network booting is also possible if you would prefer to avoid using SD Cards or other attached boot media.
|
|
|
|
|
|
** Power supply
|
|
|
|
For this cluster I am using power over ethernet using [[https://www.pbtech.co.nz/product/SEVRBP0184/Raspberry-Pi-Add-On-Board-Power-over-Ethernet-PoE][Pi POE Hat]] addon boards. This means each Pi does not need a separate cable for power supply so is much cleaner.
|
|
|
|
*Note:* A power over ethernet cable switch is required for this configration. I am using [[https://www.pbtech.co.nz/product/SWHNGR1208120/NETGEAR-ProSAFE-GS108PP-8-Port-Gigabit-Unmanaged-P][this 8 port gigabit Netgear switch]] which supports 120 watts for POE. Each Pi will require at least 15 watts.
|
|
|
|
|
|
* Step 1 - Prepare boot media for master
|
|
|
|
** Download sd card imaging utility
|
|
|
|
Our first step is to create the bootable SD Card with a minimal install of [[https://ubuntu.com/download/raspberry-pi][Ubuntu Server 20.04 x64]], which is a free and open source operating system based on [[https://www.debian.org/][Debian]].
|
|
|
|
To simplify the sd card creation process we can use the open source [[https://github.com/raspberrypi/rpi-imager][rpi-imager]] utility, the code snippet below will download the latest release:
|
|
|
|
#+NAME: Download rpi-imager utility
|
|
#+begin_src bash
|
|
echo Downloading latest release zip from github
|
|
curl -s https://api.github.com/repos/raspberrypi/rpi-imager/releases/latest \
|
|
| grep "browser_download_url.*exe" \
|
|
| cut -d : -f 2,3 \
|
|
| tr -d \" \
|
|
| wget -O /mnt/c/Users/$USER/Downloads/imager.exe -i -
|
|
|
|
echo Checking file is now present
|
|
ls -l /mnt/c/Users/$USER/Downloads/imager.exe
|
|
#+end_src
|
|
|
|
With the software downloaded, let's fire it up the installer and get it setup.
|
|
|
|
#+NAME: Open imager software installer
|
|
#+begin_src shell :results silent
|
|
cmd.exe /mnt/c/Users/$USER/Downloads/imager.exe
|
|
#+end_src
|
|
|
|
Once the software is installed let's run it, the code snippet below will launch the application from the default install directory, or alternatively just launch it from your start menu or preferred shortcut.
|
|
|
|
#+NAME: Launch rpi-imager utility
|
|
#+begin_src shell :results silent
|
|
cd /mnt/c/Program\ Files\ \(x86\)/Raspberry\ Pi\ Imager/ && cmd.exe /k rpi-imager.exe
|
|
#+end_src
|
|
|
|
|
|
** Image sd card
|
|
|
|
Now that we have sd card imaging software installed and running let's put the first sd card into our system and begin the imaging process.
|
|
|
|
First select an Operating System, click ~CHOOSE OS~ --> ~Other general purpose OS~ --> ~Ubuntu Server 20.04.2 LTS x64~.
|
|
|
|
[[./images/imager-distribution.png]]
|
|
|
|
Once you've selected the operating system and sd card, click ~WRITE~. The process will take a few minutes to complete.
|
|
|
|
[[./images/imager-finished.png]]
|
|
|
|
|
|
* Step 2 - Copy the install media to sd card
|
|
|
|
Our next step is to copy the custom ~user-data~ and ~network-config~ files included in this repository to the newly created SD Card.
|
|
|
|
Note: The code block below assumes the SD Card boot partition will be ~D:\~. You may need to adjust for your environment.
|
|
|
|
#+NAME: Mount and copy the new media
|
|
#+begin_src tmate
|
|
echo Mount the new partition in wsl
|
|
sudo mkdir /mnt/d
|
|
sudo mount -t drvfs d: /mnt/d/
|
|
|
|
echo Copy the contents of installer to sd
|
|
cp network-config /mnt/d/
|
|
cp user-data /mnt/d/
|
|
|
|
# We need to wait before we can eject
|
|
sleep 5
|
|
sudo umount /mnt/d
|
|
|
|
sleep 5
|
|
echo Eject the sd card ready for use
|
|
powershell.exe -nologo -command "(new-object -comobject shell.application).namespace(17).parsename('D:').invokeverb('eject')"
|
|
#+end_src
|
|
|
|
|
|
* Step 3 - Boot the pi and remotely connect
|
|
|
|
Provided the configuration on the sd card is valid and the pi has been able to successfully configure networking then following a brief install process the pi will be online and accessible via ssh using the private key corresponding to the public key we supplied in our ~user-data~ file.
|
|
|
|
#+NAME: Connect to the pi
|
|
#+begin_src tmate
|
|
# Setup machine variables
|
|
export port=2142
|
|
export machineip=192.168.1.142
|
|
|
|
# Gather ssh keys if not already known
|
|
ssh-keyscan -p $port $machineip >> ~/.ssh/known_hosts
|
|
|
|
# Connect via ssh
|
|
ssh -p $port $machineip
|
|
#+end_src
|
|
|
|
|
|
* Step 4 - Configure distributed storage
|
|
|
|
One of the goals for this raspberry pi cluster is to run with distributed storage, rather than a traditional single device raid array that the server this cluster is replacing is currently running.
|
|
|
|
The reason I'm interested in this is primarily to explore options for greater hardware redunancy and reliability in the event that a node may go down within the cluster.
|
|
|
|
** Format and mount storage volumes
|
|
|
|
Now that our machines are online and we have connected we can setup our storage cluster.
|
|
|
|
For a distributed storage cluster we are using [[https://www.gluster.org/][glusterfs]]. As part of our earlier setup gluster was automatically installed. We just need to configure it.
|
|
|
|
Our first step is to ensure our storage drives attached to our raspberry pi's are formatted. In our case our drives are all showing as ~/dev/sda~ with no existing partitions, ensure you review your situation with ~lsblk~ first and ajdust the commands below as neccessary!
|
|
|
|
#+NAME: Format and mount storage bricks
|
|
#+begin_src tmate
|
|
# Format the /dev/sda1 partition as xfs
|
|
sudo mkfs.xfs -i size=512 /dev/sda1
|
|
|
|
# Make the mount point directory
|
|
sudo mkdir -p /data/brick1
|
|
|
|
# Update fstab to ensure the mount will resume on boot
|
|
echo '/dev/sda1 /data/brick1 xfs defaults 1 2' | sudo tee -a /etc/fstab
|
|
|
|
# Mount the new filesystem now
|
|
sudo mount -a && sudo mount
|
|
#+end_src
|
|
|
|
|
|
** Configure firewall rules
|
|
|
|
The gluster processes on the nodes need to be able to communicate with each other. To simplify this setup, configure the [[https://en.wikipedia.org/wiki/Iptables][iptables]] firewall on each node to accept all traffic from the other node(s).
|
|
|
|
In our four node cluster this means ensuring we have rules present for all nodes. Adjust as neccessary for the requirements of your cluster!
|
|
|
|
#+NAME: Setup firewall rules for inter cluster communication
|
|
#+begin_src tmate
|
|
# Add the firewall rules
|
|
sudo iptables -I INPUT -p all -s 192.168.1.122 -j ACCEPT
|
|
sudo iptables -I INPUT -p all -s 192.168.1.124 -j ACCEPT
|
|
sudo iptables -I INPUT -p all -s 192.168.1.126 -j ACCEPT
|
|
sudo iptables -I INPUT -p all -s 192.168.1.128 -j ACCEPT
|
|
sudo iptables -I INPUT -p all -s 192.168.1.130 -j ACCEPT
|
|
|
|
# Ensure these are saved permanently
|
|
sudo netfilter-persistent save
|
|
#+end_src
|
|
|
|
|
|
** Ensure the daemon is running
|
|
|
|
Next we need to ensure the glusterfs daemon is enabled and started.
|
|
|
|
#+NAME: Ensure glusterd is enabled and running
|
|
#+begin_src tmate
|
|
# Ensure the gluster service starts on boot
|
|
sudo systemctl enable glusterd
|
|
|
|
# Start the gluster service now
|
|
sudo systemctl start glusterd
|
|
|
|
# Check the service status to confirm running
|
|
sudo systemctl status glusterd
|
|
#+end_src
|
|
|
|
|
|
** Test connectivity between peers
|
|
|
|
Now we're ready to test connectivity between all the gluster peers.
|
|
|
|
#+NAME: Complete cluster probes
|
|
#+begin_src tmate
|
|
# Complete the peer probes
|
|
sudo gluster peer probe 192.168.1.122
|
|
sudo gluster peer probe 192.168.1.124
|
|
sudo gluster peer probe 192.168.1.126
|
|
sudo gluster peer probe 192.168.1.128
|
|
sudo gluster peer probe 192.168.1.130
|
|
|
|
# Validate the peer status
|
|
sudo gluster peer status
|
|
#+end_src
|
|
|
|
|
|
** Setup gluster volume
|
|
|
|
Provided connectivity was established successfully you are now ready to setup a gluster volume.
|
|
|
|
*Note:* The ~gluster volume create~ command only needs to be run from any one node.
|
|
|
|
#+NAME: Setup gluster volume
|
|
#+begin_src shell :wrap example
|
|
# Create the gluster volume folder (all nodes)
|
|
sudo mkdir -p /data/brick1/jammaraid
|
|
|
|
# Create the gluster volume itself (one node)
|
|
sudo gluster volume create jammaraid 192.168.1.122:/data/brick1/jammaraid 192.168.1.124:/data/brick1/jammaraid 192.168.1.126:/data/brick1/jammaraid 192.168.1.128:/data/brick1/jammaraid force
|
|
|
|
# Ensure the volume is started
|
|
sudo gluster volume start jammaraid
|
|
|
|
# Confirm the volume has been created
|
|
sudo gluster volume info
|
|
#+end_src
|
|
|
|
|
|
** Mount and use the new volume
|
|
|
|
Now that the gluster volume has been created and started we can mount it within each node so it is accessible for use :)
|
|
|
|
#+NAME: Mount the gluster volume
|
|
#+begin_src tmate
|
|
# Create the gluster volume mount point
|
|
sudo mkdir -p /media/raid
|
|
|
|
# Mount the volume
|
|
sudo mount -t glusterfs localhost:jammaraid /media/raid
|
|
#+end_src
|
|
|
|
|
|
* Step 5 - Create kubernetes cluster
|
|
|
|
Now can begin installing [[http://k3s.io/][k3s]] on each of the cluster nodes, and then join them into one compute cluster. This will set us up to be able to deploy workloads to that kubernetes cluster.
|
|
|
|
** Download k3s setup binary
|
|
|
|
Our first step is to download the latest ~k3s-armhf~ setup binary from github. Repeat the steps below for each potential cluster node.
|
|
|
|
#+NAME: Knock and enter
|
|
#+begin_src tmate
|
|
# Setup machine variables
|
|
export port=2128
|
|
export machineip=192.168.1.128
|
|
export knocksequence="[SEQUENCE HERE]"
|
|
|
|
# Gather ssh keys if not already known
|
|
ssh-keyscan -p $port $machineip >> ~/.ssh/known_hosts
|
|
|
|
# Knock and enter
|
|
knock $machineip $knocksequence && sleep 2 && ssh -p $port $machineip
|
|
#+end_src
|
|
|
|
#+NAME: Download latest setup binary
|
|
#+begin_src tmate :wrap example
|
|
# Download the latest release dynamically
|
|
curl -s https://api.github.com/repos/rancher/k3s/releases/latest \
|
|
| grep "browser_download_url.*k3s-armhf" \
|
|
| cut -d : -f 2,3 \
|
|
| tr -d \" \
|
|
| wget -i -
|
|
|
|
# Make it executable
|
|
chmod +x k3s-armhf
|
|
|
|
# Leave the node
|
|
exit
|
|
#+end_src
|
|
|
|
|
|
** Initialise the cluster
|
|
|
|
Our next step we only run on the one node that will operate as our cluster master. K3s provides an installation script that is a convenient way to install it as a service on systemd or openrc based systems. This script is available at https://get.k3s.io.
|
|
|
|
After running this installation:
|
|
|
|
* The ~k3s~ service will be configured to automatically restart after node reboots or if the process crashes or is killed.
|
|
* Additional utilities will be installed, including ~kubectl~, ~crictl~, ~ctr~, ~k3s-killall.sh~, and ~k3s-uninstall.sh~.
|
|
* A ~kubeconfig~ file will be written to ~/etc/rancher/k3s/k3s.yaml~ and the kubectl installed by K3s will automatically use it.
|
|
|
|
First step, let's login to our chosen master.
|
|
|
|
#+NAME: Knock and enter
|
|
#+begin_src tmate
|
|
# Setup machine variables
|
|
export port=2124
|
|
export machineip=192.168.1.124
|
|
export knocksequence="[SEQUENCE HERE]"
|
|
|
|
# Gather ssh keys if not already known
|
|
ssh-keyscan -p $port $machineip >> ~/.ssh/known_hosts
|
|
|
|
# Knock and enter
|
|
knock $machineip $knocksequence && sleep 2 && ssh -p $port $machineip
|
|
#+end_src
|
|
|
|
|
|
Once we have logged in we can run the install script.
|
|
|
|
#+NAME: Initialise the master node
|
|
#+begin_src tmate
|
|
curl -sfL https://get.k3s.io | sh -
|
|
#+end_src
|
|
|
|
|
|
Once our master has been deployed by the installation script we can check ~kubectl~ to ensure they are listed as expected.
|
|
|
|
#+NAME: Check cluster nodes
|
|
#+begin_src tmate
|
|
# Check kubectl
|
|
sudo kubectl get nodes
|
|
|
|
# Obtain cluster token
|
|
sudo cat /var/lib/rancher/k3s/server/node-token
|
|
#+end_src
|
|
|
|
|
|
** Join worker nodes
|
|
|
|
Once we have established our cluster masters we need to join workers into the cluster. To install on worker nodes and add them to the cluster, run the installation script with the K3S_URL and K3S_TOKEN environment variables.
|
|
|
|
Repeat the steps below for each worker node, ensuring the node port, machineip and knocksequence are set correctly.
|
|
|
|
#+NAME: Knock and enter
|
|
#+begin_src tmate
|
|
# Setup machine variables
|
|
export port=2128
|
|
export machineip=192.168.1.128
|
|
export knocksequence="[SEQUENCE HERE]"
|
|
|
|
# Gather ssh keys if not already known
|
|
ssh-keyscan -p $port $machineip >> ~/.ssh/known_hosts
|
|
|
|
# Knock and enter
|
|
knock $machineip $knocksequence && sleep 2 && ssh -p $port $machineip
|
|
#+end_src
|
|
|
|
#+NAME: Join worker
|
|
#+begin_src tmate
|
|
# Set environment variables
|
|
export K3S_URL=https://192.168.1.124:6443
|
|
export K3S_TOKEN=[TOKEN_HERE]
|
|
|
|
# Run the installation script
|
|
curl -sfL https://get.k3s.io | sh -
|
|
|
|
# Leave the worker
|
|
exit
|
|
#+end_src
|
|
|
|
|
|
** Check the cluster status
|
|
|
|
Once all workers have been joined lets hop back onto the master and confirm that all nodes are listed as expected.
|
|
|
|
#+NAME: Knock and enter
|
|
#+begin_src tmate
|
|
# Setup machine variables
|
|
export port=2124
|
|
export machineip=192.168.1.124
|
|
export knocksequence="[SEQUENCE HERE]"
|
|
|
|
# Gather ssh keys if not already known
|
|
ssh-keyscan -p $port $machineip >> ~/.ssh/known_hosts
|
|
|
|
# Knock and enter
|
|
knock $machineip $knocksequence && sleep 2 && ssh -p $port $machineip
|
|
#+end_src
|
|
|
|
|
|
#+NAME: Check cluster nodes
|
|
#+begin_src tmate
|
|
# Check kubectl
|
|
sudo kubectl get nodes
|
|
#+end_src
|
|
|
|
|
|
* Step 6 - Deploy a service
|
|
|
|
With our cluster now running, now we can take it for a spin! Let's deploy a simple service. We'll deploy figlet which will take a body over HTTP on port 8080 and return an ASCII-formatted string.
|
|
|
|
We'll need to be logged into our cluster master to do this.
|
|
|
|
#+NAME: Create the service
|
|
#+begin_src tmate
|
|
cat <<EOF > openfaas-figlet-svc.yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: openfaas-figlet
|
|
labels:
|
|
app: openfaas-figlet
|
|
spec:
|
|
type: NodePort
|
|
ports:
|
|
- port: 8080
|
|
protocol: TCP
|
|
targetPort: 8080
|
|
nodePort: 31111
|
|
selector:
|
|
app: openfaas-figlet
|
|
EOF
|
|
#+end_src
|