#+TITLE: Raspberry pi k3s cluster guide #+AUTHOR: James Blair #+EMAIL: mail@jamesblair.net #+DATE: 24th December 2019 This file serves as a complete step by step guide for creating a bare metal raspberry pi kubernetes cluster using [[https://k3s.io/][k3s]] from [[https://rancher.com/][Rancher]]. My goal for this build is to replace a server I currently run at home that hosts several workloads via Docker with a scalable k8s cluster. Additionally in future I would like the cluster to be portable and operate via 3G-5G Cellular network and an array of batteries. I chose k3s as it incredibly lightweight but still CNCF certified and production grade software that is optimised for resource constraints of raspberry pis. * Pre-requisites ** Cluster machines For this guide I am using three [[https://www.pishop.us/product/raspberry-pi-4-model-b-4gb/][Raspberry Pi 4 4GB]] machines. The cluster will have one leader node and two worker nodes. For resiliency puposes in future I will update the cluster to run with two leader nodes. *** TODO Migration to high availability control plane For resiliency purposes in future I will update the cluster and this documentation to support a control plane of more than one machine. ** Boot media This guide requires each Raspberry Pi to have a removable SD card or other removable boot media. I am use three 32GB SD Cards though any USB or SD card at least 8GB in size should work fine. *** TODO Migration to network booting In future it would be preferable for the raspberry pi's to be able to network boot and setup automatically without an SD card. This is a nice to have that I will pursue at a later date once I have a deployed cluster that allows me to migrate off the current server setup I have deployed. * Step 1 - Prepare boot media for master ** Download the latest release Our first step is to create the bootable SD Card with a minimal install of [[https://www.raspbian.org/][Raspbian]], which is a free operating system based on [[https://www.debian.org/][Debian]] and is optimised for Raspberry Pi hardware. Rather than doing an installation and configuration of an operating system image from scratch I found [[https://github.com/FooDeas/raspberrypi-ua-netinst][this project]] on Github which automates the install and configuration process nicely. #+NAME: Download the latest release zip #+begin_src shell :results output verbatim replace :wrap example echo Downloading latest release zip from github curl -s https://api.github.com/repos/foodeas/raspberrypi-ua-netinst/releases/latest \ | grep "browser_download_url.*zip" \ | cut -d : -f 2,3 \ | tr -d \" \ | wget -i - echo Checking file is now present ls -l | grep *.zip echo Extracting the zip file unzip -q -d installer *.zip ls -l | grep installer #+end_src #+RESULTS: Download the latest release zip #+begin_example Downloading latest release zip from github Checking file is now present -rw-rw-rw- 1 james james 60299545 Aug 12 08:35 raspberrypi-ua-netinst-v2.4.0.zip Extracting the zip file drwxrwxrwx 1 james james 4096 Dec 24 19:35 installer -rw-rw-rw- 1 james james 1351 Dec 24 18:02 installer-config.txt #+end_example ** Apply custom configuration Our next step after downloading the latest release is to apply our own installation configuration using a simple txt file. There is great documentation online howing what configuration options are available [[https://github.com/malignus/raspberrypi-ua-netinst/blob/master/doc/INSTALL_CUSTOM.md][here]]. For our purposes we just over-write the file downloaded and extracted in the previous step with one we have prepared earlier :) #+NAME: Overwrite installer configuration file #+begin_src shell :results output code verbatim replace :wrap example echo Display wordcount of original file for comparison wc installer/raspberrypi-ua-netinst/config/installer-config.txt echo Overwriting /installer/raspberrypi-ua-netinst/config/installer-config.txt cp installer-config.txt /installer/raspberrypi-ua-netinst/config/ echo Display wordcount of file after copy to validate update wc installer/raspberrypi-ua-netinst/config/installer-config.txt #+end_src #+RESULTS: Overwrite installer configuration file #+begin_example Display wordcount of original file for comparison 33 64 1351 installer/raspberrypi-ua-netinst/config/installer-config.txt Overwriting /installer/raspberrypi-ua-netinst/config/installer-config.txt Display wordcount of file after copy to validate update 33 64 1351 installer/raspberrypi-ua-netinst/config/installer-config.txt #+end_example * Step 2 - Copy the install media to sd card Our next step is to copy the contents of the ~installer/~ folder to a *FAT32* formatted SD Card. Unfortunately this is currently a windows step as my dev environment is a Windows 10 laptop with Debian via Windows Subsystem for Linux which does not support ~lsblk~ or other disk management commands. ** Obtain sd card partition information Our first step is to insert the SD Card and ensure it is formatted correctly as ~FAT32~. To do that we need to know the number of the disk we want to format, we can find that via powershell. #+NAME: Get disks via windows powershell #+begin_src shell :results output code verbatim replace :wrap example echo Retrieving disk list via powershell powershell.exe -nologo get-disk | select Number, FriendlyName, Size echo Retrieving partition list via powershell get-disk | get-partition | select PartitionNumber, DriveLetter, Size, Type exit #+end_src #+RESULTS: Get disks via windows powershell #+begin_example Retrieving disk list via powershell Number FriendlyName Size ------ ------------ ---- 1 Realtek PCIE Card Reader 31104958464 0 SAMSUNG MZVLB256HAHQ-000H1 256060514304 Retrieving partition list via powershell PartitionNumber DriveLetter Size Type --------------- ----------- ---- ---- 1 D 268435456 FAT32 XINT13 2 E 30832328704 Unknown 1 272629760 System 2 16777216 Reserved 3 C 254735810560 Basic 4 1027604480 Recovery #+end_example ** Create and format sd card partition Once we know the number of the disk we want to format we can proceed. In the example above I have a 32GB SD Card which shows as number ~1~. Checking the disk we can see some partitions that exist already from previous use of the card. To delete these partitions you can use the ~Remove-Partition -DiskNumber X -PartitionNumber Y~ command where ~X~ and ~Y~ relate to the output of your disk and partition number. Due to the risk of data loss this step is not automated. Once existing partitions have been cleared we can use the following block to: - Create a new partition using masixmum available space - Assign a free drive letter in windows - Mount the disk in WSL so we can copy to it - Copy the install media over to the partition #+NAME: Create sd card partition and copy media #+begin_src shell :results output code verbatim replace echo Use powershell to create new partition and format powershell.exe -nologo new-partition -disknumber 1 -usemaximumsize -driveletter d format-volume -driveletter d -filesystem FAT32 -newfilesystemlabel sd exit echo Mount the new partition in wsl sudo mkdir /mnt/d sudo mount -t drvfs d: /mnt/d echo Copy the contents of installer to sd cp installer/* /mnt/d/ #+end_src * Step 3 - Boot the pi and remotely connect Provided the configuration on the sd card is valid and the pi has been able to successfully obtain an ip address via dhcp on boot then following a 10-20minute net install process the pi will be online and accessible via ssh using the private key corresponding to the public key we supplied in our ~installer-config.txt~ file. ** Setup ssh and connect #+NAME: Ensure our ssh-agent is setup