gitlab-gcp-deployment/README.org

#+TITLE: Gitlab GCP Deployment
#+AUTHOR: James Blair
#+EMAIL: mail@jamesblair.net
#+DATE: 5th January 2021

This org file is intended to capture and automate the end to end workflow to deploy an instance of [[https://gitlab.com][Gitlab]] on [[https://console.cloud.google.com][Google Cloud Platform]].

We'll use shell blocks inside this file which can be executed with [[https://orgmode.org/worg/org-contrib/babel/][Babel]]. Additionally we want to explore tangling these source code blocks to shell script files within this document so that the scripts can then be executed by a continous delivery pipeline.

*Notes:*
 1. To interact with this org file we're using the [[https://github.com/humacs/humacs][Humacs]] distribution of [[https://www.gnu.org/software/emacs/][Emacs]].
 1. This workflow has only been tested on the ~Ubuntu 20.04~ linux distribution, via [[https://ubuntu.com/wsl][WSL 2]].


* Step 1 - Ensure GCP SDK is installed

To automate our interactions with Google Cloud Platform we'll use the [[https://cloud.google.com/sdk/docs/install#deb][GCP SDK]] which provides us with a number of command line tools to interact with the platform, such as ~gcloud~, ~gsutil~ and ~kubectl~.

Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:

#+NAME: Install google cloud sdk
#+BEGIN_SRC bash :shebang #!/bin/bash :tangle 1-install-gcp-sdk.sh
# Add the Cloud SDK distribution URI as a package source
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list

# Make sure apt-transport-https is installed
sudo apt-get install -y apt-transport-https ca-certificates gnupg

# Import the Google Cloud public key
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

# Update and install the SDK
sudo apt-get update && sudo apt-get install -y google-cloud-sdk
#+END_SRC


* Step 2 - Configure cloud resources

With GCP SDK now installed we need to authenticate, create a project and then create a virtual machine instance that we will install Gitlab into later in the workflow.

First up is authentication so our GCP SDK installation can carry out actions in a given account and project. This part of the process is currently a manual step as the authentication process includes some interactive steps.

In future we could automate this process as part of a continous delivery pipeline using a GCP service account with permission to create virtual machine instances.

#+NAME: Authenticate with google cloud platform
#+BEGIN_SRC bash :shebang #!/bin/bash :tangle no
gcloud auth login
#+END_SRC


Once we have authenticated we can create a project and then create a new virtual machine instance within that project.

Firstly let's create a new project, a project is the logical boundary all our cloud resources for this deployment will live within. To be able to deploy resources we also need to enable billing.

Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:

#+NAME: Create a new google cloud project
#+begin_src bash :shebang #!/bin/bash :tangle 2-configure-gcp-project.sh
# Create a project id based on date
export gcp_project_id="gitlab-gcp-"$(date +"%s")

# Create new project using a random project id
gcloud projects create $gcp_project_id

# Ensure billing is enabled for the project
export gcp_billing_account=$(gcloud alpha billing accounts list --limit=1 --format='value(name.basename())')
gcloud alpha billing projects link $gcp_project_id --billing-account $gcp_billing_account

# Make sure the project is set active
gcloud config set project $gcp_project_id
#+end_src

#+RESULTS: Create a new google cloud project
#+begin_src bash
billingAccountName: billingAccounts/0175A9-AC77BE-CE5586
billingEnabled: true
name: projects/gitlab-gcp-1611720361/billingInfo
projectId: gitlab-gcp-1611720361
#+end_src


* Step 3 - Create vertual machine

Once we have a project we can create a new virtual machine. To create a virtual machine we need to ensure compute engine apis are enabled.

Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:

#+begin_src bash :shebang #!/bin/bash :tangle 3-create-virtual-machine.sh
# Ensure compute engine apis are enabled in the project
gcloud services enable compute.googleapis.com

# Create name for virtual machine based on date
export gcp_machine_name="gitlab-gcp-"$(date +"%s")

# Create the new machine
gcloud compute instances create $gcp_machine_name --zone australia-southeast1-a
#+end_src


* Step 4 - Install docker on virtual machine

Next up we need to install [[https://docker.com][Docker]] on the newly created virtual machine so that we can then deploy Gitlab as a container.

By default the virtual machine operating system for the vm we created on GCP is [[https://debian.org][Debian]]. There are instructions for installing Docker on a debian machine [[https://docs.docker.com/engine/install/debian/#install-using-the-repository][here]].

#+begin_src bash :shebang #!/bin/bash :tangle 4-install-docker.sh
# Retrieve the vm name
export gcp_machine_name=$(gcloud compute instances list --limit=1 --format='value(name.basename())')

# Connect to the machine using ssh
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh"

# Install docker compose
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "sudo curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose"

gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "sudo chmod +x /usr/local/bin/docker-compose"
#+end_src


* Step 5 - Install gitlab via docker

Next step to create docker compose file for gitlab

#+begin_src yaml :tangle docker-compose.yaml
   web:
     image: 'gitlab/gitlab-ce:latest'
     restart: always
     hostname: 'gitlab.example.com'
     environment:
       GITLAB_OMNIBUS_CONFIG: |
         external_url 'https://gitlab.example.com:3200'
     # Add any other gitlab.rb configuration here, each on its own line
     ports:
       - '3200:3200'
       - '443:443'
       - '2222:22'
     volumes:
       - '$GITLAB_HOME/config:/etc/gitlab'
       - '$GITLAB_HOME/logs:/var/log/gitlab'
       - '$GITLAB_HOME/data:/var/opt/gitlab'

#+end_src

Next up we need to copy file to gloud vm

#+begin_src bash :shebang #!/bin/bash :tangle 5-install-gitlab-via-composer.sh
# Retrieve the vm name
export gcp_machine_name=$(gcloud compute instances list --limit=1 --format='value(name.basename())')

# Copy file to vm
gloud compute scp docker-compose.yaml $gcp_machine_name --ssh-key-file ~/.ssh/$USER --strict-host-key-checking=no

# Install gotlan with docker compose
sleep 2
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- 'sudo docker compose up -d'
#+end_src


* Step 6 - Teardown cloud resources

The Google Cloud Platform resources created by this process come at a cost, so it's important we have an easy way to teardown those resources as soon as we're finished with them!

The script below will delete any projects containing ~gitlab~ in the name along with any compute instances running in those projects.

Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:

#+begin_src bash :shebang #!/bin/bash :tangle 6-teardown-cloud-resources.sh
# Iterate over any matching projects
for project in $(gcloud projects list | awk '{ print $1 }' | grep gitlab); do

  # Iterate over any instances in the project
  for instance in $(gcloud compute instances list --project $project --format="value(name)"); do

    # Delete the instance
    gcloud compute instances delete --quiet $instance --zone australia-southeast1-a --project $project

  done

  # Delete the project as well
  gcloud projects delete $project --quiet

done
#+end_src