184 lines
7.9 KiB
Org Mode
184 lines
7.9 KiB
Org Mode
#+TITLE: Gitlab GCP Deployment
|
|
#+AUTHOR: James Blair
|
|
#+EMAIL: mail@jamesblair.net
|
|
#+DATE: 5th January 2021
|
|
|
|
This org file is intended to capture and automate the end to end workflow to deploy an instance of [[https://gitlab.com][Gitlab]] on [[https://console.cloud.google.com][Google Cloud Platform]].
|
|
|
|
We'll use shell blocks inside this file which can be executed with [[https://orgmode.org/worg/org-contrib/babel/][Babel]]. Additionally we want to explore tangling these source code blocks to shell script files within this document so that the scripts can then be executed by a continous delivery pipeline.
|
|
|
|
*Notes:*
|
|
1. To interact with this org file we're using the [[https://github.com/humacs/humacs][Humacs]] distribution of [[https://www.gnu.org/software/emacs/][Emacs]].
|
|
1. This workflow has only been tested on the ~Ubuntu 20.04~ linux distribution, via [[https://ubuntu.com/wsl][WSL 2]].
|
|
|
|
|
|
* Step 1 - Ensure GCP SDK is installed
|
|
|
|
To automate our interactions with Google Cloud Platform we'll use the [[https://cloud.google.com/sdk/docs/install#deb][GCP SDK]] which provides us with a number of command line tools to interact with the platform, such as ~gcloud~, ~gsutil~ and ~kubectl~.
|
|
|
|
Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:
|
|
|
|
#+NAME: Install google cloud sdk
|
|
#+BEGIN_SRC bash :shebang #!/bin/bash :tangle 1-install-gcp-sdk.sh
|
|
# Add the Cloud SDK distribution URI as a package source
|
|
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list
|
|
|
|
# Make sure apt-transport-https is installed
|
|
sudo apt-get install -y apt-transport-https ca-certificates gnupg
|
|
|
|
# Import the Google Cloud public key
|
|
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
|
|
|
|
# Update and install the SDK
|
|
sudo apt-get update && sudo apt-get install -y google-cloud-sdk
|
|
#+END_SRC
|
|
|
|
|
|
* Step 2 - Configure cloud resources
|
|
|
|
With GCP SDK now installed we need to authenticate, create a project and then create a virtual machine instance that we will install Gitlab into later in the workflow.
|
|
|
|
First up is authentication so our GCP SDK installation can carry out actions in a given account and project. This part of the process is currently a manual step as the authentication process includes some interactive steps.
|
|
|
|
In future we could automate this process as part of a continous delivery pipeline using a GCP service account with permission to create virtual machine instances.
|
|
|
|
#+NAME: Authenticate with google cloud platform
|
|
#+BEGIN_SRC bash :shebang #!/bin/bash :tangle no
|
|
gcloud auth login
|
|
#+END_SRC
|
|
|
|
|
|
Once we have authenticated we can create a project and then create a new virtual machine instance within that project.
|
|
|
|
Firstly let's create a new project, a project is the logical boundary all our cloud resources for this deployment will live within. To be able to deploy resources we also need to enable billing.
|
|
|
|
Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:
|
|
|
|
#+NAME: Create a new google cloud project
|
|
#+begin_src bash :shebang #!/bin/bash :tangle 2-configure-gcp-project.sh
|
|
# Create a project id based on date
|
|
export gcp_project_id="gitlab-gcp-"$(date +"%s")
|
|
|
|
# Create new project using a random project id
|
|
gcloud projects create $gcp_project_id
|
|
|
|
# Ensure billing is enabled for the project
|
|
export gcp_billing_account=$(gcloud alpha billing accounts list --limit=1 --format='value(name.basename())')
|
|
gcloud alpha billing projects link $gcp_project_id --billing-account $gcp_billing_account
|
|
|
|
# Make sure the project is set active
|
|
gcloud config set project $gcp_project_id
|
|
#+end_src
|
|
|
|
#+RESULTS: Create a new google cloud project
|
|
#+begin_src bash
|
|
billingAccountName: billingAccounts/0175A9-AC77BE-CE5586
|
|
billingEnabled: true
|
|
name: projects/gitlab-gcp-1611720361/billingInfo
|
|
projectId: gitlab-gcp-1611720361
|
|
#+end_src
|
|
|
|
|
|
* Step 3 - Create vertual machine
|
|
|
|
Once we have a project we can create a new virtual machine. To create a virtual machine we need to ensure compute engine apis are enabled.
|
|
|
|
Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:
|
|
|
|
#+begin_src bash :shebang #!/bin/bash :tangle 3-create-virtual-machine.sh
|
|
# Ensure compute engine apis are enabled in the project
|
|
gcloud services enable compute.googleapis.com
|
|
|
|
# Create name for virtual machine based on date
|
|
export gcp_machine_name="gitlab-gcp-"$(date +"%s")
|
|
|
|
# Create the new machine
|
|
gcloud compute instances create $gcp_machine_name --zone australia-southeast1-a
|
|
#+end_src
|
|
|
|
|
|
* Step 4 - Install docker on virtual machine
|
|
|
|
Next up we need to install [[https://docker.com][Docker]] on the newly created virtual machine so that we can then deploy Gitlab as a container.
|
|
|
|
By default the virtual machine operating system for the vm we created on GCP is [[https://debian.org][Debian]]. There are instructions for installing Docker on a debian machine [[https://docs.docker.com/engine/install/debian/#install-using-the-repository][here]].
|
|
|
|
#+begin_src bash :shebang #!/bin/bash :tangle 4-install-docker.sh
|
|
# Retrieve the vm name
|
|
export gcp_machine_name=$(gcloud compute instances list --limit=1 --format='value(name.basename())')
|
|
|
|
# Connect to the machine using ssh
|
|
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh"
|
|
|
|
# Install docker compose
|
|
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "sudo curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose"
|
|
|
|
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- "sudo chmod +x /usr/local/bin/docker-compose"
|
|
#+end_src
|
|
|
|
|
|
* Step 5 - Install gitlab via docker
|
|
|
|
Next step to create docker compose file for gitlab
|
|
|
|
#+begin_src yaml :tangle docker-compose.yaml
|
|
web:
|
|
image: 'gitlab/gitlab-ce:latest'
|
|
restart: always
|
|
hostname: 'gitlab.example.com'
|
|
environment:
|
|
GITLAB_OMNIBUS_CONFIG: |
|
|
external_url 'https://gitlab.example.com:3200'
|
|
# Add any other gitlab.rb configuration here, each on its own line
|
|
ports:
|
|
- '3200:3200'
|
|
- '443:443'
|
|
- '2222:22'
|
|
volumes:
|
|
- '$GITLAB_HOME/config:/etc/gitlab'
|
|
- '$GITLAB_HOME/logs:/var/log/gitlab'
|
|
- '$GITLAB_HOME/data:/var/opt/gitlab'
|
|
|
|
#+end_src
|
|
|
|
Next up we need to tranfer file to gcloud vm and install the gitlab via docker composer
|
|
|
|
#+begin_src bash :shebang #!/bin/bash :tangle 5-install-gitlab-via-composer.sh
|
|
# Retrieve the vm name
|
|
export gcp_machine_name=$(gcloud compute instances list --limit=1 --format='value(name.basename())')
|
|
|
|
# Copy file to vm
|
|
gcloud compute scp docker-compose.yaml $gcp_machine_name:/home/$USER --ssh-key-file ~/.ssh/$USER --strict-host-key-checking=no
|
|
|
|
# Install gotlan with docker compose
|
|
sleep 2
|
|
gcloud compute ssh $gcp_machine_name --ssh-key-file ~/.ssh/$USER -- 'sudo docker-compose up -d'
|
|
#+end_src
|
|
|
|
|
|
* Step 6 - Teardown cloud resources
|
|
|
|
The Google Cloud Platform resources created by this process come at a cost, so it's important we have an easy way to teardown those resources as soon as we're finished with them!
|
|
|
|
The script below will delete any projects containing ~gitlab~ in the name along with any compute instances running in those projects.
|
|
|
|
Tangle the shell block below to a shell script by pressing *, b t* in emacs command mode:
|
|
|
|
#+begin_src bash :shebang #!/bin/bash :tangle 6-teardown-cloud-resources.sh
|
|
# Iterate over any matching projects
|
|
for project in $(gcloud projects list | awk '{ print $1 }' | grep gitlab); do
|
|
|
|
# Iterate over any instances in the project
|
|
for instance in $(gcloud compute instances list --project $project --format="value(name)"); do
|
|
|
|
# Delete the instance
|
|
gcloud compute instances delete --quiet $instance --zone australia-southeast1-a --project $project
|
|
|
|
done
|
|
|
|
# Delete the project as well
|
|
gcloud projects delete $project --quiet
|
|
|
|
done
|
|
#+end_src
|