#+TITLE: Workshop: Chaos Engineering
#+AUTHOR: James Blair
#+DATE: <2023-06-14 Wed 21:00>

Chaos Engineering is [[https://principlesofchaos.org/][defined]] as the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

This document captures some hands on excercises I used during a chaos engineering workshop.



* Application level experiments

Leveraging a combination of OpenShift, Istio, Kiali, ArgoCD and Grafana we can run a great workshop for application level chaos engineering experiments using service mesh fault injection.

A guide for this portion of the workshop is available [[https://redhat-scholars.github.io/chaos-engineering-guide/chaos-engineering/5.0/index.html][here]].


* Cluster level experiments

After completing the above individual hands on excercises the workshop group will come back together to discuss cluster level experiments and follow through the outline below to run some basic experiments.


** Ensure we are logged into our experiment cluster

Before we begin we need to ensure we have the ~oc~ cli installed locally and we have authenticated to the cluster we will be performing experiments on. This guide assuments the file ~/.kube/config~ will be present.

#+begin_src bash
oc login --token <token> --server <server>
#+end_src


** Start a cerberus cluster monitoring instance

Once we are logged into our cluster lets deploy a local instance of cerberus to monitor the cluster and provide go/no go signals to our chaos experiments based on the health of the cluster.

We want to deploy cerberus separately to our cluster to ensure our chaos experiments can't impact our chaos monitoring so we'll spin up cerberus locally with ~podman~.

#+begin_src bash
podman run --net=host --name=cerberus --env-host=true --privileged -d -v /home/james/.kube/config:/root/.kube/config:Z quay.io/openshift-scale/cerberus:kraken-hub
#+end_src


** Test that cerberus is serving and cluster is ready

Once the podman cerberus container is running we can curl the local port ~8080~ to check the cluster health status.

#+begin_src bash
curl localhost:8080
#+end_src


** Complete pod failure experiments with kraken

If all is well with our cluster and cerberus lets start wreaking havoc! This example below will disrupt our etcd cluster in the control plane to verify that even while we temporarily lose a member the cluster continues to operate and recovers automatically.

#+begin_src bash
export CERBERUS_ENABLED=true
export CERBERUS_URL=http://0.0.0.0:8080
export NAMESPACE=openshift-etcd
export POD_LABEL=app=etcd
export DISRUPTION_COUNT=1
export EXPECTED_POD_COUNT=3
podman run --privileged --name=pod_scenario --net=host --env-host=true -v /home/james/.kube/config:/root/.kube/config:Z -d quay.io/openshift-scale/kraken:pod-scenarios
#+end_src


With the scenario running we can follow the progress by reviewing the ~pod_scenario~ pod logs in podman.

#+begin_src bash
podman logs -f pod_scenario
#+end_src


** Complete node failure experiments with kraken

A more destructive test could be starting and stopping nodes, lets try that now!

#+begin_src bash
export CERBERUS_ENABLED=true
export CERBERUS_URL=http://0.0.0.0:8080
export ACTION=node_stop_start_scenario
export INSTANCE_COUNT=1
export CLOUD_TYPE=aws
export WAIT_DURATION=20
podman run --name=node_scenario --net=host --env-host=true -v /home/james/.kube/config:/root/.kube/config:Z -d quay.io/openshift-scale/kraken:node-scenarios
#+end_src


As we did earleir, with the scenario running we can follow the progress by reviewing the ~node_scenario~ pod logs in podman. We can also check the node status through command line or the web console.

#+begin_src bash
podman logs -f node_scenario
#+end_src
