OpenShift Advanced Cluster Management Observability
- Introduction
- Pre-requisites
- 1 - Logging into aws locally
- 2 - Creating s3 bucket
- 3 - Install openshift clusters
- 4 - Install advanced cluster management
- 5 - Enable acm observability
- 6 - Import the single node openshift cluster into acm
- 7 - Creating the edge workload on SNO
Introduction
This document captures the environment setup steps for a ~30 minute live demo of the Red Hat Advanced Cluster Management observability feature for Openshift.
Pre-requisites
This guide assumes you:
- Have access to an Amazon Web Services account with permissions to be able to create resources including
s3buckets andec2instances. In my case I have an AWS Blank Open Environment provisioned through the Red Hat demo system. - Already have the
awsandoccli utilities installed. - Have registered for a Red Hat account (required for obtaining an OpenShift install image pull secret).
1 - Logging into aws locally
Our first step is to login to our aws account locally via the aws cli which will prompt for four values:
aws configure2 - Creating s3 bucket
After logging into aws lets confirm our permissions are working by creating the s3 bucket we will need later on.
aws s3 mb "s3://open-cluster-management-observability" --region "$(aws configure get region)"3 - Install openshift clusters
With our aws credentials working let's move on to deploying the hub and single node openshift cluster required for the live demo.
3.1 Download installer tools
Our first step will be to ensure we have the openshift-install cli tool. We can download it as follows:
# Download the installer
wget "https://mirror.openshift.com/pub/openshift-v4/$(uname -m)/clients/ocp/stable/openshift-install-linux.tar.gz"
# Extract the archive
tar xf openshift-install-linux.tar.gz3.2 Obtain install pull secret
Next we have a manual step to login to the Red Hat Hybrid Cloud Console and obtain our Pull Secret which will be required for our installation configuration.
Open the Console and click Download pull secret. This will download a file called pull-secret.txt which will be used later on.
3.3 Create ssh key
For access to our soon to be created clusters we need an ssh key, let's generate those now via ssh-keygen.
ssh-keygen -t rsa -b 4096 -f ~/.ssh/hubkey -q -N ""
ssh-keygen -t rsa -b 4096 -f ~/.ssh/snokey -q -N ""3.3 Initiate the hub cluster install
Once our install tooling is available let's kick off the installation of our hub cluster by creating a configuration file and then running openshift-install.
cat << EOF > hub/install-config.yaml
additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: $(aws route53 list-hosted-zones | jq '.HostedZones[].Name' -r | sed 's/.$//')
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform: {}
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: hub
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: $(aws configure get region)
publish: External
pullSecret: |
$(cat pull-secret.txt)
sshKey: |
$(cat ~/.ssh/hubkey.pub)
EOF
Once the configuration file is created we can kick off the install with openshift-install as follows:
./openshift-install create cluster --dir hub --log-level info3.4 Initiate the sno cluster install
We can run our single node openshift cluster install at the same time in a separate terminal to speed things up. The process is the same we will first create an install-config.yaml file, then run openshift-install.
cat << EOF > sno/install-config.yaml
additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: $(aws route53 list-hosted-zones | jq '.HostedZones[].Name' -r | sed 's/.$//')
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform: {}
replicas: 0
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform: {}
replicas: 1
metadata:
creationTimestamp: null
name: sno
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: $(aws configure get region)
publish: External
pullSecret: |
$(cat pull-secret.txt)
sshKey: |
$(cat ~/.ssh/snokey.pub)
EOF
Once the configuration file is created we can kick off the install with openshift-install as follows:
./openshift-install create cluster --dir sno --log-level info4 - Install advanced cluster management
To make use of the Red Hat Advanced Cluster Management Observability feature we need to first install advanced cluster management on our hub cluster via the acm operator.
Let's get started by creating an OperatorGroup and Subscription which will install the operator.
oc create namespace open-cluster-management
cat << EOF | oc apply --filename -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: acm-operator-group
namespace: open-cluster-management
spec:
targetNamespaces:
- open-cluster-management
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: acm-operator-subscription
namespace: open-cluster-management
spec:
sourceNamespace: openshift-marketplace
source: redhat-operators
channel: release-2.9
installPlanApproval: Automatic
name: advanced-cluster-management
EOF
Once the operator is installed we can create the MultiClusterHub resource to install Advanced Cluster Management.
Note: It can take up to ten minutes for this to complete.
cat << EOF | oc apply --filename -
apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
name: multiclusterhub
namespace: open-cluster-management
spec: {}
EOF5 - Enable acm observability
Now, with our clusters deployed and acm installed we can enable the observability service by creating a MultiClusterObservability custom resource instance on the hub cluster.
Our first step towards this is to create two secrets.
oc create namespace open-cluster-management-observability
DOCKER_CONFIG_JSON=`oc extract secret/pull-secret -n openshift-config --to=-`
oc create secret generic multiclusterhub-operator-pull-secret \
-n open-cluster-management-observability \
--from-literal=.dockerconfigjson="$DOCKER_CONFIG_JSON" \
--type=kubernetes.io/dockerconfigjson
cat << EOF | oc apply --filename -
apiVersion: v1
kind: Secret
metadata:
name: thanos-object-storage
namespace: open-cluster-management-observability
type: Opaque
stringData:
thanos.yaml: |
type: s3
config:
bucket: open-cluster-management-observability
endpoint: s3.$(aws configure get region).amazonaws.com
insecure: true
access_key: $(aws configure get aws_access_key_id)
secret_key: $(aws configure get aws_secret_access_key)
EOF
Once the two required secrets exist we can create the MultiClusterObservability resource as follows:
cat << EOF | oc apply --filename -
apiVersion: observability.open-cluster-management.io/v1beta2
kind: MultiClusterObservability
metadata:
name: observability
spec:
observabilityAddonSpec: {}
storageConfig:
metricObjectStorage:
name: thanos-object-storage
key: thanos.yaml
EOF
After creating the resource and waiting briefyl we can access the grafana console via the Route to confirm everything is running:
echo "https://$(oc get route -n open-cluster-management-observability grafana -o jsonpath={.spec.host})"6 - Import the single node openshift cluster into acm
oc new-project sno
oc label namespace sno cluster.open-cluster-management.io/managedCluster=snocat << EOF | oc apply --filename -
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
name: sno
spec:
hubAcceptsClient: true
---
apiVersion: agent.open-cluster-management.io/v1
kind: KlusterletAddonConfig
metadata:
name: sno
namespace: sno
spec:
clusterName: sno
clusterNamespace: sno
applicationManager:
enabled: true
certPolicyController:
enabled: true
clusterLabels:
cloud: auto-detect
vendor: auto-detect
iamPolicyController:
enabled: true
policyController:
enabled: true
searchCollector:
enabled: true
version: 2.0.0
EOF
The ManagedCluster-Import-Controller will generate a secret named sno-import. The sno-import secret contains the import.yaml that the user applies to a managed cluster to install klusterlet.
oc get secret sno-import -n sno -o jsonpath={.data.crds\\.yaml} | base64 --decode > klusterlet-crd.yaml
oc get secret sno-import -n sno -o jsonpath={.data.import\\.yaml} | base64 --decode > import.yaml
oc --kubeconfig sno/auth/kubeconfig apply --filename klusterlet-crd.yaml
oc --kubeconfig sno/auth/kubeconfig apply --filename import.yamlIf everything works fine you should see JOINED and AVAILABLE sno cluster from within your hub cluster
❯ kubectl get managedcluster -n sno
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE
local-cluster true https://api.hub.<yourdomain>.com:6443 True True 5h12m
sno true https://api.cluster-vzmvz.<yourdomain>.com:6443 True True 31m7 - Creating the edge workload on SNO
For edge scenarios we only send metrics to the hub cluster if certain thresholds are hit for a certain period of time (here 70% for more than 2 minutes - you can see this configuration in the open-cluster-management-addon-observability namespace under ConfigMaps observability-metrics-allowlist in the collect_rules section under SNOHighCPUUsage). In order to hit that trigger we now deploy a cpu-heavy workload in order for sno-cluster metrics being sent to the ACM hub cluster.
Let's get started by creating a new project on the sno cluster:
oc new-project cpu-load-testand deploy the cpu-load-container workload on a busybox container
cat << EOF | oc apply --filename -
apiVersion: apps/v1
kind: Deployment
metadata:
name: cpu-load-test
spec:
replicas: 5
selector:
matchLabels:
app: cpu-load-test
template:
metadata:
labels:
app: cpu-load-test
spec:
containers:
- name: cpu-load-container
image: busybox
command: ["/bin/sh", "-c"]
args:
- while true; do
echo "Performing CPU load...";
dd if=/dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 1000 | head -n 1000000 > /dev/null;
done
EOF