Check SNO Deployment is Finished

In the previous section we have learned how to follow the cluster deployment process, but the cluster finished its deployment doesn’t mean that the SNO deployment is complete.

We say that the SNO deployment finished when the SNO cluster has been deployed and day2 configurations has been applied. In this section we will learn how to verify that the configs have been applied and our SNO is ready to run 5G RAN workloads.

Check SNO Deployment has Finished via the WebUI

Sometimes you may hit a bug that will cause the policies to not be properly applied. Follow the instructions here if after 20 minutes policies are not ready.

  1. Access the RHACM WebUI and login with the OpenShift credentials.

  2. Once you’re in, click on Infrastructure → Clusters. You will see a screen like the one below, notice that there is a label saying ztp-done for the SNO2 cluster (that means ztp pipeline has finished):

    RHACM SNO2 ztp-done

  3. Additionally, we can verify that all policies are compliant for our SNO cluster. On the RHACM WebUI go to Governance → Policies and you will see the following screen where we can see all policies are green:

    Applying policies on sno2 can take up to 15 minutes after cluster has been fully deployed.
    RHACM SNO2 policies done

  4. At this point the SNO is ready to run 5G RAN workloads.

Check SNO Deployment has Finished via the CLI

Sometimes you may hit a bug that will cause the policies to not be properly applied. Follow the instructions here if after 20 minutes policies are not ready.
Below commands must be executed from the workstation host if not specified otherwise.

  1. Check the ManagedCluster for SNO2 cluster (including its labels). You should see the ztp-done label, just as we saw in the WebUI.

    oc --kubeconfig ~/5g-deployment-lab/hub-kubeconfig get managedcluster sno2 --show-labels
    NAME   HUB ACCEPTED   MANAGED CLUSTER URLS                      JOINED   AVAILABLE   AGE    LABELS
sno2   true           https://api.sno2.5g-deployment.lab:6443   True     True        122m   <OMITTED_LABELS>,ztp-done=

  2. Check the policies are compliant for our SNO cluster.

    oc --kubeconfig ~/5g-deployment-lab/hub-kubeconfig -n sno2 get policies
    Applying policies on sno2 can take up to 15 minutes after cluster has been fully deployed. 
    NAME                                        REMEDIATION ACTION   COMPLIANCE STATE   AGE
ztp-policies.common-config-policies         inform               Compliant          74m
ztp-policies.common-subscription-policies   inform               Compliant          74m
ztp-policies.du-sno-group-policies          inform               Compliant          74m
ztp-policies.sno2-site-policies             inform               Compliant          74m

  3. At this point the SNO is ready to run 5G RAN workloads.

Ensure AAP Automation Ran Successfully

Now that SNO deployment is finished, automation should run as soon as the cluster is marked with the ztp-done label. We can easily check the labels on our sno2 cluster via the OpenShift web console, under the Infrastructure > Clusters tab.

There are a couple ways to ensure automation has run successfully.

First, the Ansible job template used in this lab was configured to add an aap-done label to a cluster after it has run automation on it successfully, so notice that our sno1 and sno2 clusters now have the label aap-done.

SNO1/SNO2 AAP Done Label

Additionally, under the Applications tab on our OpenShift web console, if we search for monitor-ztp-done, we will see our automation Application. Let’s take a look at this Application by clicking on its name. Notice the green check next to Cluster resource status, indicating automation has ran successfully. Now, navigating to the Topology tab we see a visual representation of the automation that has ran.

AAP Application
AAP Topology

Lastly, to view status and logs for the actual Ansible Controller job that ran, we can navigate to https://automation-aap.apps.hub.5g-deployment.lab, click on the "Jobs" tab, and view the latest job that ran.

AAP Jobs
Single AAP Job
As we can see the automation has run on every managed cluster with the label ztp-done and that is not the Hub cluster itself. This model of maintain one automation application for day 2 operations for N clusters makes utilizing ZTP and AAP automation together in a 5G RAN environment great at scale while also maintaining configuration control, as controlling which clusters run automation is as simple as adjusting the automation application’s PlacementRule.