There may be times when you need to change aspects of your worker nodes. Things like scaling, changing the type, adding labels or taints to name a few. Most of these things are done through the use of machine sets. A machine is a unit that describes the host for a node and a machine set is a group of machines. Think of a machine set as a “template” for the kinds of machines that make up the worker nodes of your cluster. Similar to how a replicaset is to pods. A machine set allows users to manage many machines as a single entity though it is contained to a specific availability zone. If you'd like to learn more see [Overview of machine management](https://docs.openshift.com/container-platform/latest/machine_management/index.html)
### Scaling worker nodes
{% collapsible %}
#### View the machine sets that are in the cluster
Let's see which machine sets we have in our cluster. If you are following this lab, you should only have three so far (one for each availability zone).
From the terminal run:
`oc get machinesets -n openshift-machine-api`
You will see a response like:
```
$ oc get machinesets -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
ok0620-rq5tl-worker-westus21 1 1 1 1 72m
ok0620-rq5tl-worker-westus22 1 1 1 1 72m
ok0620-rq5tl-worker-westus23 1 1 1 1 72m
```
This is telling us that there is a machine set defined for each availability zone in westus2 and that each has one machine.
#### View the machines that are in the cluster
Let's see which machines (nodes) we have in our cluster.
As you can see we have 3 master nodes, 3 worker nodes, the types of nodes, and which region/zone they are in.
#### Scale the number of nodes up via the CLI
Now that we know that we have 3 worker nodes, let's scale the cluster up to have 4 worker nodes. We can accomplish this through the CLI or through the OpenShift Web Console. We'll explore both.
From the terminal run the following to imperatively scale up a machine set to 2 worker nodes for a total of 4. Remember that each machine set is tied to an availability zone so with 3 machine sets with 1 machine each, in order to get to a TOTAL of 4 nodes we need to select one of the machine sets to scale up to 2 machines.
#### Scale the number of nodes down via the Web Console
Now let's scale the cluster back down to a total of 3 worker nodes, but this time, from the web console. (If you need the URL or credentials in order to access it please go back to the relevant portion of Lab 1)
Access your OpenShift web console from the relevant URL. If you need to find the URL you can run:
In the main pane you will see the same information about the machine sets from the command line. Now click on the "three dots" at the end of the line for the machine set that you scaled up to "2". Select "Edit machine count" and decrease it to "1". Click save.
This will now decrease that machine set to only have one machine in it.
{% endcollapsible %}
### Cluster Autoscaling
{% collapsible %}
The cluster autoscaler adjusts the size of an OpenShift Container Platform cluster to meet its current deployment needs. The cluster autoscaler increases the size of the cluster when there are pods that fail to schedule on any of the current worker nodes due to insufficient resources or when another node is necessary to meet deployment needs. The cluster autoscaler does not increase the cluster resources beyond the limits that you specify. To learn more visit the documentation for [cluster autoscaling](https://docs.openshift.com/container-platform/latest/machine_management/applying-autoscaling.html).
A ClusterAutoscaler must have at least 1 machine autoscaler in order for the cluster autoscaler to scale the machines. The cluster autoscaler uses the annotations on machine sets that the machine autoscaler sets to determine the resources that it can scale. If you define a cluster autoscaler without also defining machine autoscalers, the cluster autoscaler will never scale your cluster.
Download the sample [MachineAutoscaler resource definition](https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/machine-autoscaler.yaml) and open it in your favorite editor.
For `metadata.name` give this machine autoscaler a name. Technically, this can be anything you want. But to make it easier to identify which machine set this machine autoscaler affects, specify or include the name of the machine set to scale. The machine set name takes the following form: \<clusterid>-\<machineset>-\<region-az>.
This is the sample [ClusterAutoscaler resource definition](https://raw.githubusercontent.com/microsoft/aroworkshop/master/yaml/cluster-autoscaler.yaml) for this lab.
See the [documentation](https://docs.openshift.com/container-platform/latest/machine_management/applying-autoscaling.html#cluster-autoscaler-cr_applying-autoscaling) for a detailed explanation of each parameter. You shouldn't need to edit this file.
clusterautoscaler.autoscaling.openshift.io/default created
```
#### Test the Cluster Autoscaler
Now we will test this out. Create a new project where we will define a job with a load that this cluster cannot handle. This should force the cluster to autoscale to handle the load.
We see a lot of pods in a pending state. This should trigger the cluster autoscaler to create more machines using the MachineAutoscaler we created. If we check on the MachineSets:
```
$ oc get machinesets -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
ok0620-rq5tl-worker-westus21 5 5 1 1 7h17m
ok0620-rq5tl-worker-westus22 1 1 1 1 7h17m
ok0620-rq5tl-worker-westus23 1 1 1 1 7h17m
```
We see that the cluster autoscaler has already scaled the machine set up to 5 in our example. Though it is still waiting for those machines to be ready.
If we check on the machines we should see that 4 are in a "Provisioned" state (there was 1 already existing from before for a total of 5 in this machine set).
After a few minutes we should see all 5 are provisioned.
```
$ oc get machinesets -n openshift-machine-api
NAME DESIRED CURRENT READY AVAILABLE AGE
ok0620-rq5tl-worker-westus21 5 5 5 5 7h23m
ok0620-rq5tl-worker-westus22 1 1 1 1 7h23m
ok0620-rq5tl-worker-westus23 1 1 1 1 7h23m
```
If we now wait a few more minutes for the pods to complete, we should see the cluster autoscaler begin scale down the machine set and thus delete machines.
To add a node label it is recommended to set the label in the machine set. While you can directly add a label the node, this is not recommended since nodes could be overwritten and then the label would disappear. Once the machine set is modified to contain the desired label any new machines created from that set would have the newly added labels. This means that existing machines (nodes) will not get the label. Therefore, to make sure all nodes have the label, you should scale the machine set down to zero and then scale the machine set back up.
#### Using the web console
Select "MachineSets" from the left menu. You will see the list of machinesets.
We'll select the first one "ok0620-rq5tl-worker-westus21"
Click on the second tab "YAML"
Click into the YAML and under `spec.template.metadata.labels` add a key:value pair for the label you want. In our example we can add a label "tier: frontend". Click Save.
The already existing machine won't get this label but any new machines will. So to ensure that all machines get the label, we will scale down this machine set to zero, then once completed we will scale it back up as we did earlier.
