# Upstream differences This file catalogues the differences of install approach between ARO and upstream OCP. ## Installer carry patches See https://github.com/openshift/installer/compare/release-4.10...jewzaam:release-4.10-azure. ## Installation differences * ARO does not use Terraform to create clusters, and instead uses ARM templates directly * ARO persists the install graph in the cluster storage account in a new "aro" container / "graph" blob. * No managed identity (for now). * No IPv6 support (for now). * Upstream installer closely binds the installConfig (cluster) name, cluster domain name, infra ID and Azure resource name prefix. ARO separates these out a little. The installConfig (cluster) name and the domain name remain bound; the infra ID and Azure resource name prefix are taken from the ARO resource name. * API server public IP domain name label is not set. * ARO uses first party RHCOS OS images published by Microsoft. * ARO never creates xxxxx-bootstrap-pip-* for bootstrap VM, or the corresponding NSG rule. * ARO does not create a outbound-provider Service on port 27627. * ARO deploys a private link service in order for the RP to be able to communicate with the cluster. * ARO runs a dnsmasq service on the nodes through the use of a machineconfig to resolve api-int and *.apps domains on the node locally allowing for custom DNS configured on the VNET. # Introducing new OCP release into ARO RP To support a new version of OpenShift on ARO, you will need to reconcile [upstream changes](https://github.com/openshift/installer) with our [forked installer](https://github.com/jewzaam/installer-aro). This will not be a merge, but a cherry-pick of patches we've implemented. ## Update installer fork To bring new OCP release branch into ARO installer fork: 1. Assess and document differences in X.Y and X.Y-1 in upstream ```sh # clone our forked installer git clone https://github.com/jewzaam/installer-aro.git cd installer-aro # add the upstream as a remote source git remote add upstream https://github.com/openshift/installer.git git fetch upstream -a # diff the upstream X.Y with X.Y-1 and search for architecture changes git diff upstream/release-X.Y-1 upstream/release-X.Y # pay particular attention to Terraform files, which may need to be moved into ARO's ARM templates git diff upstream/release-X.Y-1 upstream/release-X.Y */azure/*.tf ``` 2. Create a new X.Y release branch in our forked installer ```sh # create a new release branch in the fork based on the upstream git checkout upstream/release-X.Y git checkout -b release-X.Y-azure ``` 3. If there is a golang version bump in this release, modify `./hack/build.sh` and `./hack/go-test.sh` with the new version, then verify these scripts still work and commit them 4. Determine the patches you need to cherry-pick, based on the last (Y-1) release ```sh # find commit shas to cherry-pick from last time git checkout release-X.Y-1-azure git log ``` 5. For every commit you need to cherry-pick (in-order), do: ```sh # WARNING: when you reach the commit for `commit data/assets_vfsdata.go`, look ahead git cherry-pick abc123 # may require manually fixing a merge ./hack/build.sh # fix any failures ./hack/go-test.sh # fix any failures # if you had to manually merge, you can now `git cherry-pick --continue` ``` - When cherry-picking the specific patch `commit data/assets_vfsdata.go`, instead run: ```sh git cherry-pick abc123 # may require manually fixing a merge ./hack/build.sh # fix any failures ./hack/go-test.sh # fix any failures # if you had to manually merge, you can now `git cherry-pick --continue` pushd ./hack/assets && go run ./assets.go && popd ./hack/build.sh # fix any failures ./hack/go-test.sh # fix any failures git add data/assets_vfsdata.go git commit --amend ``` **Note:** If any changes are required during the process, make sure to amend the relevant patch or create a new one. Each commit should be atomic/complete - you should be able to cherry-pick it into the upstream installer and bring the fix or feature it carries in full, without a need to cherry-pick additional commits. This makes it easier to understand the nature of the patch as well as contribute our carry patches back to the upstream installer. # Update ARO-RP Once installer fork is ready: 1. Update `go mod edit -replace` calls in `hack/update-go-module-dependencies.sh` to use a new release-X.Y branch. * Make sure to read comments in the script. 1. `make vendor`. * You most likely will have to make changes to the RP codebase at this point to adjust it to new versions of dependencies. * Also you likely will have to repeat this step several time until you resolve all conflicting dependencies. Follow `go mod` failures, which will tell you what module requires what other module. You will probably need to look at the `go.mod` files of these modules and see whether they set own replace directives, as the script is likely to fail with something like this: ``` go: github.com/openshift/installer@v0.16.1 requires github.com/openshift/cluster-api-provider-kubevirt@v0.0.0-20201214114543-e5aed9c73f1f requires kubevirt.io/client-go@v0.0.0-00010101000000-000000000000: invalid version: unknown revision 000000000000 ``` In the example above you need to: * Checkout `github.com/openshift/cluster-api-provider-kubevirt` at commit `e5aed9c73f1f`. * In go.mod find a replace directive for `kubevirt.io/client-go`. * Add/update relevant replace directive in ARO-RP `go.mod`. 1. `make generate`. 1. Update `pkg/util/version/const.go` to point to the new release. * You should be able to find latest published release and image hash [on quay.io](https://quay.io/repository/openshift-release-dev/ocp-release?tab=tags). 1. Publish RHCOS image. See [this document](./publish-rhcos-image.md). 1. After this point, you should be able to create a dev cluster using the RP and it should use the new release. 1. `make discoverycache`. * This command requires a running cluster with the new version. 1. The list of the hard-coded namespaces in `pkg/util/namespace/namespace.go` needs to be updated regularly as every minor version of upstream OCP introduces a new namespace or two.