We keep track of the current stage by using the 'currentStage' variable, so that we can properly report what failed.
There were a couple of problems with this:
* The 'Package Xamarin.Mac tests' is not a fatal step, which means that the
pipeline will continue executing even if it fails. In this case the previous
code would assign any failure to package the XM tests to the last stage
('Test docs').
* Something similar seems to happen if there are failures in the Xamarin.Mac
tests executed in parallel (on other bots), where any failure would be
reported as a failure in the last stage ('Test docs').
* No support for reporting failure in multiple stages.
So do a couple of things:
* Clear out the 'currentStage' variable at the end, and handle it being empty
when reporting results. This should solve at least some of the problems
where blame as being assigned incorrectly (we'll probably run into cases
where blame isn't assigned at all, but this is still half the way of getting
it right).
* Add a 'failedStages' variables where we can keep track of (and report)
multiple failed stages.
Implement support for slack notifications.
Notifications will only be sent when something fails, and unfortunately I wasn't able to make notifications ping due to https://github.com/jenkinsci/slack-plugin/issues/353.
Add support for the 'run-internal-tests' label on pull requests to indicate
that the internal jenkins should run tests when that label is applied.
Also make it so that either the 'run-internal-tests' or the 'build-package'
label will actually make the internal jenkins execute (otherwise the 'run-
internal-tests' label would also require the 'build-package' label, which
wouldn't be very obvious/user-friendly).
* [Jenkins] Create artifacts.json and set a GH status as 'Jenkins: Artifacts'.
* [jenkins] Include the url in artifacts.json
* [jenkins] Add sha256 checksum to artifacts.json as well.
* [Jenkins] Enable xamarin before provisioning so that we auto-provision Xcode.
* [jenkins] Fix passing flags to configure.
Quoting empty CONFIGURE_FLAGS ends up doing this:
./configure "" --disable-ios-device
and since configure parses arguments until it finds an empty argument, it
would stop parsing at the first argument, effectively not disabling the device
build.
So don't quote CONFIGURE_FLAGS when invoking configure. shellcheck doesn't
quite like this, but the better code is much more complex, and not really
needed, so just add an exception.
Add support for building on internal Jenkins.
Jenkins has been configured to build every branch on xamarin/xamarin-macios that contains a `jenkins/Jenkinsfile`, which means it will start working as soon as this PR is merged.
Results will be posted as statuses on each commit, which can be viewed using the url `https://github.com/xamarin/xamarin-macios/commits/<branch>`:
* The `continuous-integration/jenkins/branch` status links to the jenkins job.
* The other two are XI and XM packages (the `Jenkins-` prefix will be removed once we officially switch from Wrench to Jenkins).
More detailed information will be added as a comment to each commit, which can be seen by clicking on the commit and scrolling to the bottom (url of the format `https://github.com/xamarin/xamarin-macios/commit/<sha1>`)
Unfortunately GitHub does not display the commit statuses when viewing a single commit, so to view those statuses you'll have to view the list of commits (the `/commits/` url). Tip: it's possible to use `<sha1>` instead of `<branch>` (and vice versa for that matter) if you're interested in the statuses of a particular commit.
Pull requests will also be built (only from contributors with write access), but by default nothing will be done (the job will exit immediately, although a green check mark will still show up). Jenkins will **not** add a comment in the pull request in this case.
However, if the label `build-package` [1] is set for a pull request, the internal jenkins job will run (it will do everything except the local xharness test run: this includes creating and publishing packages, creating various diffs, run tests on older macOS versions, test docs, etc). A detailed comment will also be added to the pull request (see below for multiple examples), which means that there will be two Jenkins comments: one for the public Jenkins which builds every PR, and one for the internal Jenkins [2].
[1] I don't quite like the name of the label, because it doesn't get even close to explain all that will actually happen, but `run-on-internal-jenkins-and-create-package` is a bit too long IMHO... Also it's non-obvious that this is the label to apply if the reason for executing on the internal jenkins is some other reason (for instance to test a maccore bump). Other ideas:
* `run-internal-jenkins`: doesn't make it obvious that a package will be created (which is probably the most common reason to want to run on internal jenkins)
* We could have multiple labels that mean the same thing: `build-package`, `internal-build`, `run-internal-jenkins`, etc, but it's redundant and I don't quite like it either.
* Any other ideas?
[2] I'm noticing now that these two look quite similar and this might end up confusing (the main difference is that the comment from the public jenkins will say **Build success/failure** and **Build comment file:** at the top. If something goes wrong the failure will also show up differently). Should this be made clearer?
Rolf Bjarne Kvinge
