* [tests] Add make target to manually launch a device test run for the current commit.
* Upload a manifest that can be found using only the hash, and use it to find the packages.
* Only notarize CI builds; timeout after 90 min
* Update jenkins/Jenkinsfile
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Don't upload notarized pkgs if it fails
* Update jenkins/Jenkinsfile
Co-Authored-By: Manuel de la Pena <mandel@microsoft.com>
* Automatically build mono from source if any of the archives aren't available.
* [jenkins] Show in the PR report if mono was built from source.
* [jenkins] Make sure we got everything after running configure.
* Add a dummy x86_64 slice to all our native libraries that don't have one. (#6848)
Apple's notarization tool has a bug where they incorrectly flag Mach-O
binaries without an x86_64 slice, so make sure all our libraries have one.
* Jenkinsfile notarization (#6869)
* Add in notarization script for xamarin.mac/xamarin.iOS
* Flatten the list to get rid of the braces
* Add in keychain password
* Add login.keychain back in to access codesigning certificates
* Always sign pkgs, upload notarized copies
* Enable ios notarization and make notarized pkgs public
* Make notarization non-fatal
* Publish GH statuses for notarized PKGs
* Don't forget to declare URI variables for notarized pkgs
* report proper package links
* [jenkins] Improve package reporting.
* Use dummy function name which our tests won't complain about.
* Add in notarization script for xamarin.mac/xamarin.iOS
* Flatten the list to get rid of the braces
* Add in keychain password
* Add login.keychain back in to access codesigning certificates
* Always sign pkgs, upload notarized copies
* Enable ios notarization and make notarized pkgs public
* Make notarization non-fatal
* Publish GH statuses for notarized PKGs
* Don't forget to declare URI variables for notarized pkgs
* report proper package links
* [jenkins] Improve package reporting.
brew may fail to install 7z if 7z is already installed, so ignore any brew
failures when trying to install 7z.
If 7z really is non-functional on the machine, then it'll fail later anyway.
* macOS 10.15 starts putting up permission dialogs we can't automatically
dismiss anymore, so start honoring the 'IncludeSystemPermissionTests' option
for macOS tests.
* Improve the 'IncludeSystemPermissionTests' option to have three states: if
set (either true or false), that takes precedence, but if not set, we now
don't run any tests that require permission dialogs on macOS or on device if
we're running in CI. Tests executed locally will still put up dialogs, both
on macOS and on device.
* This needed a few changes to the html report, since the
'IncludeSystemPermissionTests' is exposed in the UI and the code didn't
handle the three different states.
* Update a few tests to check for permission to the contacts.
Fixes https://github.com/xamarin/maccore/issues/1856.
Many people are running into problems building xcode11 because they have to
"./configure --disable-packaged-mono" first.
So disable the packaged mono by default (and add a way to enable it again
manually with "./configure--disable-packaged-mono=no").
We have a 13-hour timeout for running the tests [1], but that won't work if
the timeout for the entire Jenkins pipeline is lower, so adjust the timeout
for the entire Jenkins pipeline to be more than the sum of all the nested
timeouts.
[1] 94fe39b118 (diff-68c8473bbe1d4346ecff3d74522a31f8R675)
This updates the project generation. We cannot yet fully remove the submodule because:
* We are missing the xunit dlls which should be added in the SDK.
* We have not yet remove all the old style tests. Would make the PR huge, better to deal with it in a diff PR.
* The xunit CoreLib tests have issues loading all the tests, needs some extra work and again, the PR is already large.
Fixes: xamarin/maccore#1199Fixes: xamarin/maccore#1204Fixes: xamarin/maccore#1209Fixes: xamarin/maccore#1510
We had issues in the code that adds a type found in an assembly to
ensure that it was not removed by the linker. This resulted in some
assemblies having 0 tests.
Added the needed ignore for the corlib tests and system ones.
* [builds] Fix race condition when building device runtimes.
We need to complete the corresponding cross builds first, because the cross
builds will also try to configure device builds, so we might end up with two
make processes trying to configure the same target simulatenously.
* [jenkins] Add support for building mono from source in pull requests by using labels.
* [jenkins] Add support to build.sh for timing out the build.
* Test timeout support
* [jenkins] Tell the world what happened.
* [jenkins] Minor announcement fix + disable test mode.
* [tests] Add sample tester.
Add a unit project that looks for iOS/macOS/tvOS sample projects in several
repositories, and builds them all.
* [tests][sampletester] Remove known issue which has now been fixed.
* [tests] Only run sample tests on CI in Azure Devops.
* Remove the possibility of automatically running the sample tests with
xharness (so the sample tests won't run on PR bots or internal bots when the
'run-all-tests' label is added). It's still possible to run the sample tests
manually from the xharness web UI.
* Automatically trigger the sample test run in Azure Devops if the
'run-sample-tests' label is applied to a PR (and that PR is executed on
internal Jenkins).
* Fix typo.
* Fix path.
* Verbose output to track down scheduling failure.
* Bump maccore to get improved debug spew.
Diff: f527c9c526..f89d74b165
* [tests][sampletester] Fix build for TodoWCF.
Builds take approximately 30 minutes on a bot now, so 1 hour should be plenty.
This makes sure a 32-bit dialog doesn't waste 9h of bot time just sitting
there.
Updates the jenkinsFile to include generating ci-checkout.json and dependency-info.json.
ci-checkout.json is a list of all of the repositories included in the workspace of the project and their respective hashs.
dependency-info.json includes all of the info in ci-checkout.json, as well as a scan of the containing cs/fsproj files and their containing package and project references.
Fixes this warning:
make: *** tools/siminstaller: No such file or directory. Stop.
Can't check if simulators are available, because siminstaller failed to build.
* [jenkins] Don't execute the packaged XM tests on the main macOS version.
Don't execute the packaged XM tests on the main macOS version, since it's redundant.
It also prevents a potential test deadlock if all main bots are busy.
* Rework to make it show better in the Jenkins UI.
Jenkins has a limitation where you can't mark a step a failure, it has to
*fail* to be reported as such.
This means that running multiple tests, and reporting a failure if any of
those tests fail is not possible.
We run into this with the normal test run + the docs tests; where we currently
don't show a red result in the UI if any of those fail.
So incorporate the test-docs step into xharness, so that we only have one
thing to in the Jenkinsfile, which makes it possible for us to fail the test
run step properly.
This also required a few upgrades to xharness to get more info for pull
requests, since the logic to enable the docs tests is a bit more complicated
than anything else we have (if the current branch (or the target branch for a
PR) is 'master' AND xamarin mode is enabled).
Fix retry-download-on-error-56 logic to not fail forever if the first time
failed by resetting the EC variable after each try.
Fixes https://github.com/xamarin/maccore/issues/1098 (third time's the charm!)
* [jenkins] Don't execute the packaged XM tests on the main macOS version.
Don't execute the packaged XM tests on the main macOS version, since it's redundant.
It also prevents a potential test deadlock if all main bots are busy.
* Rework to make it show better in the Jenkins UI.
* Bump maccore to get fix for test-docs.
Commit list for xamarin/maccore:
* xamarin/maccore@6e9b63e537 Merge pull request #1162 from rolfbjarne/docfixer-fixes
* xamarin/maccore@4cf1d63b7b [docfixer] Create project files and fix a few issues. Fixes#1118.
* xamarin/maccore@714bd73a55 [docfixer] Avoid declaring the same target twice.
* xamarin/maccore@34f4bfa339 [populate] Directories need to call Directory.Exists for existence checks.
* xamarin/maccore@341333db88 [docs] Remove truly ancient and outdated targets to publish updated docs.
Diff: 7ba9c5a962...6e9b63e537
* [jenkins] Make it possible to force docs testing by applying a label to pull requests.
This means we'll get an empty diff if there are no generator changes, so
update the logic that detects/reports empty generator diffs.
Also there's no need to ignore mdbs anymore, since we don't create mdbs.
* [builds] Adjust ifdefs to fix not building device architectures. Fixes maccore#1074.
Fixes https://github.com/xamarin/maccore/issues/1074.
* [jenkins] Running configure and then cleaning everything is kind of useless, so reverse the order.
* [builds] Adjust ifdefs to fix not building device architectures. Fixes maccore#1074.
Fixes https://github.com/xamarin/maccore/issues/1074.
* [jenkins] Running configure and then cleaning everything is kind of useless, so reverse the order.
Our tests can take quite a while to run sometimes, and this makes sure we
don't get random failures when the tests that access the keychain are executed
at the end of the test run.
Bump maccore to get the same fix there.
Fixes https://github.com/xamarin/maccore/issues/973.
Commit diff for maccore bump: 9937926f56...07fde84362
Something in VSTS changed, and now fake branch names don't work anymore.
So instead use real branch names (and for pull requests I've created a
'pull-request' branch we can use).
We keep track of the current stage by using the 'currentStage' variable, so that we can properly report what failed.
There were a couple of problems with this:
* The 'Package Xamarin.Mac tests' is not a fatal step, which means that the
pipeline will continue executing even if it fails. In this case the previous
code would assign any failure to package the XM tests to the last stage
('Test docs').
* Something similar seems to happen if there are failures in the Xamarin.Mac
tests executed in parallel (on other bots), where any failure would be
reported as a failure in the last stage ('Test docs').
* No support for reporting failure in multiple stages.
So do a couple of things:
* Clear out the 'currentStage' variable at the end, and handle it being empty
when reporting results. This should solve at least some of the problems
where blame as being assigned incorrectly (we'll probably run into cases
where blame isn't assigned at all, but this is still half the way of getting
it right).
* Add a 'failedStages' variables where we can keep track of (and report)
multiple failed stages.
Implement support for slack notifications.
Notifications will only be sent when something fails, and unfortunately I wasn't able to make notifications ping due to https://github.com/jenkinsci/slack-plugin/issues/353.
Add support for the 'run-internal-tests' label on pull requests to indicate
that the internal jenkins should run tests when that label is applied.
Also make it so that either the 'run-internal-tests' or the 'build-package'
label will actually make the internal jenkins execute (otherwise the 'run-
internal-tests' label would also require the 'build-package' label, which
wouldn't be very obvious/user-friendly).
This can be useful when working on the support for private jenkins, since many
of those changes can only be tested as a pull request, and they also tend to
include a lot of commits (because nothing can be tested locally).
So until whatever needs implementing for private jenkins is complete and
working, there's no need for the public bots to do anything for such pull
requests.
* [Jenkins] Create artifacts.json and set a GH status as 'Jenkins: Artifacts'.
* [jenkins] Include the url in artifacts.json
* [jenkins] Add sha256 checksum to artifacts.json as well.
* [Jenkins] Enable xamarin before provisioning so that we auto-provision Xcode.
* [jenkins] Fix passing flags to configure.
Quoting empty CONFIGURE_FLAGS ends up doing this:
./configure "" --disable-ios-device
and since configure parses arguments until it finds an empty argument, it
would stop parsing at the first argument, effectively not disabling the device
build.
So don't quote CONFIGURE_FLAGS when invoking configure. shellcheck doesn't
quite like this, but the better code is much more complex, and not really
needed, so just add an exception.
Add support for building on internal Jenkins.
Jenkins has been configured to build every branch on xamarin/xamarin-macios that contains a `jenkins/Jenkinsfile`, which means it will start working as soon as this PR is merged.
Results will be posted as statuses on each commit, which can be viewed using the url `https://github.com/xamarin/xamarin-macios/commits/<branch>`:
* The `continuous-integration/jenkins/branch` status links to the jenkins job.
* The other two are XI and XM packages (the `Jenkins-` prefix will be removed once we officially switch from Wrench to Jenkins).
More detailed information will be added as a comment to each commit, which can be seen by clicking on the commit and scrolling to the bottom (url of the format `https://github.com/xamarin/xamarin-macios/commit/<sha1>`)
Unfortunately GitHub does not display the commit statuses when viewing a single commit, so to view those statuses you'll have to view the list of commits (the `/commits/` url). Tip: it's possible to use `<sha1>` instead of `<branch>` (and vice versa for that matter) if you're interested in the statuses of a particular commit.
Pull requests will also be built (only from contributors with write access), but by default nothing will be done (the job will exit immediately, although a green check mark will still show up). Jenkins will **not** add a comment in the pull request in this case.
However, if the label `build-package` [1] is set for a pull request, the internal jenkins job will run (it will do everything except the local xharness test run: this includes creating and publishing packages, creating various diffs, run tests on older macOS versions, test docs, etc). A detailed comment will also be added to the pull request (see below for multiple examples), which means that there will be two Jenkins comments: one for the public Jenkins which builds every PR, and one for the internal Jenkins [2].
[1] I don't quite like the name of the label, because it doesn't get even close to explain all that will actually happen, but `run-on-internal-jenkins-and-create-package` is a bit too long IMHO... Also it's non-obvious that this is the label to apply if the reason for executing on the internal jenkins is some other reason (for instance to test a maccore bump). Other ideas:
* `run-internal-jenkins`: doesn't make it obvious that a package will be created (which is probably the most common reason to want to run on internal jenkins)
* We could have multiple labels that mean the same thing: `build-package`, `internal-build`, `run-internal-jenkins`, etc, but it's redundant and I don't quite like it either.
* Any other ideas?
[2] I'm noticing now that these two look quite similar and this might end up confusing (the main difference is that the comment from the public jenkins will say **Build success/failure** and **Build comment file:** at the top. If something goes wrong the failure will also show up differently). Should this be made clearer?
For the api diff for this PR we now show:
* `🔥 breaking changes 🔥`: If there are any breaking changes in the api diff (for this PR).
* `please review changes`: If there are any non-breaking changes in the api diff.
* `no change`: If the api diff is empty.
For the generator diff we show:
* `only version changes`: If there were only changes related to version numbers (since the XI/XM version number is added to the generator, that version number will always show up as a diff when comparing the generated source code)
* `please review changes`: If anything other that version numbers changed in the generated source code.
A few general categories of fixes:
* Sprinkle lots of quotes everywhere.
* Don't use environment variables in the format string to printf, instead pass them as arguments.
* Don't use backticks to execute commands (it's deprecated), use the new "$(...)" syntax instead.
Implement support for skipping API comparison by applying a label to a pull
request.
This also required some refactoring to move existing code to fetch the labels
for a pull request to a separate script.
We can't create an API/generator diff for a pull request with conflicts, so
detect this scenario and show a better error than this (from #3961):
Comparing the changes between origin/pr/3961/merge^1 and HEAD:
fatal: ambiguous argument 'origin/pr/3961/merge^1..HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument 'origin/pr/3961/merge^1': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Rolf Bjarne Kvinge
Connor Adsit
Alexander Köplinger
Manuel de la Pena
Vincent Dondain
Tim Miller
Sam Schwarz
Xamarin Jenkins User
Sebastien Pouliot
