Inspiration was taken from other yaml pipelines but those multiple jobs are not behaving the way I expected.
1. Each job will clone `xamarin-macios` in a dedicated folder (for 3 jobs you'll have `./1/s`; `./2/s`; `./3/s`)
2. The goal of the last job was to upload the `TestSummary.md` but because of the cloning strategy it couldn't find it. There might be ways to pass files (artifacts?) between jobs but there's no need to be that fancy right now.
3. Cloning takes time so I'm assuming having everything in one job can actually save us some time.
- We had `./jenkins/vsts-device-tests-set-status.sh: line 88: HTML_REPORT: unbound variable` in the `Report results to GitHub as comment / status` step. Now that the HTML_REPORT is set this shouldn't happen anymore.
- The `Add summaries` task is reporting errors but it's working regardless. I believe this might be due to -x being set instead of the original +x (I changed that by mistake).
- Needed for the DDFun migration of our tests
- Don't use SSH because it's restricted on the new bots
- Use brand new DevOps feature to clone multiple repos via HTTPS
- Update names of all tasks (clearer)
- Simplify and re-use code
- Update all paths because our cloning strategy is different compared to old pipeline
- Update xharness to print "experimental" titles in GitHub comments
- Update xharness to not add the Html link for tests from the DDFun pipeline
- Fix all timeouts to be relevant to the tests length
- Split in multiple jobs (for GitHub status)
- Move longer scripts to independant files
- Make inline bash prettier
- Use a variable for the pool name
- Fix single space alignment
- Remove unused variable in `vsts-device-tests-set-status.sh` that broke it
- Bump maccore to use the right certificates and profiles
- Bump maccore to target the right pipeline
- Bump maccore to not use ssh in scripts
- Bump maccore and use persistCredentials
Notes:
- Right now the xharness test output can only be seen when downloading the zip file locally (better output soon)
* [sample-tester] Collect app size and build duration statistics.
* Always publish performance data.
* [mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_OPTIONS.
* [sample-tester] Make mmp/mtouch show timing information, and get the diagnostic msbuild log.
* Collect task and target info.
* Add more perf data and a baseline test.
* Redo the base test a bit.
* More tweaks.
* More tweaks.
* Huh?
* Debug stuff.
* diagnostics.
* Better diagnostics.
* cleanup
* tmp
* Use a separate stage to push data.
* Next attempt.
* Unique artifact names.
* Make the right name unique.
* More progress
* Improvements
* Don't need new maccore.
* cleanup
* Remove debug spew.
* Realign stuff.
* duh
* More path fixes.
* zippity zip.
* Show publishing errors.
* dependencies
* Not fake results anymore.
* Tweaks
* Merge xml files
* [sampletester] Bump nuget timeout to 5 minutes.
The TodoREST.iOS test seems to use a lot of packages, and it can apparently
take a while to restore them, so give a few more minutes.
* Improvements.
* Fix glob.
* [sampletester] Fix fetching of sample repository and hash.
* [sampletester] Centralize verbosity for mtouch and mmp.
* [tests] Don't use unsupported characters in matrix names for yml scripts. Fixesxamarin/maccore#1831.
Matrix names must be alphanumeric (+underscore), and recently Azure DevOps
stopped working correctly if that wasn't the case (unfortunately without a
good error message though, so it took a while to figure it out).
Fixes https://github.com/xamarin/maccore/issues/1831.
* [jenkins] Fix lookup of environment variables from matrix jobs.
* [tests] Don't use unsupported characters in matrix names for yml scripts. Fixesxamarin/maccore#1831.
Matrix names must be alphanumeric (+underscore), and recently Azure DevOps
stopped working correctly if that wasn't the case (unfortunately without a
good error message though, so it took a while to figure it out).
Fixes https://github.com/xamarin/maccore/issues/1831.
* [jenkins] Fix lookup of environment variables from matrix jobs.
Add a separate provisioning script to install Xcode if it's not already installed on the bot.
For some unknown reason it needs to be a separate script, otherwise the provisionator will complain it doesn't know the required GitHub token to download Xcode.
Fixes https://github.com/xamarin/xamarin-macios/issues/6326.
* [tests] Add sample tester.
Add a unit project that looks for iOS/macOS/tvOS sample projects in several
repositories, and builds them all.
* [tests][sampletester] Remove known issue which has now been fixed.
* [tests] Only run sample tests on CI in Azure Devops.
* Remove the possibility of automatically running the sample tests with
xharness (so the sample tests won't run on PR bots or internal bots when the
'run-all-tests' label is added). It's still possible to run the sample tests
manually from the xharness web UI.
* Automatically trigger the sample test run in Azure Devops if the
'run-sample-tests' label is applied to a PR (and that PR is executed on
internal Jenkins).
* Fix typo.
* Fix path.
* Verbose output to track down scheduling failure.
* Bump maccore to get improved debug spew.
Diff: f527c9c526..f89d74b165
* [tests][sampletester] Fix build for TodoWCF.
Vincent Dondain
Rolf Bjarne Kvinge
monojenkins
Alexander Köplinger
Alex Soto