Since 'packages' is a common dir name that is ignored. Revert the change
in the .gitignore and rename the template path since it just means a one
liner change in the entry yaml file.
fixes: https://github.com/xamarin/maccore/issues/2359
The prefix of the location of the logs depends on the platform where it
was executed (else we step on them) but that was not added in the env
var use by xharness.
fixes: https://github.com/xamarin/maccore/issues/2349
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The GitHup url to be used to create comments in PRs is diff to the one
for comments. Use the build reason AND the changeID to identify if we
are building due to a PR and use the correct url.
fixes: https://github.com/xamarin/maccore/issues/2356
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Alex Soto <alex@alexsoto.me>
The wrong variable was used and therefore the dir path was used instead
of the mac pkg name.
fixes: https://github.com/xamarin/maccore/issues/2353
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The GitHup url to be used to create comments in PRs is diff to the one
for comments. Use the build reason AND the changeID to identify if we
are building due to a PR and use the correct url.
fixes: https://github.com/xamarin/maccore/issues/2356
Co-authored-by: Alex Soto <alex@alexsoto.me>
The prefix of the location of the logs depends on the platform where it
was executed (else we step on them) but that was not added in the env
var use by xharness.
fixes: https://github.com/xamarin/maccore/issues/2349
Using provisionator could result in the UI prompt being shown which will
make the pipeline hang until there is human interaction.
Move the step to be one of the first ones to ensure provisionator does
not get stuck.
* Fix links that point to master to point to main instead.
* Implement support in the sample tester for specifying the default branch for
each sample repo.
* Fix various text / documentation to say 'main' instead of 'master.'
* Push to 'main' instead of 'master' in xamarin-macios-data.
* Fix xharness to make 'main' the special branch with regards to documentation tests as opposed to 'master'.
* Fix various CI to use 'main' instead of 'master'.
This is a backport of PR #9561
Once we have done all the check of the bot, set the status of the commit
to pending since the next task is going to run the tests.
This way we ensure that the pending status should be cleaned at the end
rather than be left behind.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Once we have done all the check of the bot, set the status of the commit
to pending since the next task is going to run the tests.
This way we ensure that the pending status should be cleaned at the end
rather than be left behind.
The tests are not executed in the same agent that sets the github
status, for that reason, the github comment is not giving the correct
bot name, but the name of the bot that executed the commit message and
not the tests.
Add an output var in the runTests step to set the bot name and pass it
to the comment job in a different bot.
Co-authored-by: Whitney Schmidt <whschm@microsoft.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The credentials for maccore are downloaded to a pat file (to be found).
When we call make git-clean, because we do use the -x options, all
files are deleted, including the pat file.
We move to call git clean -xdf inside xamarin-macios, which will delete
the test result files.
Once we find the exact path pattern, we can update the make git-clean to
not remove them but this commit unblocks the failing CI builds.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The credentials for maccore are downloaded to a pat file (to be found).
When we call make git-clean, because we do use the -x options, all
files are deleted, including the pat file.
We move to call git clean -xdf inside xamarin-macios, which will delete
the test result files.
Once we find the exact path pattern, we can update the make git-clean to
not remove them but this commit unblocks the failing CI builds.
* [DevOps] Report the correct bot used.
The tests are not executed in the same agent that sets the github
status, for that reason, the github comment is not giving the correct
bot name, but the name of the bot that executed the commit message and
not the tests.
Add an output var in the runTests step to set the bot name and pass it
to the comment job in a different bot.
* Update tools/devops/device-tests/scripts/GitHub.psm1
Co-authored-by: Whitney Schmidt <whschm@microsoft.com>
Co-authored-by: Whitney Schmidt <whschm@microsoft.com>
Depending on the pipeline run, we should have the following result:
* failed: issues with bots or provisioning (red in the UI)
* partiallySucceeded: no issues with bots but failing tests.
* succeeded: no issues with bots and all tests pass.
This adds a psh cmdlet that allows to set the result AND sets the
correct result when we have test failures.
Co-authored-by: Whitney Schmidt <whschm@microsoft.com>
Few things:
1. Move from Write-Error to Write-Debug so that we do not set the exit
code and an error.
2. Ig we fail in the Clear-HD step, continue, we should be ok or cancel
if not enough space is found.
* [DevOps] Fix issues with the xm stroage path.
For some reason, and it is very probably a bug in the azurepipelines,
the following bash code creates a path with an extra single quote at the
end:
```bash
P=jenkins/xamarin-macios/$BUILD_LANE/$BUILD_REVISION/$ID/device-tests
echo "##vso[task.setvariable variable=XM_STORAGE_PATH]$P"
```
* Should it do it? Not, it should not.
* Does it do it? Yes, it does.
We work around it via td -d \'\" and remove all single and double quotes
in the string. How long did it take to discover this? More than it
should have.
There is another interesting bug with the variable expansion, the
following
```bash
echo '##vso[task.setvariable variable=XAMARIN_STORAGE_PATH;isOutput=true]$P'
```
Does not equal
```bash
echo '##vso[task.setvariable variable=XAMARIN_STORAGE_PATH;isOutput=true]'$P
```
The first will not expand the variable, the second one will. We do need
the value of $P not '$P'.
Co-authored-by: Whitney Schmidt <51677938+whitneyschmidt@users.noreply.github.com>
The microsoft hosted images have a limit of 10gb, or logs are getting
close to that size when expanded, therefore we will get into issues.
Move back to the self-hosted pool before we have problems.
The upload of the results takes 2 hours:
* 1 hour uploading to vsts test results.
* 1 hour uploading to vsdrops.
This means that we are blocking a device bot for an extra hours (since
the vsdrops is done in a windows bot) when it is not needed. We could
add both uploads to the vsdrops job, but since steps cannot be ran in
parallel, it means that we re waiting for 2 hours when we really do not
have to.
We create two jobs to parallelize the upload of files and we release the
device bot ASAP. That means that our device bots can be free to get new
jobs (and they are the bottle neck) while the windows bots perform the
different uploads.
To make things easier a template that downloads the artifacts was added
so that there is not much code repetition.
* add error handling for storage path
* Revert "add error handling for storage path"
This reverts commit 751e54c254.
* revert whitespace changes, only add real change
* small fixes
Use both in those labs that support xamarin-storage so that we have a
backup. To do so, move the parameter to be a bool but keep the rest of
the logic the same.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Try to create the folder in xamarin-storage, if that fails, rather than
stop with an error, execute the tests and st an env var.
Once the tests are complete, when adding the summaries, check if we did
fail to reach xamarin-storage and notify it.
This will make sure that tests won't fail if:
* xamarin-storage dies.
* bots are not in the vpn and could not reach xamarin-storage.
This commit moves all the different pipelines we have to fully use yaml.
That includes the old ddfun pipeline that was using a mix between using
yaml and the UI interface to configure it.
The new pipelines are found under the tools/devops/device-tests
directory. All of the use a very similar structure and therefore we use
a template. The README.md inside the directory should explain most of
the logic and the devices-tests.yml contains comments for each of the
steps.
Most of the bash scripts have been moved to use pwsh which is inside the
scripts directory and contains tests. Tests can be ran via:
```bash
make -C tools/devops/device-tests/scripts run-tests
```
This commit will only affect the ddfun pipeline that will need to be
updated to use the new yml path. All the other pipelines are not
affected and they will be to be updated manually.
The pipelines support two different storages for the html report.
* xamarin-storage: Old school.
* vsdrops: New age.
Unfortunatly we yet do not know how to workaround the fact that vsdrops
does not serve the files, therefore we use the xamarin-storage in all
pipelines but those in the ddfun lab because they do not have the right
access.
This later can be easily changed editing the
tools/devops/device-tests/tempaltes/cambridge-variables.yml which will
update all the pipelines in the cambridge lab.
PS: Sorry for the size.
* Fix links that point to master to point to main instead.
* Implement support in the sample tester for specifying the default branch for
each sample repo.
* Fix various text / documentation to say 'main' instead of 'master.'
* Push to 'main' instead of 'master' in xamarin-macios-data.
* Fix xharness to make 'main' the special branch with regards to documentation tests as opposed to 'master'.
* Fix various CI to use 'main' instead of 'master'.
* Bump maccore
New commits in xamarin/maccore:
* xamarin/maccore@ed6d146822 Rename 'master' to 'main'. (#2233)
Diff: 424fa26148..ed6d146822
* page through github statuses for manifest URL
* Check to see if the content has values (rather than being the empty string)
* Fix code style.
Co-authored-by: cadsit <connor.adsit@gmail.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Move to use the provisionator and add a make file to generat the script
so that we only have to bump xcode in a single place.
Move all deps to provisionator and make provisionator download and
install the xi pkg that has been built to be tested in the pipeline, we
do not longer use curl to do it.
The previous pattern would match the output files we created from processing
these directories, and somehow the pattern would match such a file (even
though the pattern should only be executed before doing any processing, when
those files don't exist yet, so I don't know exactly why this is happening),
leading us to try to process those files as if they were directories with
sample data in, with the predictable (disastrous) result.
Fixes https://github.com/xamarin/maccore/issues/2202.
The previous pattern would match the output files we created from processing
these directories, and somehow the pattern would match such a file (even
though the pattern should only be executed before doing any processing, when
those files don't exist yet, so I don't know exactly why this is happening),
leading us to try to process those files as if they were directories with
sample data in, with the predictable (disastrous) result.
Fixes https://github.com/xamarin/maccore/issues/2202.
xamarin-storage can not be reachable or a network issue might happen
when we try to provision the dependencies. In that case, skip those
tasks that are skipped when we cannot provision the certs.
The provisioning dependencies will only execute in the provisioning
profiles was successful, therefore, the if statement is not stepping in
any value that was set by the profiles step.
Add timeouts to the steps to catch possible issues when a step takes
longer than expected. Numbers have been taken from common runs and
rounded up a little to have some buffer.
In cases where a `Make.config` doesn't exist, there's an unhandled exception and the user friendly one cannot be reached.
Arguably, an edge case not applicable to xamarin-macios I discovered when using that code in an other context (where obviously I don't have a Make.config :P). Still this is making the code more robust (;
It seems we only want to provision a few specific things, so modify the
selection logic to first ignore everything, then enable what we want.
This way we don't have to add new ignore flags here every time we add
something new to the provisioning script.
We also don't need to check if 7z and python3 provisioning is possible, since
we already know that (by manually checking the system-dependencies.sh script
for the current hash).
If the pipeline is set on debug, to be able to see any issues, propagate
that to the make calls so that we can also get the information there,
else debugging issues with the pipelines + make is really hard.
Some of the bots fail to do the checkout (miss configuration). The clean
up step is always executed and assumes the pressence of a script, which
will fail since the script is not there.
The script is small, there is no need to add the rm in an extra file
that needs to be checkout.
This removes an extra warning that is set in the pipeline which is noise
when monitoring.
If the bot could no get the provisioning profiles installed, there is no
reason for certain tasks to run since they are all going to fail. This
adds A LOT of noise in the pipeline for the monitoring person to check
when there is no reason.
Move to use a template for the DDFun pipeline. Copy the current pipeline
to a template, set some parameters and recreate the pipeline importing
the template.
* [sample-tester] Collect app size and build duration statistics.
* Always publish performance data.
* [mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_OPTIONS.
* [sample-tester] Make mmp/mtouch show timing information, and get the diagnostic msbuild log.
* Collect task and target info.
* Add more perf data and a baseline test.
* Redo the base test a bit.
* More tweaks.
* More tweaks.
* Huh?
* Debug stuff.
* diagnostics.
* Better diagnostics.
* cleanup
* tmp
* Use a separate stage to push data.
* Next attempt.
* Unique artifact names.
* Make the right name unique.
* More progress
* Improvements
* Don't need new maccore.
* cleanup
* Remove debug spew.
* Realign stuff.
* duh
* More path fixes.
* zippity zip.
* Show publishing errors.
* dependencies
* Not fake results anymore.
* Tweaks
* Merge xml files
* [sampletester] Bump nuget timeout to 5 minutes.
The TodoREST.iOS test seems to use a lot of packages, and it can apparently
take a while to restore them, so give a few more minutes.
* Improvements.
* Fix glob.
* [sampletester] Fix fetching of sample repository and hash.
* [sampletester] Centralize verbosity for mtouch and mmp.
Inspiration was taken from other yaml pipelines but those multiple jobs are not behaving the way I expected.
1. Each job will clone `xamarin-macios` in a dedicated folder (for 3 jobs you'll have `./1/s`; `./2/s`; `./3/s`)
2. The goal of the last job was to upload the `TestSummary.md` but because of the cloning strategy it couldn't find it. There might be ways to pass files (artifacts?) between jobs but there's no need to be that fancy right now.
3. Cloning takes time so I'm assuming having everything in one job can actually save us some time.
- We had `./jenkins/vsts-device-tests-set-status.sh: line 88: HTML_REPORT: unbound variable` in the `Report results to GitHub as comment / status` step. Now that the HTML_REPORT is set this shouldn't happen anymore.
- The `Add summaries` task is reporting errors but it's working regardless. I believe this might be due to -x being set instead of the original +x (I changed that by mistake).
- Needed for the DDFun migration of our tests
- Don't use SSH because it's restricted on the new bots
- Use brand new DevOps feature to clone multiple repos via HTTPS
- Update names of all tasks (clearer)
- Simplify and re-use code
- Update all paths because our cloning strategy is different compared to old pipeline
- Update xharness to print "experimental" titles in GitHub comments
- Update xharness to not add the Html link for tests from the DDFun pipeline
- Fix all timeouts to be relevant to the tests length
- Split in multiple jobs (for GitHub status)
- Move longer scripts to independant files
- Make inline bash prettier
- Use a variable for the pool name
- Fix single space alignment
- Remove unused variable in `vsts-device-tests-set-status.sh` that broke it
- Bump maccore to use the right certificates and profiles
- Bump maccore to target the right pipeline
- Bump maccore to not use ssh in scripts
- Bump maccore and use persistCredentials
Notes:
- Right now the xharness test output can only be seen when downloading the zip file locally (better output soon)
* [sample-tester] Collect app size and build duration statistics.
* Always publish performance data.
* [mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_OPTIONS.
* [sample-tester] Make mmp/mtouch show timing information, and get the diagnostic msbuild log.
* Collect task and target info.
* Add more perf data and a baseline test.
* Redo the base test a bit.
* More tweaks.
* More tweaks.
* Huh?
* Debug stuff.
* diagnostics.
* Better diagnostics.
* cleanup
* tmp
* Use a separate stage to push data.
* Next attempt.
* Unique artifact names.
* Make the right name unique.
* More progress
* Improvements
* Don't need new maccore.
* cleanup
* Remove debug spew.
* Realign stuff.
* duh
* More path fixes.
* zippity zip.
* Show publishing errors.
* dependencies
* Not fake results anymore.
* Tweaks
* Merge xml files
* [sampletester] Bump nuget timeout to 5 minutes.
The TodoREST.iOS test seems to use a lot of packages, and it can apparently
take a while to restore them, so give a few more minutes.
* Improvements.
* Fix glob.
* [sampletester] Fix fetching of sample repository and hash.
* [sampletester] Centralize verbosity for mtouch and mmp.
* [tests] Don't use unsupported characters in matrix names for yml scripts. Fixesxamarin/maccore#1831.
Matrix names must be alphanumeric (+underscore), and recently Azure DevOps
stopped working correctly if that wasn't the case (unfortunately without a
good error message though, so it took a while to figure it out).
Fixes https://github.com/xamarin/maccore/issues/1831.
* [jenkins] Fix lookup of environment variables from matrix jobs.
* [tests] Don't use unsupported characters in matrix names for yml scripts. Fixesxamarin/maccore#1831.
Matrix names must be alphanumeric (+underscore), and recently Azure DevOps
stopped working correctly if that wasn't the case (unfortunately without a
good error message though, so it took a while to figure it out).
Fixes https://github.com/xamarin/maccore/issues/1831.
* [jenkins] Fix lookup of environment variables from matrix jobs.
Add a separate provisioning script to install Xcode if it's not already installed on the bot.
For some unknown reason it needs to be a separate script, otherwise the provisionator will complain it doesn't know the required GitHub token to download Xcode.
Fixes https://github.com/xamarin/xamarin-macios/issues/6326.
* [tests] Add sample tester.
Add a unit project that looks for iOS/macOS/tvOS sample projects in several
repositories, and builds them all.
* [tests][sampletester] Remove known issue which has now been fixed.
* [tests] Only run sample tests on CI in Azure Devops.
* Remove the possibility of automatically running the sample tests with
xharness (so the sample tests won't run on PR bots or internal bots when the
'run-all-tests' label is added). It's still possible to run the sample tests
manually from the xharness web UI.
* Automatically trigger the sample test run in Azure Devops if the
'run-sample-tests' label is applied to a PR (and that PR is executed on
internal Jenkins).
* Fix typo.
* Fix path.
* Verbose output to track down scheduling failure.
* Bump maccore to get improved debug spew.
Diff: f527c9c526..f89d74b165
* [tests][sampletester] Fix build for TodoWCF.
Manuel de la Pena
vs-mobiletools-engineering-service2
Rolf Bjarne Kvinge
Connor Adsit
Alex Soto
monojenkins
Sebastien Pouliot
Whitney Schmidt
Vincent Dondain
Alexander Köplinger