Stop building the test dependencies on each test run, and instead use the archived test dependencies we have from the main build:
* Stop running 'make all' in tests/ on every separate test run.
* Add a lot more stuff in the package-test-libraries.zip archive.
* Extract all the new stuff on every test run. We add stuff from outside the tests/
directory, so adjust archive creation and extraction to use the root directory
of the repository as the root of the zip archive as well.
* Also add the introspection dependencies to the same archive to simplify the logic.
* Fix xharness to not store absolute paths in generated projects.
* Fix test project to not automatically run make in tests/test-libraries when running
on the bots.
Building the test dependencies takes ~10 minutes for each test run, so this saves
about that time for each test run.
This is because the SetTopLevelPropertyGroupValue method doesn't always
work as expected (it doesn't always set seomthing), while SetProperty does.
Fixing the SetTopLevelPropertyGroup method is somewhat complex, since it
lives in the dotnet/xharness repository, so instead use the SetProperty
method, which is our own (working) version.
Change all null checking expressions to use 'is null' and 'is not null'
instead of '== null' and '!= null'.
This was mostly done with sed, so code can probably be improved in many
other ways with manual inspection, but that will come over time.
Also add code to the autoformat script to automatically fix these issues in the future.
Make the code that creates test variations set a single BundlingArguments
property, and then when we generate the corresponding test project we set both
MtouchExtraArgs and MonoBundlingExtraArgs. The property that doesn't apply to
the current platform will just be ignored.
When generating/cloning test projects and modifying
MtouchExtraArgs/MonoBundlingExtraArgs, we always want to add to any existing
properties instead of overwriting them, so do exactly that.
With this change we now find the latest top-level PropertyGroup in the project
file with no Condition, and add a MtouchExtraArgs/MonoBundlingArgs that adds
to any existing property.
If we're creating a universal app, and here are satellite assemblies that are not
identical across all RuntimeIdentifiers, those assemblies will be stored in a RuntimeIdentifier-specific
subdirectory during the build.
Unfortunately we didn't know how to find those assemblies at runtime, causing localizations
in universal apps to not work.
This change will:
* Add support for looking in the directory where RID-specific satellite assemblies
are stored.
* Add an assembly resolution event handler to our CoreCLR bridge so that we can
execute our custom lookup code.
* Add an assembly resource lookup test to monotouch-test.
* Add a macOS + Mac Catalyst variation of monotouch-test to xharness that triggers
the bug (a universal test app).
Fixes https://github.com/xamarin/xamarin-macios/issues/16847.
* Print the load average every 15 minutes. Also print a full process list if
the load average > 10.
* Print a full process list of the system every hour.
* Print a summary to stdout at the end of the test run.
Co-authored-by: GitHub Actions Autoformatter <github-actions-autoformatter@xamarin.com>
Hopefully this fixes some of the recurrent build problems where we'd fail to
restore nuget packages.
Ref: https://github.com/xamarin/maccore/issues/2620
I wasn't able to execute ilrepack.exe with .NET, so that continues to be
executed with mono.
* Don't load maccore/mk/versions.mk anymore, which means we're not checking
any subsequent dependencies listed in maccore. Note that we still need the
maccore dependency itself for a little while longer.
* Remove some outdated testing code that called into maccore.
* Don't recurse into the maccore directory during make.
* Remove some code checking for ENABLE_XAMARIN that's not used anymore, in
particular in xharness.
This allows the CI to run ALL the tests that the project has in
parallel. This is divided in two main changes:
1. Xharness - We move away from using boolenas to use a flag that states
the tests to run.
2. yaml - We have move the code to use a template per label. This new
jobs all run in parallel and the results are later collected by a
funel job
3. pwsh - Added a new class that understands that we have several mark
downs with the tests results. The classes parses them and them writes
a single comment (and example can be found here: https://github.com/xamarin/xamarin-macios/pull/15201#issuecomment-1162366240
The changes gives the following advantages vs how we used to run tests:
1. The CI run for all tests moves from taking 13 hours to 3/4 hours
(depending on the number of bots in the pool).
2. The download needed to verify the results on a case of failure is
smaller. Rather than downloading several GBs we now just download
that part of the html that we are interested in.
3. Better bot utlization. Bots are just used to a max of 2 hours, this
means that we can use the bots better since they are fragmented.
4. Less VMs. VSDrops has added support for macOS and Linux, we take
advanges of that here.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
We are going to be changing code in xharness. Is a good time to clean
things a little, make code more modern then enable nullability.
Try to get as much help from the new features and try to clean alittle
debt we have from the last time we moved code.
Fixes this NUnit warning:
> labels=All is deprecated and will be removed in a future release. Please use labels=Before instead.
We don't follow the suggestion from the warning, because the advantage of
writing the label after each test is that the test result will also be
printed, which means it's possible to see if any tests failed during the test
run, as opposed to having to wait until the entire test run is completed
(which can take a while) to realize that pretty much every test failed with
some silly mistake which could have been quickly fixed before re-running the
tests.
* [xharness] Bump to the latest version of Microsoft.DotNet.XHarness.iOS.Shared (1.0.0-prerelease.21479.1).
* [xharness] Handle an exceptional condition when we fail to enumerate simulators.
* [xharness] Use new xharness API to select which simulators to use.
This also required bumping xharness.
* Bump again.
* Bump yet again.
* [xharness] Bump and fix according to API break.
* [xharness] Stop the listener before cancelling it.
Cancelling it doesn't do anything if the listener has connected, and that's
usually the case. This prevents subsequent logic from reading incomplete logs.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
* Update to use EnableSGenConc instead of MtouchEnableSGenConc (either works,
the former is the new future).
* Set EnableSGenConc as a top-level property, so we make sure it's set no
matter how the project is executed.
This makes it so that this setting work for .NET projects, where we're not using project configurations.
Skip:
* Any Thumb variation (Thumb code isn't supported anymore in .NET).
* Any AssemblyBuildTarget variation (compiling assemblies to dylibs [fastdev]
or frameworks - neither is supported/implemented for .NET).
This trims down the test list a bit.
* [xharness] Unify the list of test projects for .NET between iOS and macOS.
* [xharness] Don't enable all .NET tests if .NET is enabled.
Some tests (such as device tests) have additional checks that must be honored as well.
* [xharness] If a test is ignored, then it doesn't matter what the corresponding build task is doing.
A build task can be shared between multiple tests, and it can be building for
another (enabled) test. This happens for iOS 32-bits and 64-bits simulator:
the app is built once, but have two different tests running.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Rolf Bjarne Kvinge
Manuel de la Pena
Přemek Vysoký