This allows the CI to run ALL the tests that the project has in
parallel. This is divided in two main changes:
1. Xharness - We move away from using boolenas to use a flag that states
the tests to run.
2. yaml - We have move the code to use a template per label. This new
jobs all run in parallel and the results are later collected by a
funel job
3. pwsh - Added a new class that understands that we have several mark
downs with the tests results. The classes parses them and them writes
a single comment (and example can be found here: https://github.com/xamarin/xamarin-macios/pull/15201#issuecomment-1162366240
The changes gives the following advantages vs how we used to run tests:
1. The CI run for all tests moves from taking 13 hours to 3/4 hours
(depending on the number of bots in the pool).
2. The download needed to verify the results on a case of failure is
smaller. Rather than downloading several GBs we now just download
that part of the html that we are interested in.
3. Better bot utlization. Bots are just used to a max of 2 hours, this
means that we can use the bots better since they are fragmented.
4. Less VMs. VSDrops has added support for macOS and Linux, we take
advanges of that here.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [xharness] Bump to the latest version of Microsoft.DotNet.XHarness.iOS.Shared (1.0.0-prerelease.21479.1).
* [xharness] Handle an exceptional condition when we fail to enumerate simulators.
* [xharness] Use new xharness API to select which simulators to use.
This also required bumping xharness.
* Bump again.
* Bump yet again.
* [xharness] Bump and fix according to API break.
* [xharness] Stop the listener before cancelling it.
Cancelling it doesn't do anything if the listener has connected, and that's
usually the case. This prevents subsequent logic from reading incomplete logs.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
This is mainly to bring in this fix: dotnet/xharness@841114a
(two parameters were swapped and BundleIdentifier was set to AppPath)
Other changes contained:
We don't log the full XML when listing devices anymore, just the file size
SimulatorLoader has a new method that accepts retryCount
Split the BCL project generation in two: one part that figures out which
projects to generate, and which is done synchronously (because we need it to
compute the list of tests), then split out the actual generation and run it on
a background thread (since that doesn't have to happen until we want to
execute those tests).
This speeds up launching xharness in server mode significantly (from ~2s to
~0.2s).
This avoids confusion when tests are run locally using 'make runner' (and
would print the simulator's stdout/stderr to the terminal) versus when run on
bots (and would write the simulator's stdout/stderr to separate files that
would show up in the html report).
This is a step towards trying to make the modification of the html
easier for other team members, the are just to small logical changes:
1. Removed the GenerateReportImpl to a HtmlReportWriter.
2. Do not make the HtmlReportWriter call the Marckdown one, the are
independent.
There is room for improvement, since there are some collections
re-calculated. That change will come once we have this out of the way.
At some point, we ought to be able to make changes in the html just be
as hard as they should be (html + css, we are not experts on that).
Following other PRs, try to reduce the work of the Jenkins class to
orchestrate the different classes and move any other logic to classes
with a single concern. Add tests for the new class.
Move the generation of the markdown out and add tests. Make sure that
the report writes the expected results and specially that it shows the
known issues.
Create some enumerables that will state the tests to run, that way, if
new tests need to be added, there will not be a need to edit the
jenkins class.
Add tests to ensure that all the tests are added. That away we will make
sure that tests are not removed by accident.
Try to make the Jenkins class do a single task. Move the generation of
the variations logic out (which is hard to test atm, but will be doable
in a second round).
Moved some useful methos also out and add tests.
* [Harness] Small refactor to get the knowledge base out of jenkins.
Some small changes to make things a little easier to understand:
* Add a new method in case we find known install issues.
* enable nullable, lets go step by step.
* Move logic outside Jenkins.cs
* Add tests! \o/
* [Harness] Add support to create tunnels.
Add support to create tunnels in case the devices cannot connect to
the host. This option is false by default, which means that unless told
otherwise xharness will try to se a tcp connection over the WiFi.
We are not setting it as default because the devices in DDFun will have
access to an unrestricted network. Nevertheless after this PR we will
create a new one with the following logic:
1. Try to use the tcp connection using the network.
2. If devices cannot connect to the host via the network, fall back to
the tcp tunnel.
This change executes a tunnel process per test application, most of the
cases out of the 150 test application we execute, most of them (maybe 98%
but most % are made up) will pass, and just a few of them will fail. The
reason is that there is a daemon in the OS that gets underwater.
Rather than tryingt o find a hacky way to re-use the tunnel, lets KISS
and if we need to hack that, do it as an enhancement.
- Code that will later be moved to the `dotnet/xharness` repo is first moved to an isolated project before it will be extracted to the new repo and NuGetified
- New project for the shared code and new test project for tests accompanying the moved code are created
- Only Factories are left behind that are used only by XHarness
The Generator and the Factory classes are a xamarin-macios thing.
Initially, they were separated because the code that generated the bcl
tests was not inside xharness. That is not longer the case. We can merge
both classes, generalize the namespace and be more prepared to move out
of the xamarin-macios repo.
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
* [Harness] Remove the BCLTestProjectInfo class.
The class is not really needed. Move to a tuple, make the definition of
the group test project nicer since the dictionary initiallization makes
it cleaner.
Looks like we have cases in which we get empty nodes when listing
devices, which makes the parsing of the nodes fail and crash.
Move the code to a diff method, try to parse the data, if there is an
exception (missing data) return null. If null is returned, do not add
it.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8160
Added a test in the xml parser that verifies that the parsing of the xml
is correct and that the correct number of failures is reported in the
human readable form.
fixes: https://github.com/xamarin/xamarin-macios/issues/8214
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
Refactor result parsing and test it. Make sure that all the possible
corner cases are taken into account. The changes are as follow:
1. Move all the logic to a new class, which will take care of the reustl
parsing.
2. Simplify the parsing method by splitting the noise code to other
functions. There is no need to parse several files in the same method.
3. Add tests that make sure that the correct result is returned when:
a. The app cannot launch.
b. The app timesout.
c. The app crashes.
First step to try and clean up the psring of results and ensure that we
are doing the correct thing. We have several issue there. We want to
move all the complicated logic out of AppRunner and test the different
outputs and the full result matrix.
This commit first allows to use an instance class for the parsing, later
we will move things out. Step by step
XHarness TestAssemblyDefinition is very particular on the way we deal
with the mono test assemblies, in the general case, it only contains the
assembly name, but in the xamarin-macios it calculates the name of the
assembly using the platform.
Create an interface so that the command line was its own implementaiton
and move as much as possible to use the AssemblyLocator class. Later we
can add two simpler implementations for the more general cases.
Move the AssemblyLocator out of the Templates namespace since it makes
more sense.
Rolf Bjarne Kvinge
Manuel de la Pena
Přemek Vysoký