* [xharness] Unify the list of test projects for .NET between iOS and macOS.
* [xharness] Don't enable all .NET tests if .NET is enabled.
Some tests (such as device tests) have additional checks that must be honored as well.
* [xharness] If a test is ignored, then it doesn't matter what the corresponding build task is doing.
A build task can be shared between multiple tests, and it can be building for
another (enabled) test. This happens for iOS 32-bits and 64-bits simulator:
the app is built once, but have two different tests running.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Improve Cache.CreateTemporaryDirectory to move the deletion of the previous
root temporary directory to a background thread. In some cases this directory
can have a lot of files and directories, and deleting it synchronously can
cause a significant startup delay when running tests locally.
Fixes this warning when running xharness:
Unknown file: fsharplibrary.fsproj (extension: .fsproj). There might be a default inclusion behavior for this file.
Moved all the code that can be shared with the CLI to the common
library. We neede some small changes, but mainly due to namespaces and a
forgotten interface implemenation (was already implemented, was missing
in the class).
This was fun :)
Refactor TestProject so that we can move all the tasks to the common
assembly. We had to remove all the references from Harness, that
included the MonoNativeInfo.
Some of the Jenkins test tasks are very useful and do A LOT of stuff. So
we try to generalize the base class, to later be able to share the most
usebul ones in the shared lib.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
- Code that will later be moved to the `dotnet/xharness` repo is first moved to an isolated project before it will be extracted to the new repo and NuGetified
- New project for the shared code and new test project for tests accompanying the moved code are created
- Only Factories are left behind that are used only by XHarness
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [Harness] Refactor process management to be testable.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
* Apply suggestions from code review
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
* Move out utils into a separate namespace
* Move Cache to the test project
* Fix namespaces after merge
* Remove unneeded code
* Move Target files
* Sort csproj
* Refactor Targets
* Rename Cache to TempDirectory
* Fix using
* Move ProjectFileExtensions
* Remove dead code
* Move Extensions
* Add empty StringUtilsTests
* Add StringUtils tests
* Revert refactorings
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Add better formatarguments test
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
We do not longer compile the bcl tests, but we consume them as part or
mono. The old BCL targets and classes are not needed. As you can see,
the lists were empty.
It will be nice to be able to have a smaller set of tests to be ran by
xharness. In this commit we add a set of labels to be able to divide the
test in the following groups:
* Xamarin - all tests BUT monotouch.
* Monotouch - just monotouch tests.
* Old BCL - The BCL tests that are based on NUnit
* New BCL - The BCL tests that are based on xUnit.
* mscorlib - Just the test that exercise mscorlib (and the different
groups, mscorlib 1, mscorlib 2 etc..)
This will allow to parallelize the execution of the full test suit in
different agents in VSTS.
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
Harness.AutoConfigureMac now loads all the mac test projects both when
configuring and running tests, the only difference is that the test projects
that must be generated are only generated when configuring. This means that
the Harness.MacTestProject list contains the exact same test projects both
when configuring and when running tests.
This made it possible to remove logic to clone (mac) test projects the Jenkins
class (since Harness.MacTestProjects contains all the test projects already).
Consolidate logic to generate (mac) test projects:
* First we generate BCL and mono-native projects from their templates.
* Then we generate Full/System variations of any project that needs it.
This way we can remove logic to generate Full/System variations from the logic
to generate BCL/mono-native projects, which means less duplicated (and less
confusing) code.
To this purpose, significant changes were required:
* MacTestProject.TargetFrameworkFlavor has been modified to contain a bit mask
of the variations to generate.
* MacMonoNativeInfo has been significantly simplified, and some of the
generated code has been moved to the actual template instead.
* Some project generation (in MacTarget) to make things work as expected.
Fixes https://github.com/xamarin/xamarin-macios/issues/6322.
* [XHarness] Remove the old style mscorlib tests.
Remove the old style test and replace it with the xunit equivalent which
has more tests and is provided by the mono package.
* Only skip the mscorlib tests on watchOS devices with 32b. Run them anywhere else.
Allow to add extra mtouch arguments to the bcl test applications to configure them. This will allow to pass required specific settings that some tests have, for example, for the linker.
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes#5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
Rework BCL test importation to generate projects that won't compile instead of
giving up completely in case of generation failure.
This means that xharness can still be used for non-BCL tests even if the
generation of the BCL tests fail.
- Existing binding projects embed the native libraries within the assembly as managed resource
- This does not scale well and has performance implications
- This PR creates a new property, NoBindingEmbedding which when true processes the building and consumption of binding projects differently.
- Existing binding projects are not affected, they will continue as is
- I've written a full XM test suite and ported a subset to iOS. Since iOS only supports checked in projects, and I didn't want to make the existing situation worse by adding more, I only wrote tests that could use the existing test projects.
-When we complete some form of msbuild testing reform, we'll revisit these tests.
- Remove two files in MyiOSFrameworkBinding that are not used (we use copies elsewhere)
- Remove unnecessary sleep and fix broken touch command
- Output failing test log to console instead of test output
- VSfM does not handle thousands of lines of test failure message well
- Add ability to generate binding projects with LinkWith
* [Xharness] Add a workaround to not build the xunit tests when not needed.
This is a workaround that does not use reflection to build the bcl test
projects. We need this until we have complete support of using mono as
an SDK.
Only build the BCL tests when we need them, because we can't build them when
we're running the device tests (because we don't have a built source tree in
that case).
Since the test assemblies are required when generating the BCL projects, this
means we also delay creating the BCL projects until we need them.
The ProtocolTest test is located in the bindings-test assembly, and it
requires certain conditional compilation defines to be set in order to behave
properly.
With this fix we correctly set these defines when cloning the bindings-test
project (in addition to when we set the defines for the main project, which is
either monotouch-test or linkall for this test).
Fixes https://github.com/xamarin/maccore/issues/655.
This fixes an issue where multiple copies of the GuiUnit project would have
the same output directory, causing random problems when building those
projects in parallel.
* [tests][xammac] Fix a few path calculations to not be hardcoded.
* [xharness] Clone XM projects.
Cloning projects before building them will also clone project references,
which will make it possible to build multiple projects in parallel, when those
projects have common project references.
Fixes https://github.com/xamarin/maccore/issues/624.
* [xharness] Fix XM project cloning
* Don't clone Classic projects (they're solutions, and xharness doesn't understand them).
* Clonee projects properly when cloning execution tasks.
* [xharness] GuiUnit needs specialized cloning.
The GuiUnit project uses relative paths to write to files outside the project
directory, which means that multiple GuiUnit project files may write to the
same location.
So special-case GuiUnit cloning to make those paths subdirectories of the
project's directory instead.
* [xharness] Process imported targets when cloning projects.
Also make the msbuild-mac's custom targets file independent of the location of
the project file by making all paths relative to the custom targets file.
Rolf Bjarne Kvinge
Přemek Vysoký
Manuel de la Pena
Sebastien Pouliot
Alexander Köplinger
Marek Safar
Chris Hamons
Martin Baulig