We do not longer compile the bcl tests, but we consume them as part or
mono. The old BCL targets and classes are not needed. As you can see,
the lists were empty.
We initially added a cmd to do the bcl test import generation that has
never been used.
We are moving the bcl test importer code to xharness and creating a
NUnit test project to run the tests that we already added. Unit tests
pass.
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [tests] Remove 32-bit Xamarin.Mac tests (both Classic and Unified).
We're removing support for 32-bit Xamarin.Mac apps (#6300), which means we
don't need to run the tests anymore.
This part of the implementation for #6300, I'm starting with the tests because
some of our XM/Classic tests are failing and making the bots unnecessarily red
(since we'll remove XM/Classic support anyway). Also CI will be faster if we
don't run these tests.
* [mmptest] Fix build.
* [tests] Fix build for mono-native-mac.csproj.
This updates the project generation. We cannot yet fully remove the submodule because:
* We are missing the xunit dlls which should be added in the SDK.
* We have not yet remove all the old style tests. Would make the PR huge, better to deal with it in a diff PR.
* The xunit CoreLib tests have issues loading all the tests, needs some extra work and again, the PR is already large.
Fixes: xamarin/maccore#1199Fixes: xamarin/maccore#1204Fixes: xamarin/maccore#1209Fixes: xamarin/maccore#1510
- Existing binding projects embed the native libraries within the assembly as managed resource
- This does not scale well and has performance implications
- This PR creates a new property, NoBindingEmbedding which when true processes the building and consumption of binding projects differently.
- Existing binding projects are not affected, they will continue as is
- I've written a full XM test suite and ported a subset to iOS. Since iOS only supports checked in projects, and I didn't want to make the existing situation worse by adding more, I only wrote tests that could use the existing test projects.
-When we complete some form of msbuild testing reform, we'll revisit these tests.
- Remove two files in MyiOSFrameworkBinding that are not used (we use copies elsewhere)
- Remove unnecessary sleep and fix broken touch command
- Output failing test log to console instead of test output
- VSfM does not handle thousands of lines of test failure message well
- Add ability to generate binding projects with LinkWith
* [Xharness] Add a workaround to not build the xunit tests when not needed.
This is a workaround that does not use reflection to build the bcl test
projects. We need this until we have complete support of using mono as
an SDK.
Mono know provides a set of testing assemblies that contain the tests to be ran in each of the supported platforms. This commit adds the following:
1. Two different runners that can be used to execute NUnit and xUnit tests.
2. The necessary glue to get xharness to run the new tests and report the results.
3. A new test that will ensure that if mono adds new tests, xharness will report failures if the tests are not run or not ignored.
Improve test reporting using xml by adding custom information printed during
the test run into the xml, and then extract this information when reporting
test results in xharness (which is much easier than re-constructing the
information from the xml, which caused information to get lost since not
everything piece of interesting is in the xml).
https://bugzilla.xamarin.com/show_bug.cgi?id=51661
* Bump to Xcode 9 beta 2
* [CoreImage] Stub out new CIImage filters in Xcode 9 beta 2.
* [monotouch-test] Update tests according to Xcode 9 beta 2.
* [xharness] Ask watch simulator tests to write output directly to a file.
It seems the watch simulator in Xcode 9 beta 2 has completely lost network
access (everything times out), so instead ask the tests to store results
directly in the resulting file.
This works for now since the watch simulator does not enforce the sandbox.
* [xharness] Rewrite the logic to write unit output to a file so that it works with Jenkins.
For Jenkins we ask the unit tests to produce XML, and this needs some special
treatment.
* [xharness] Protect against exceptions in the listener thread, so that it doesn't take xharness down.
* [xharness] Protect against a potential NullReferenceException, and fix one case where it would occur.
* [xharness] A few logging fixes.
* [Intents] Fix API & tests according to beta 2 changes.
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [xharness] Don't crash if we can't find a simulator.
* [xharness] Create a device pair if none applicable is found.
* [xharness] Use an enum instead of string values for the target.
* [xharness] Unify the simulator selection code between Jenkins and Wrench.
* [Jenkins] Make test to write output as an xml file so that it can be parsed by the jenkins bot.
* Point to the correct Touvh.Unit repo.
* Use the available property to determine if we are being ran in Jenkins.
* Log where are test results stored.
* Add @MonkeyWrench: prefix.
* Ensure that we do set the build env in jenkins/run-tests.sh
* Do not mix Wrench with Jenkins. The reports in jenkins can be Xml, in Wrench we prefer the old style.
* Ensure that the main node of the unit tests does contain the target, that will improve the tests results reporting.
* Revert "Fix binding project LinkWithAttributes generation to prevent rebuilds" (#1018)
* Added xslt to be used to keep the old Test Reports until we move to only Jenkins reports.
* Add an extra log for the xslt transformation.
* Point to the correcto dir in jenkins.
* Deal with the xslt once we have finished rather than in a batch.
* Remove noise.
* Readd case removed in rebase.
* Fix indentation.
* Skip lock keychain.
Split out the code to prepare the simulator from the AppRunner class,
which is now just handling the logic required for each test run.
This way it's easier to handle simulator preparation for multiple
test runs with the same simulator.
Also revamp logging to avoid printing directly to the console, but
instead use the logging classes that permit redirecting logging
to a file. This makes the html report show better logging.
Manuel de la Pena
Rolf Bjarne Kvinge
Alexander Köplinger
Marek Safar
Chris Hamons
Martin Baulig
Vincent Dondain
Sebastien Pouliot
Vincent Dondain
Timothy Risi
Manuel de la Pena