Mono know provides a set of testing assemblies that contain the tests to be ran in each of the supported platforms. This commit adds the following:
1. Two different runners that can be used to execute NUnit and xUnit tests.
2. The necessary glue to get xharness to run the new tests and report the results.
3. A new test that will ensure that if mono adds new tests, xharness will report failures if the tests are not run or not ignored.
* [xharness] Improve logging a bit.
* Use timestamp in more log paths.
* Create numbered log subdirectories to make things nicer to look at in a
terminal (thousands of subdirectories can be annoying to shift through; this
way there's an additional subdirectory level).
* Avoid string interpolation when not needed.
This will make xharness not listen for tests to connect forever, decrease the
number of threads needed, and as well not print loads of aborted messages when
xharness exits:
SimpleTcpListener: an exception occurred in processing thread: System.Threading.ThreadAbortException: Thread was being aborted.
at xharness.SimpleTcpListener.Start () [0x0009d] in /Users/XamarinQA/vsts/_work/1/s/tests/xharness/SimpleTcpListener.cs:44
at xharness.SimpleListener.<StartAsync>b__41_0 () [0x00002] in /Users/XamarinQA/vsts/_work/1/s/tests/xharness/SimpleListener.cs:93
Make the collection of logs (the `Logs` class) more capable by making it
disposable (which will dispose all contained logs) and give it a Directory
property to state where the logs should be stored. This makes it possible to
simplify a few repeated path calculations. It also allows us to easily dispose
the entire collection of logs when done with a test, as opposed to dispoing
the logs one by one.
Make LogFile more capable:
* Add support for writing byte arrays.
* Add support for logging after disposal: this will still write to the file,
and not keep any files open after finished writing. This fixes a problem
where writing to a log after it was disposed would crash xharness (which is
not all that uncommon, given the async nature of how xharness runs tests).
This makes it possible to get rid of LogStream, and use LogFile instead.
Dispose logs (so that the corresponding files are closed) when we're done with
them, and also don't open a file log by default (usually we just want a
filename to pass to somebody else), but instead open the file if needed.
This should decrease the number of open file descriptors in xharness, which
sometimes become a problem when running many tests.
Detect MT1111 from mlaunch (which means mlaunch won't wait for the app to
exit), and instead use test run completion to determine test completion.
The main drawback is that if the test app crashes, it won't be detected (the
test run will time out, and reported as such), but it's still an improvement
over the current behavior (the tests may complete successfully, and still be
reported as timed out).
This also requires bumping maccore to get an updated mlaunch (one that reports
MT1111).
* [monotouch-test] Optionally ignore tests that show permission dialogs.
* [xharness] Add support for excluding tests that require permission dialogs.
* [xharness] Ignore introspection tests if we don't want permission dialogs.
Ignore introspection tests for now if we don't want permission dialogs, since
figuring out which API requires permissions is a tedious process (which we'll
eventually have to do though).
Improve test reporting using xml by adding custom information printed during
the test run into the xml, and then extract this information when reporting
test results in xharness (which is much easier than re-constructing the
information from the xml, which caused information to get lost since not
everything piece of interesting is in the xml).
https://bugzilla.xamarin.com/show_bug.cgi?id=51661
* Bump to Xcode 9 beta 2
* [CoreImage] Stub out new CIImage filters in Xcode 9 beta 2.
* [monotouch-test] Update tests according to Xcode 9 beta 2.
* [xharness] Ask watch simulator tests to write output directly to a file.
It seems the watch simulator in Xcode 9 beta 2 has completely lost network
access (everything times out), so instead ask the tests to store results
directly in the resulting file.
This works for now since the watch simulator does not enforce the sandbox.
* [xharness] Rewrite the logic to write unit output to a file so that it works with Jenkins.
For Jenkins we ask the unit tests to produce XML, and this needs some special
treatment.
* [xharness] Protect against exceptions in the listener thread, so that it doesn't take xharness down.
* [xharness] Protect against a potential NullReferenceException, and fix one case where it would occur.
* [xharness] A few logging fixes.
* [Intents] Fix API & tests according to beta 2 changes.
* [xharness] Show test xml files in the html report.
* [xharness] Force links to xml files to show up in a new top-level window.
Jenkins shows the html report in an iframe, and links will load in that
iframe. This means that the browser renders the contents of the link as if it
was html, and rendering xml as html results in a blank page.
So set the target attribute on links to xml files, so that those xml files
open as top-level pages, and then the browser will rendering the xml like xml.
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [xharness] Don't crash if we can't find a simulator.
* [xharness] Create a device pair if none applicable is found.
* [xharness] Use an enum instead of string values for the target.
* [xharness] Unify the simulator selection code between Jenkins and Wrench.
* [Jenkins] Make test to write output as an xml file so that it can be parsed by the jenkins bot.
* Point to the correct Touvh.Unit repo.
* Use the available property to determine if we are being ran in Jenkins.
* Log where are test results stored.
* Add @MonkeyWrench: prefix.
* Ensure that we do set the build env in jenkins/run-tests.sh
* Do not mix Wrench with Jenkins. The reports in jenkins can be Xml, in Wrench we prefer the old style.
* Ensure that the main node of the unit tests does contain the target, that will improve the tests results reporting.
* Revert "Fix binding project LinkWithAttributes generation to prevent rebuilds" (#1018)
* Added xslt to be used to keep the old Test Reports until we move to only Jenkins reports.
* Add an extra log for the xslt transformation.
* Point to the correcto dir in jenkins.
* Deal with the xslt once we have finished rather than in a batch.
* Remove noise.
* Readd case removed in rebase.
* Fix indentation.
* Skip lock keychain.
* [tests] Implement filtering based on test name to make it possible to run a subset of tests manually.
* [xharness] Add support for passing environment variables to test apps.
* [xharness] Split the mscorlib/System test run for watchOS into multiple executions.
When running on a watch, the complete set of mscorlib and System tests use too
much memory, so they end up crashing the app.
So instead don't run all the tests in the same test run, but
instead split them up.
This splits the System tests in 5 different sets, based on the first
character of the test class name.
The mscorlib tests are split in 26 different sets, one for each
character (A-Z).
Make sure as much as possible is written to the logs, and not stdout.
Also copy an extra 1024 bytes when fetching the system log (and continue
capturing a while after stopped, since there may be unflushed system log
data in memory).
Split out the code to prepare the simulator from the AppRunner class,
which is now just handling the logic required for each test run.
This way it's easier to handle simulator preparation for multiple
test runs with the same simulator.
Also revamp logging to avoid printing directly to the console, but
instead use the logging classes that permit redirecting logging
to a file. This makes the html report show better logging.
Manuel de la Pena
Rolf Bjarne Kvinge
Sebastien Pouliot
Vincent Dondain
Timothy Risi
Manuel de la Pena