Граф коммитов

37 Коммитов

Автор SHA1 Сообщение Дата
Rolf Bjarne Kvinge 8391a3ca39 [xharness] Major rewrite (add server mode, add device support, add today extension support).
* Add a server mode, which launches a web server (and a web page) that can be
  used to interactively run tests and view their results.

* Add support for running test assemblies in a today extension (generating a
  new set of projects, similar to how we generate tvOS/watchOS projects based
  on the iOS project, we now generate a today extension project in addition to
  the tvOS and watchOS projects).

* Load all the different tests (and show them in the html report, although
  they show up as 'ignored'), even for disabled/ignored tests. This makes
  disabled/ignored tests more visible, and also makes it possible to actually
  run them using the embedded web server.

* Add support for running tests on device. Tests will be executed on multiple
  devices simulatenously (any connected devices will be used).
2017-01-05 13:08:48 +01:00
Rolf Bjarne Kvinge ab21da1b85 [xharness] Add support for uninstalling apps from device. (#1413)
The new error message / number is used in mlaunch.
2017-01-02 08:58:46 +01:00
Rolf Bjarne Kvinge 9ffade2b16 [xharness] When a device from a list of multiple device candidates, order them first to make sure we select consistently. (#1401) 2016-12-22 18:55:25 +01:00
Rolf Bjarne Kvinge 1f81e38a1f [jenkins] Try to show a backtrace for processes that time out. (#1358) 2016-12-16 12:24:08 +01:00
Rolf Bjarne Kvinge 89757e6f60 [xharness] Capture entire log files on wrench. (#1206) 2016-11-17 08:07:09 +01:00
Rolf Bjarne Kvinge c83fa8a9bb [xharness] Fix target selection for simulators on wrench. (#1204) 2016-11-16 19:06:26 +01:00
Rolf Bjarne Kvinge 15b1204874 [xharness] Unify simulator selection code between Jenkins and Wrench and automatically create device pairs if none applicable is found. (#1202)
* [xharness] Don't crash if we can't find a simulator.

* [xharness] Create a device pair if none applicable is found.

* [xharness] Use an enum instead of string values for the target.

* [xharness] Unify the simulator selection code between Jenkins and Wrench.
2016-11-16 15:23:11 +01:00
Rolf Bjarne Kvinge d646d522f3 [xharness] When finding watchos device types, only find the ones that are in available device pairs. (#1198) 2016-11-15 20:20:36 +01:00
Manuel de la Pena aa38ab68ac [Jenkins] Add nuint xml logs (#883)
* [Jenkins] Make test to write output as an xml file so that it can be parsed by the jenkins bot.

* Point to the correct Touvh.Unit repo.

* Use the available property to determine if we are being ran in Jenkins.

* Log where are test results stored.

* Add @MonkeyWrench: prefix.

* Ensure that we do set the build env in jenkins/run-tests.sh

* Do not mix Wrench with Jenkins. The reports in jenkins can be Xml, in Wrench we prefer the old style.

* Ensure that the main node of the unit tests does contain the target, that will improve the tests results reporting.

* Revert "Fix binding project LinkWithAttributes generation to prevent rebuilds" (#1018)

* Added xslt to be used to keep the old Test Reports until we move to only Jenkins reports.

* Add an extra log for the xslt transformation.

* Point to the correcto dir in jenkins.

* Deal with the xslt once we have finished rather than in a batch.

* Remove noise.

* Readd case removed in rebase.

* Fix indentation.

* Skip lock keychain.
2016-11-15 19:04:37 +01:00
Rolf Bjarne Kvinge 816fca0fa1 [xharness] Show better log names. 2016-11-14 14:47:23 +01:00
Rolf Bjarne Kvinge 5ef045d764 [xharness] Capture stdout/stderr from simulator apps. (#1116)
And if we have a tty, just show the output directly there. Otherwise capture
it, and store it in log files.
2016-11-04 12:13:20 +01:00
Rolf Bjarne Kvinge b13bbe0dfb [xharness] Detect and report launch failures better. (#1079) 2016-11-02 15:43:55 +01:00
Rolf Bjarne Kvinge d940b592c8 [xharness] Fetch watch crash reports when running watch apps. (#973) 2016-10-11 19:30:11 +02:00
Rolf Bjarne Kvinge 16e36ce163 [xharness] Add support for watchOS Series 2 simulators. (#812) 2016-09-12 10:14:55 -04:00
Sebastien Pouliot 38148c718a Merge branch 'xcode8' 2016-09-09 15:29:37 -04:00
Rolf Bjarne Kvinge a617bee08f [xharness] Crashing is not OK. (#680) 2016-08-26 09:12:44 -04:00
Rolf Bjarne Kvinge 56fc26b26f [xharness] Split the mscorlib/System test run for watchOS into multiple executions. (#564)
* [tests] Implement filtering based on test name to make it possible to run a subset of tests manually.

* [xharness] Add support for passing environment variables to test apps.

* [xharness] Split the mscorlib/System test run for watchOS into multiple executions.

When running on a watch, the complete set of mscorlib and System tests use too
much memory, so they end up crashing the app.

So instead don't run all the tests in the same test run, but
instead split them up.

This splits the System tests in 5 different sets, based on the first
character of the test class name.

The mscorlib tests are split in 26 different sets, one for each
character (A-Z).
2016-08-05 22:28:13 +02:00
Rolf Bjarne Kvinge 9f3d438169 [xharness] Attach a native debugger when running watchOS tests on device. (#560)
Attaching a native debugger prevents the watch from backgrounding
the app, and makes the test run able to complete without having
to babysit it.
2016-08-05 21:03:56 +02:00
Rolf Bjarne Kvinge 40137dcfb6 [xharness] Don't use variables (isSimulator) before they've been initialized. (#433)
Fixes an issue where we would try to fetch crash logs from device when
running in the simulator.
2016-07-19 14:16:02 -04:00
Rolf Bjarne Kvinge ea9cfe533c [xharness] Don't use variables (isSimulator) before they've been initialized. (#397)
Fixes an issue where we would try to fetch crash logs from device when
running in the simulator.
2016-07-14 15:31:55 -04:00
Rolf Bjarne Kvinge 2be5478669 [xharness] Mark timed out runs as timed out. 2016-07-14 19:23:02 +02:00
Rolf Bjarne Kvinge 74312ec77c [xharness] Collect and report crash reports for Xamarin.Mac tests. (#316) 2016-06-29 09:21:03 -07:00
Rolf Bjarne Kvinge cc8a1bd07b [xharness] Fix boolean logic for skipping simulator setup. (#305) 2016-06-28 13:34:29 -04:00
Rolf Bjarne Kvinge 85d94face2 [xharness] Improve logging.
Make sure as much as possible is written to the logs, and not stdout.

Also copy an extra 1024 bytes when fetching the system log (and continue
capturing a while after stopped, since there may be unflushed system log
data in memory).
2016-06-28 13:04:52 +02:00
Rolf Bjarne Kvinge c355cf91be [xharness] Refactor how we do simulator configuration.
Split out the code to prepare the simulator from the AppRunner class,
which is now just handling the logic required for each test run.

This way it's easier to handle simulator preparation for multiple
test runs with the same simulator.

Also revamp logging to avoid printing directly to the console, but
instead use the logging classes that permit redirecting logging
to a file. This makes the html report show better logging.
2016-06-28 13:04:47 +02:00
Rolf Bjarne Kvinge 1a48c36efd [xharness] When capturing the system log for simulator runs, don't capture contents of the log from before our test run starts. 2016-06-28 13:02:12 +02:00
Rolf Bjarne Kvinge fa33176ba8 [xharness] Cleanup must be done by the last test, not the first test. 2016-06-28 13:02:09 +02:00
Rolf Bjarne Kvinge 93f0b92eac [xharness] iPhone 4s doesn't exist anymore, use iPhone 5 instead. 2016-06-28 13:02:03 +02:00
Sebastien Pouliot ddc3a56e68 [tests][harness] Add kTCCServiceMediaLibrary to avoid privacy popup (#230)
Running some API without permission can crash some tests.
2016-06-28 12:59:25 +02:00
Rolf Bjarne Kvinge 278550795e [xharness] Fix simulator validation check to not fail when it shouldn't. (#265) 2016-06-24 01:03:36 -07:00
Rolf Bjarne Kvinge 64bc391669 [xharness] Fix NRE when running from the command line. (#207)
* [xharness] Log simulator loading and handle any exceptions.

* [xharness] Add missing tests.

* [xharness] Throw a helpful exception instead of NRE when a simulator can't be found.

* [xharness] Fix null reference exception.
2016-06-15 18:57:47 -07:00
Rolf Bjarne Kvinge 9a20b42bd6 [xharness] No need to prepare the simulator more than once. 2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge 0c1f235800 [xharness] Fix simulator configuration to do all configuration before first launch. 2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge 2d3001cd25 [xharness] Don't fail a machine has never seen a crash report. 2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge d135612f59 [tests] Implement support in xharness for running the simulator tests grouped by simulator, and write out an html report. 2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge aaae585d52 [xharness] Xcode 7.3 ships symbolicatecrash in a new place, so look there as well. (#134) 2016-06-06 15:32:09 +02:00
Rolf Bjarne Kvinge ecfdea9508 [tests] Import 2016-05-26 15:06:52 +02:00