Rolf Bjarne Kvinge
2c892e2b02
[xharness] Get new mlaunch for beta 4.
2016-08-02 18:15:54 +02:00
Rolf Bjarne Kvinge
c30a2f52ce
[xharness] Don't run into an infinite loop if failing to run sqlite3. ( #515 )
2016-08-01 16:36:14 +02:00
Rolf Bjarne Kvinge
dec9ff608f
[xharness] Don't try killing a process that has already exited. ( #483 )
2016-07-25 15:02:35 -04:00
Rolf Bjarne Kvinge
51937dc7e3
[xharness] Don't generate makefile targets for non-executable projects. ( #482 )
2016-07-25 15:01:34 -04:00
Rolf Bjarne Kvinge
d1d0e9a501
[xharness] Simplify BCL project generation by merging iOS and tvOS logic. ( #480 )
2016-07-25 15:01:14 -04:00
Sebastien Pouliot
308db4d1ea
Merge branch 'cycle8' into xcode8
2016-07-19 14:27:59 -04:00
Rolf Bjarne Kvinge
40137dcfb6
[xharness] Don't use variables (isSimulator) before they've been initialized. ( #433 )
...
Fixes an issue where we would try to fetch crash logs from device when
running in the simulator.
2016-07-19 14:16:02 -04:00
Rolf Bjarne Kvinge
ea9cfe533c
[xharness] Don't use variables (isSimulator) before they've been initialized. ( #397 )
...
Fixes an issue where we would try to fetch crash logs from device when
running in the simulator.
2016-07-14 15:31:55 -04:00
Rolf Bjarne Kvinge
a429d2981c
[xharness] Add support for re-reading console output.
...
Fixes a confusing exception/error message when a test run times out.
2016-07-14 19:23:02 +02:00
Rolf Bjarne Kvinge
2be5478669
[xharness] Mark timed out runs as timed out.
2016-07-14 19:23:02 +02:00
Sebastien Pouliot
cf27c2e7a8
Merge branch 'cycle8' into xcode8
2016-07-13 16:43:00 -04:00
Sebastien Pouliot
8af1e47588
[tests] Change location to download mlaunch, old one is not available anymore ( #352 )
2016-07-06 19:40:27 -04:00
Sebastien Pouliot
88bccb268d
[tests] Change location to download mlaunch, old one is not available anymore ( #352 )
2016-07-06 17:04:08 -04:00
Rolf Bjarne Kvinge
cf2d43e85c
[xharness] Create a file if it doesn't exist when creating a writer for a log file. ( #332 )
2016-06-30 02:39:33 -07:00
Rolf Bjarne Kvinge
74312ec77c
[xharness] Collect and report crash reports for Xamarin.Mac tests. ( #316 )
2016-06-29 09:21:03 -07:00
Rolf Bjarne Kvinge
610ef84d37
[xharness] Fix result propagation. ( #321 )
2016-06-29 07:50:02 -07:00
Rolf Bjarne Kvinge
1080164506
[jenkins] Fail if test run fails. ( #245 )
...
* [jenkins] Fail if test run fails.
* [xharness] Fix success check.
2016-06-29 03:23:26 -07:00
Rolf Bjarne Kvinge
0636832673
[xharness] Run watchOS tests on jenkins. ( #313 )
2016-06-29 01:48:53 -07:00
Rolf Bjarne Kvinge
7e773080a9
Merge remote-tracking branch 'origin/master' into xcode8
2016-06-29 10:23:03 +02:00
Rolf Bjarne Kvinge
dff594bb5a
[xharness] Add the XI MSBuild tests to the jenkins test run.
2016-06-29 00:06:35 +02:00
Rolf Bjarne Kvinge
d91258fb32
[xharness] Simplify non-grouped tasks in the html report a little bit.
2016-06-29 00:06:31 +02:00
Rolf Bjarne Kvinge
38af4434e0
Merge pull request #256 from rolfbjarne/jenkins-ios-msbuild
...
[xharness] Add the XI MSBuild tests to the jenkins test run.
2016-06-28 15:05:25 -07:00
Rolf Bjarne Kvinge
fb607bef03
[xharness] Rename variable to fix compiler warning. ( #308 )
...
Fixes this warning:
SimpleHttpListener.cs(12,8): warning CS0108: `xharness.SimpleHttpListener.connected' hides inherited member `xharness.SimpleListener.connected'. Use the new keyword if hiding was intended
2016-06-28 17:13:36 -04:00
Rolf Bjarne Kvinge
cc8a1bd07b
[xharness] Fix boolean logic for skipping simulator setup. ( #305 )
2016-06-28 13:34:29 -04:00
Rolf Bjarne Kvinge
d32b7746c5
[xharness] Protect against duplicated connected/finished notifications. ( #301 )
2016-06-28 08:20:07 -07:00
Rolf Bjarne Kvinge
63c6ae06dd
[xharness] Add the XI MSBuild tests to the jenkins test run.
2016-06-28 16:05:48 +02:00
Rolf Bjarne Kvinge
c4591f44d2
[xharness] Simplify non-grouped tasks in the html report a little bit.
2016-06-28 16:05:27 +02:00
Rolf Bjarne Kvinge
3bf7b92d88
[xharness] Give permissions to watchkit apps as well. ( #296 )
...
* [xharness] Only execute sqlite3 once.
* [xharness] Give permissions to watchkit apps as well.
2016-06-28 07:04:10 -07:00
Rolf Bjarne Kvinge
0d2d6e36b1
[xharness] Don't prepare simulator for tests that didn't build. ( #297 )
2016-06-28 07:03:40 -07:00
Rolf Bjarne Kvinge
613fb2118c
[jenkins] Skip mmptest to see if it'll speed up the test run on jenkins. ( #248 )
2016-06-28 13:05:00 +02:00
Rolf Bjarne Kvinge
455c741820
[xharness] Try to bump the launch timeout to 3 minutes.
2016-06-28 13:04:56 +02:00
Rolf Bjarne Kvinge
85d94face2
[xharness] Improve logging.
...
Make sure as much as possible is written to the logs, and not stdout.
Also copy an extra 1024 bytes when fetching the system log (and continue
capturing a while after stopped, since there may be unflushed system log
data in memory).
2016-06-28 13:04:52 +02:00
Rolf Bjarne Kvinge
c355cf91be
[xharness] Refactor how we do simulator configuration.
...
Split out the code to prepare the simulator from the AppRunner class,
which is now just handling the logic required for each test run.
This way it's easier to handle simulator preparation for multiple
test runs with the same simulator.
Also revamp logging to avoid printing directly to the console, but
instead use the logging classes that permit redirecting logging
to a file. This makes the html report show better logging.
2016-06-28 13:04:47 +02:00
Rolf Bjarne Kvinge
61485522d4
[xharness] Always setup/cleanup tvOS test runs.
...
Something strange is going on on the bots where only the first tvOS
test run actually launches the test app successfully.
2016-06-28 13:04:41 +02:00
Rolf Bjarne Kvinge
a59a762e84
[xharness] Download mlaunch if we don't have a local copy.
2016-06-28 13:04:10 +02:00
Rolf Bjarne Kvinge
1e6be7f547
[xharness] Add a few more variables to control what to include in a test run.
2016-06-28 13:02:21 +02:00
Rolf Bjarne Kvinge
c98cc1dc19
[xharness] Ignore ignored tests when deciding which color to use for a group of tests.
2016-06-28 13:02:16 +02:00
Rolf Bjarne Kvinge
1a48c36efd
[xharness] When capturing the system log for simulator runs, don't capture contents of the log from before our test run starts.
2016-06-28 13:02:12 +02:00
Rolf Bjarne Kvinge
fa33176ba8
[xharness] Cleanup must be done by the last test, not the first test.
2016-06-28 13:02:09 +02:00
Rolf Bjarne Kvinge
93f0b92eac
[xharness] iPhone 4s doesn't exist anymore, use iPhone 5 instead.
2016-06-28 13:02:03 +02:00
Rolf Bjarne Kvinge
c09c74dba5
[xharness] Improve html report a bit when some tests are ignored.
2016-06-28 12:59:54 +02:00
Rolf Bjarne Kvinge
ab7bd56e63
[xharness] Add a few variables to control if to include iOS/Mac tests in jenkins test mode.
2016-06-28 12:59:50 +02:00
Rolf Bjarne Kvinge
a7b20c1a43
[xharness] Fix emptyness check.
2016-06-28 12:59:42 +02:00
Rolf Bjarne Kvinge
3a825fc5e0
[xharness] Improve watching objects in the debugger.
2016-06-28 12:59:36 +02:00
Sebastien Pouliot
ddc3a56e68
[tests][harness] Add kTCCServiceMediaLibrary to avoid privacy popup ( #230 )
...
Running some API without permission can crash some tests.
2016-06-28 12:59:25 +02:00
Rolf Bjarne Kvinge
1a1958f52d
Merge remote-tracking branch 'origin/master' into xcode8
2016-06-28 12:44:08 +02:00
Sebastien Pouliot
4a1f7fa778
Merge branch 'master' into xcode8
2016-06-27 16:30:29 -04:00
Rolf Bjarne Kvinge
833ea3fc8b
[xharness] Fix name of referenced projects.
2016-06-27 13:07:37 +02:00
Rolf Bjarne Kvinge
278550795e
[xharness] Fix simulator validation check to not fail when it shouldn't. ( #265 )
2016-06-24 01:03:36 -07:00
Rolf Bjarne Kvinge
d62295f0bd
[jenkins] Skip mmptest to see if it'll speed up the test run on jenkins. ( #248 )
2016-06-22 07:12:51 -07:00
Rolf Bjarne Kvinge
fdccb019a5
[xharness] Try to bump the launch timeout to 3 minutes.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
fb37a21ef9
[xharness] Improve logging.
...
Make sure as much as possible is written to the logs, and not stdout.
Also copy an extra 1024 bytes when fetching the system log (and continue
capturing a while after stopped, since there may be unflushed system log
data in memory).
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
dfd0fbe81a
[xharness] Refactor how we do simulator configuration.
...
Split out the code to prepare the simulator from the AppRunner class,
which is now just handling the logic required for each test run.
This way it's easier to handle simulator preparation for multiple
test runs with the same simulator.
Also revamp logging to avoid printing directly to the console, but
instead use the logging classes that permit redirecting logging
to a file. This makes the html report show better logging.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
755a0622bd
[xharness] Always setup/cleanup tvOS test runs.
...
Something strange is going on on the bots where only the first tvOS
test run actually launches the test app successfully.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
6b5a394244
[xharness] Add a few more variables to control what to include in a test run.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
0a39ec940f
[xharness] Ignore ignored tests when deciding which color to use for a group of tests.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
1e1bbec8a3
[xharness] When capturing the system log for simulator runs, don't capture contents of the log from before our test run starts.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
bd993fbe0d
[xharness] Cleanup must be done by the last test, not the first test.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
cf060f4611
[xharness] iPhone 4s doesn't exist anymore, use iPhone 5 instead.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
d6f27c2e65
[xharness] Improve html report a bit when some tests are ignored.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
ade84d6397
[xharness] Add a few variables to control if to include iOS/Mac tests in jenkins test mode.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
64bf75c533
[xharness] Fix emptyness check.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
b2431898b0
[xharness] Improve watching objects in the debugger.
2016-06-22 10:42:22 +02:00
Rolf Bjarne Kvinge
5a9c9003fa
[xharness] Use Xcode 8 when debugging.
2016-06-22 10:42:22 +02:00
Sebastien Pouliot
d9fb8dc937
[tests][harness] Add kTCCServiceMediaLibrary to avoid privacy popup ( #230 )
...
Running some API without permission can crash some tests.
2016-06-18 06:36:59 -07:00
Rolf Bjarne Kvinge
b8ceeeb044
[xharness] Fix logdirectory path to use correct directory separator. ( #212 )
2016-06-16 00:03:04 -07:00
Rolf Bjarne Kvinge
70747b3317
[xharness] Fix logdirectory path to use correct directory separator. ( #210 )
2016-06-15 22:05:12 -07:00
Rolf Bjarne Kvinge
64bc391669
[xharness] Fix NRE when running from the command line. ( #207 )
...
* [xharness] Log simulator loading and handle any exceptions.
* [xharness] Add missing tests.
* [xharness] Throw a helpful exception instead of NRE when a simulator can't be found.
* [xharness] Fix null reference exception.
2016-06-15 18:57:47 -07:00
Rolf Bjarne Kvinge
8a164bc8d0
[xharness] Download mlaunch if we don't have a local copy.
2016-06-15 17:00:09 -07:00
Rolf Bjarne Kvinge
f17241e6e3
[xharness] Fix null reference exception.
2016-06-15 16:52:45 -07:00
Rolf Bjarne Kvinge
df326a1d59
[xharness] Throw a helpful exception instead of NRE when a simulator can't be found.
2016-06-15 16:52:24 -07:00
Rolf Bjarne Kvinge
ed1cbe003e
[xharness] Add missing tests.
2016-06-15 16:51:42 -07:00
Rolf Bjarne Kvinge
73b3a9b53e
[xharness] Log simulator loading and handle any exceptions.
2016-06-15 13:32:09 -07:00
Rolf Bjarne Kvinge
15a5af3342
[xharness] Exclude bcl tests by default.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
a5f5a76cc0
[xharness] Try and fix some strange file sharing exceptions.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
126de6836a
[xharness] Use full path to mdtool.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
0910cacf51
[xharness] Fix processing of ignored tasks.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
dcc605ed22
[xharness] Build XM/Classic projects using mdtool.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
af4a93a181
[xharness] Pass the right environment variables when invoking xbuild.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
ae1bf7fb5f
[xharness] Add support for ignoring tests.
...
And use it to ignore framework-test (never run on wrench) and
watchOS tests (currently ignored on wrench).
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
d9bf0e1728
[xharness] Add support for building and running mac tests on jenkins.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
9a20b42bd6
[xharness] No need to prepare the simulator more than once.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
0c1f235800
[xharness] Fix simulator configuration to do all configuration before first launch.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
2d3001cd25
[xharness] Don't fail a machine has never seen a crash report.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
d135612f59
[tests] Implement support in xharness for running the simulator tests grouped by simulator, and write out an html report.
2016-06-15 10:51:14 -07:00
Rolf Bjarne Kvinge
adde200556
[xharness] Exclude bcl tests by default.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
fdf2bd5d7b
[xharness] Try and fix some strange file sharing exceptions.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
4bad2c1485
[xharness] Use full path to mdtool.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
266606528b
[xharness] Fix processing of ignored tasks.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
eec56fefc6
[xharness] Build XM/Classic projects using mdtool.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
f6948f8183
[xharness] Pass the right environment variables when invoking xbuild.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
39a0873403
[xharness] Add support for ignoring tests.
...
And use it to ignore framework-test (never run on wrench) and
watchOS tests (currently ignored on wrench).
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
ad229cef71
[xharness] Add support for building and running mac tests on jenkins.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
969f2d7ca1
[xharness] No need to prepare the simulator more than once.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
3d5455b54c
[xharness] Fix simulator configuration to do all configuration before first launch.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
646bd59c85
[xharness] Don't fail a machine has never seen a crash report.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
d087ba2d5a
[tests] Implement support in xharness for running the simulator tests grouped by simulator, and write out an html report.
2016-06-15 09:57:24 -07:00
Rolf Bjarne Kvinge
8af0326105
[xharness] Clean targets are named 'Clean', not 'clean'. ( #162 )
...
At least according to mdtool.
Fixes this:
[...]
Target 'clean not supported
make[4]: *** [clean-mac-classic-introspection] Error 1
2016-06-09 14:23:58 -04:00
Rolf Bjarne Kvinge
e3cc7c31bc
[xharness] Write relative paths in the makefiles. ( #160 )
...
Fixes the packaged mac tests when they're executed on a different
machine with a different path.
2016-06-09 17:36:32 +02:00
Rolf Bjarne Kvinge
9c01fe6c9d
[tests] Use the right sources for dontlink, linksdk and linkall tests. ( #152 )
2016-06-08 23:39:47 +02:00