* [xharness] Show test xml files in the html report.
* [xharness] Force links to xml files to show up in a new top-level window.
Jenkins shows the html report in an iframe, and links will load in that
iframe. This means that the browser renders the contents of the link as if it
was html, and rendering xml as html results in a blank page.
So set the target attribute on links to xml files, so that those xml files
open as top-level pages, and then the browser will rendering the xml like xml.
Don't create multiple writer streams for the same underlying file, instead
store the writer stream on the log instance, and return it whenever needed.
This fixes an issue where there would be multiple writer streams, which could
cause exceptions when creating more writer streams (access defined errors
because the file is already open).
* [tests] Use the target directory from the loaded configuration.
* [xharness] Find the root directory based on xharness.exe's location (unless specified).
* [tests] Add makefile target to generate test config using the system XI.
* [xharness] Don't copy a ExecutionResult before executing.
The source might be an ignored test, even though the cloned task isn't, so if
we copy the ExecutionResult we might unexpectedly ignore the cloned task.
* [xharness] Provide better failure message for failing mac tests.
Unexpected exceptions are now caught in a more central location
(TestTask.RunInternalAsync), which means it's not necessary to catch
unexpected exceptions in other places (so those catch handlers have been
removed).
Additionally we now set the failure message when such an exception occurs, in
addition to writing the full exception details to a custom log file.
We're shipping mlaunch now, which means we have other file system locations
where it might be.
Also don't look in Xamarin Studio anymore, it will always be an old version,
instead check the system's Xamarin.iOS version.
* Add a server mode, which launches a web server (and a web page) that can be
used to interactively run tests and view their results.
* Add support for running test assemblies in a today extension (generating a
new set of projects, similar to how we generate tvOS/watchOS projects based
on the iOS project, we now generate a today extension project in addition to
the tvOS and watchOS projects).
* Load all the different tests (and show them in the html report, although
they show up as 'ignored'), even for disabled/ignored tests. This makes
disabled/ignored tests more visible, and also makes it possible to actually
run them using the embedded web server.
* Add support for running tests on device. Tests will be executed on multiple
devices simulatenously (any connected devices will be used).
* [jenkins] Make jenkins non-parellel to see if it fixes#46097.
* [jenkins] Limit serialization to watchos builds.
* [jenkins] Only allow Mac builds to be paralellized.
* [xharness] Add Jenkins support for mac binding project tests.
* [jenkins] Rework how mac binding projects are tested.
Facts:
* The wrench-mac-binding-project target will clean itself when done: [1]
* The mac-msbuild tests depend on mac-binding-project build artifacts (but
will build those artifacts automatically if they don't exist when the mac-
msbuild tests are built).
* Jenkins runs tests in parallel.
Result:
* The mac-msbuild tests may fail if the mac-binding-project tests are run
concurrently and delete its build artifacts.
Fix:
* Don't call the wrench-mac-binding-project target (which will clean itself),
instead just run the tests (without cleaning).
[1]: d99212a712/tests/Makefile (L280-L282)
* [tests] Fix framework-test to actually work.
* [xharness] Properly replace 'ios' with corresponding platform for paths to our test frameworks as well.
* [framework-test] Fix watchOS build.
* Bump maccore to get new mlaunch.
A new mlaunch that:
* Should have fewer random failures when launching watchOS apps.
* Supports launching extensions on device.
* Supports uninstalling apps from devices.
* [jenkins] Automatically detect mono bumps and enable device build.
And do this before fetching labels, so that we can skip fetching labels if we
know we're already enabling the device build.
* [tests] Bumping LLVM merits enabling device build and running mtouch + BCL tests.
e.g. running twice
> make run-ios-sim32-introspection
results in
Unhandled Exception:
System.AggregateException: One or more errors occurred. ---> System.IO.IOException: /Users/poupou/git/master/xamarin-macios/tests/logs/exec-ios-sim32-introspection/iPhone 5.log already exists
at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x001bd] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/corlib/System.IO/File.cs:109
at System.IO.File.Copy (System.String sourceFileName, System.String destFileName) [0x00000] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/corlib/System.IO/File.cs:69
at xharness.CaptureLog.StopCapture () [0x00019] in /Users/poupou/git/master/xamarin-macios/tests/xharness/Log.cs:252
* [XM] Add 32-bit variants of apitest and introspection and fix bindings
- Previous to this we were only running 64-bit tests, which caused us
to make mistakes
* Fix cherry-pick to build
* Correctly skip jenkins XM45 when asked
* Fix logic
* Run 32-bit tests on Jenkins
* Fix Jenkins grouping by respecting -32 suffix
* More fixing of jenkins. This time make enum for 32-bit types
The previous method of getting the base and head sha from the pull request
doesn't work if the pull request has been rebased.
Example: https://github.com/xamarin/xamarin-macios/pull/1238
Resulted in this [1]:
Fetching modified files for commit range cf07825667aa444c988c82b7e29cefc5f8ba7bcd..6aa2b9517ac35374dfa4ded41d1e2ff52778da07
git diff-tree --no-commit-id --name-only -r cf07825667aa444c988c82b7e29cefc5f8ba7bcd..6aa2b9517ac35374dfa4ded41d1e2ff52778da07
Found 20 modified file(s) in the pull request #1238.
Makefile
Versions-ios.plist.in
Versions-mac.plist.in
external/llvm
external/mono
msbuild/Xamarin.Mac.Tasks.Core/Tasks/MmpTaskBase.cs
msbuild/Xamarin.Mac.Tasks/Xamarin.Mac.Common.props
msbuild/Xamarin.Mac.Tasks/Xamarin.Mac.Common.targets
msbuild/Xamarin.iOS.Tasks.Core/Tasks/MTouchTaskBase.cs
msbuild/Xamarin.iOS.Tasks.Core/Xamarin.iOS.Common.props
msbuild/Xamarin.iOS.Tasks.Core/Xamarin.iOS.Common.targets
src/Makefile
src/UIKit/UIScreen.cs
src/appkit.cs
src/generator-diff.mk
tests/introspection/ApiTypoTest.cs
tests/introspection/Mac/MacApiTypoTest.cs
tests/introspection/Mac/introspection-mac.csproj
tests/xharness/Harness.cs
versions-check.csharp
which is listing way too many files.
[1]: https://jenkins.mono-project.com/job/xamarin-macios-pr-builder/2118/Test_Report/Harness.log
* [jenkins] Add support for enabling device builds using labels.
* [xharness] Give the iOS MSBuild tests 30 minutes to finish.
* [mtouch tests] Give the BuildTestProject 10 minutes to compile each test case.
Wrench bots build the dontlink test in ~3m40, but that's apparently not enough
for the Jenkins bots (slower bots?), which time out the test after 5 minutes.
So double the timeout to 10 minutes, which will hopefully give the Jenkins
bots enough time to run the test to completion.