Граф коммитов

451 Коммитов

Автор SHA1 Сообщение Дата
Rolf Bjarne Kvinge bf4015bfbd
[xharness] Detect another scenario where failing to pair devices, and try again with a new watch device. Fixes maccore#961. (#4702)
Fixes https://github.com/xamarin/maccore/issues/961.
2018-08-28 07:41:22 +02:00
Rolf Bjarne Kvinge ea4909034b
[xharness] Be resilient when trying to copying contents from an inexistent file. (#4676)
Should fix this (or at the very least not prevent xharness from writing out the report):

    21:07:30.3947450 Failed to write log: System.IO.FileNotFoundException: Could not find file '/Users/builder/Library/Logs/CoreSimulator/6DA2ED3C-B1FA-4D0B-9DD6-113E5F9A1381/system.log'.
    File name: '/Users/builder/Library/Logs/CoreSimulator/6DA2ED3C-B1FA-4D0B-9DD6-113E5F9A1381/system.log'
        at System.IO.__Error.WinIOError (System.Int32 errorCode, System.String maybeFullPath) [0x00207] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-02/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/io/__error.cs:188
        at System.IO.FileInfo.get_Length () [0x00038] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-02/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/io/fileinfo.cs:171
        at xharness.CaptureLog.Capture () [0x0004a] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Log.cs:334
        at xharness.CaptureLog.Flush () [0x00008] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Log.cs:373
        at xharness.Jenkins.GenerateReportImpl (System.IO.Stream stream, System.IO.StreamWriter markdown_summary) [0x017db] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Jenkins.cs:2012
        at xharness.Jenkins.GenerateReport (System.Boolean only_if_ci) [0x00075] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Jenkins.cs:1313
2018-08-22 16:54:19 +02:00
Rolf Bjarne Kvinge 1d0e058636
[xharness] Replace existing content in TCC.db. Might fix maccore#951. (#4666)
There seems to be an issue where adding stuff to the TCC.db might fail
partially. In that case we try again, but we try to add every entry once more,
which now might fail due to existing entries.

So always replace when adding new entries in TCC.db. Also dump the database
when done to help debugging if it turns out this doesn't fix maccore#915.

Might fix https://github.com/xamarin/maccore/issues/951.
2018-08-21 19:11:23 +02:00
Rolf Bjarne Kvinge 965e1f7bcc Merge remote-tracking branch 'origin/xcode10' into xcode10-rebase-15.8 2018-08-16 17:09:30 +02:00
Sebastien Pouliot 605b992c62
Bump for beta 6 (#4613) 2018-08-15 13:23:31 -04:00
Vincent Dondain eec95e881c Merge branch 'd15-8' into xcode10-rebase-15.8 2018-08-13 21:04:56 -04:00
Rolf Bjarne Kvinge 9d65e42f58 Fix testing 32-bit XM apps. Fixes maccore issue 884. (#4555)
* [jenkins] Only XM apps with variations are Classic/32-bit apps, so adjust ignore logic accordingly. Fixes maccore issue 884.

Fixes https://github.com/xamarin/maccore/issues/884.

* [mmp] Fix passing -stdlib=libc++ to clang.

* [tests] Fix 32-bit XM issues.

* [xharness] Add support for building 32-bit XM apps by using Xcode 9.4.

* [xharness] Since xharness can now build 32-bit mac apps, enable them by default.

* Remove debug code.

* Remove unused variable.
2018-08-03 17:06:23 -04:00
Alex Soto 313cb7df72
[xharness] Fix simulator hangs in our tests 2018-07-19 09:07:15 -05:00
Rolf Bjarne Kvinge 89df923fac
[xharness] Don't try to add additional information to xml logs. Fixes maccore#827. (#4349)
Don't try to inject additional information into xml logs, it seems we may
corrupt the xml file.

Fixes https://github.com/xamarin/maccore/issues/827.
2018-06-28 14:51:25 +02:00
Rolf Bjarne Kvinge ae6058dd6b
[xharness] Fix publishing xtro's html report. (#4270)
1. Put xtro's results in a subdirectory of the current test's log directory,
   not a sibling directory of the root log directory (which would prevent it
   from being uploaded after a test run, since only the root log directory is
   uploaded).

2. Just add the existing index.html as the html report to the collection of
   logs, no need to create a new file pointing to it. This fixes a problem
   where the generated html file would redirect to a local file, which would
   not work when served from a web server.
2018-06-15 18:01:02 +02:00
Rolf Bjarne Kvinge 6ea6a497b7
[xharness] Add run configuration to project to easily debug internal jenkins execution. (#4266) 2018-06-15 11:11:25 +02:00
Rolf Bjarne Kvinge 61baa008d4 [xharness] Don't include 32-bit XM targets in grouped Makefile targets.
Fixes an issue with the packaged XM tests, where they don't build because we try to build 32-bit tests.
2018-06-13 18:53:57 +02:00
Rolf Bjarne Kvinge 189fffb994
[xharness] Check the right (unpaired) device when determining whether a new watch device must be created or not. (#4222) (#4239)
This fixes a NullReferenceException when there are already paired watch devices:

    [0x700009ad7000:] EXCEPTION handling: System.NullReferenceException: Object reference not set to an instance of an object

    "<unnamed thread>" tid=0x0x700009ad7000 this=0x0x10b903ad8 , thread handle : 0x7f9ad49109a0, state : not waiting
      at xharness.Simulators/<CreateDevicePair>d__18.MoveNext () [0x001ae] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Simulators.cs:169
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<bool>.Start<xharness.Simulators/<CreateDevicePair>d__18> (xharness.Simulators/<CreateDevicePair>d__18&) [0x0002c] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-02/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/runtime/compilerservices/AsyncMethodBuilder.cs:471
      at xharness.Simulators.CreateDevicePair (xharness.Log,xharness.SimDevice,xharness.SimDevice,string,string,bool) [0x00057] in <7c5e77efeb3146c095a26043fb517189>:0
      at xharness.Simulators/<FindOrCreateDevicePairAsync>d__19.MoveNext () [0x000fc] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Simulators.cs:206
      at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<TResult_REF>.Start<TStateMachine_REF> (TStateMachine_REF&) [0x0002c] in <0f9df4881040473f9da7cf6c2e2cb8c3>:0
      at xharness.Simulators.FindOrCreateDevicePairAsync (xharness.Log,System.Collections.Generic.IEnumerable`1<xharness.SimDevice>,System.Collections.Generic.IEnumerable`1<xharness.SimDevice>) [0x0003f] in <7c5e77efeb3146c095a26043fb517189>:0
      at xharness.Simulators/<FindAsync>d__20.MoveNext () [0x00364] in /Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tests/xharness/Simulators.cs:275
2018-06-13 01:04:26 -07:00
Sebastien Pouliot 2d6ce7459d
Bump to use Xcode 10 beta 1 (#4179)
* Bump to use Xcode 10 beta 1

* Update Versions.plist

* Add a dependency on Xcode 9.4.

* [msbuild] Fix build with Xcode 10 beta 1. (#4182)

Many years ago (in Xcode 7 according to code comment)
Developer/Platforms/iPhoneOS.platform/Developer/usr disappeared, and we coped
by looking at Developer/usr instead (and also the subsequent code to locate
the bin directory was based on the location of the usr directory).

Developer/Platforms/iPhoneOS.platform/Developer/usr reappeared in Xcode 10
beta 1, but it seems useless (for one it doesn't contain a bin directory), so
in order to try to keep things sane don't look for this directory in Xcode 10
and instead go directly for Developer/usr (which is what we've been using as
the usr directory for years anyway).

Fixes this problem when building apps with Xcode 10 beta 1:

      /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets(626,3): error : Could not locate SDK bin directory [/Users/rolf/Projects/TestApp/test-app.csproj]

* [runtime] Build 32-bit mac executables using Xcode 9.4.

* [mtouch] Work around broken tvOS headers in Xcode 10 beta 1.

* [mtouch] Work around build problem with Apple's simd headers in Objective-C++ mode.

* Use version-agnostic paths to sdk directories.

* [tests][xtro] Add todo files (from unclassified) and adjust ignore files to avoid errors

* [macos][security] Re-enable SSL[Get|Set]AlpnProtocols. Fixes #4001 (#4022)

* [macos][security] Re-enable SSL[Get}Set]AlpnProtocols. Fixes #4001

This was fixed in macOS 10.13.4

https://github.com/xamarin/xamarin-macios/issues/4001

* [tests][monotouch-tests] Disable a few test cases (one crasher, other failures). Causes to be verified later

* [xharness] Fix permission dialog suppression in Xcode 10.

* [xharness] Ignore 32-bit macOS tests by default.

* [tests] Execute mmp regression tests with Xcode 9.4 since many of them are 32-bit and needs porting to 64-bit.

* [mmptest] Ignore 32-bit XM tests if we don't have a 32-bit-capable Xcode.

* [registrar] Add workaround for broken headers in Xcode 10 beta 1 (radar 40824697).

* [mtouch] Restrict another workaround for an Xcode 10 beta 1 bug to a specific Xcode version to remove it asap.

* [tests] Fix some protocol changes (public or not) find by introspection tests

* [tests][intro] Fix DefaultCtorAllowed failures

* [Intents] Obsolete several Intents classes in watchOS.

Several existing Intents classes have been marked as unavailable in watchOS in
the headers in Xcode 10 beta 1, and corresponding tests are now failing.

So obsolete the managed wrapper types, and fix tests accordingly.

* Fix xtro wrt previous Ietents/intro changes

* [tests] Minor adjustments to mtouch tests to work with Xcode 10.

* [msbuild] Update tests to cope with additional files produced by the Core ML compiler.

* [msbuild] Xcode 10 doesn't support building watchOS 1 apps, so show a clear error message explaining it.

Also update tests accordingly.

* [coreimage] Stub new filters and exclude ?removed? ones from tests

* Update GameplayKit and SpriteKit NSSecureCoding _upgrade_ and fix other non-public cases (in tests)

* [tests] Ignore some GameKit selectors that don't respond anymore (but seems to be available, at least in header files)

* [tests] Fix intro 32bits testing for filters resutls

* [msbuild] Slightly change error message to be better English.
2018-06-08 18:45:24 -07:00
Rolf Bjarne Kvinge d5263edd69
[xharness] Hide summary of test failures in an expandable block in the markdown summary. (#4167)
Hide summary of test failures in an expandable block in the markdown summary,
so that when there are many failures it's still a summary and not pages and
pages of text.

Horror example this will fix: https://github.com/xamarin/xamarin-macios/pull/4163#issuecomment-393566282
2018-06-01 01:17:21 -04:00
Rolf Bjarne Kvinge 11a23af2de [xharness] Handle failure to create a device pair for the simulator better. Fixes xamarin/maccore#830.
Detect if we failed to create a simulator device pair due to trying with a
watch that's already paired. We were already detecting this scenario if the
watch was a member of an available pair, but not if the watch was a member of
an _unavailable_ pair (since mlaunch doesn't list unavailable device pairs).

So detect this scenario from the error message simctl gives us instead, and in
that case create a new watch device and try to create the pair again.

Fixes https://github.com/xamarin/maccore/issues/830.
2018-05-31 18:03:02 +02:00
Rolf Bjarne Kvinge 3df3253807
[xharness] Propagate harness failures from aggregated simulator tasks to each contained task. (#4152)
This makes it so that if anything fails in the aggregated task, it will show
up as a failure in each contained simulator task as well.
2018-05-30 17:08:16 -04:00
Rolf Bjarne Kvinge e784c796dc
[xharness] Create simulators if they don't already exist. (#4085)
* [xharness] Create simulators if they don't already exist.

Create any needed simulators, instead of relying on them already existing (or
someone manually creating them).

A nice side effect is that it will become possible to delete all simulators on
a bot (to reclaim space), and they'll be re-created when needed.

* [xharness] If a watch device is already paired, create a new watch device and pair that instead.
2018-05-30 06:29:51 -04:00
Rolf Bjarne Kvinge 46c30b3d3d
[xharness] Add device test configuration to project file. (#4143)
Add a device test configuration to the project file so that it's easier to
debug xharness when running device tests.
2018-05-29 11:20:30 -04:00
Rolf Bjarne Kvinge 2fe011af3b
[xharness] Parse 'Extension test log' as 'Test log' to improve reporting of extension unit tests. (#4142) 2018-05-29 07:18:17 -04:00
Rolf Bjarne Kvinge a5a6347fe4
[xharness] Override Flush in LogFile to implement Flush as expected. (#4140)
This should make some logs show up faster when running locally in server mode.
2018-05-29 07:17:51 -04:00
Rolf Bjarne Kvinge e35869dfdd
[xharness] Make sure to cancel the test listener both for device and simulator builds. (#4081)
This will make xharness not listen for tests to connect forever, decrease the
number of threads needed, and as well not print loads of aborted messages when
xharness exits:

    SimpleTcpListener: an exception occurred in processing thread: System.Threading.ThreadAbortException: Thread was being aborted.
      at xharness.SimpleTcpListener.Start () [0x0009d] in /Users/XamarinQA/vsts/_work/1/s/tests/xharness/SimpleTcpListener.cs:44
      at xharness.SimpleListener.<StartAsync>b__41_0 () [0x00002] in /Users/XamarinQA/vsts/_work/1/s/tests/xharness/SimpleListener.cs:93
2018-05-15 09:07:45 -04:00
Rolf Bjarne Kvinge 5f70f02fc6
[xharness] Don't load simulators multiple times at the same time. (#4082)
Things become unpredictable (and bad) otherwise.
2018-05-15 09:07:15 -04:00
Rolf Bjarne Kvinge 5a57ff958c
[xharness] Don't make a potentially blocking call to enumerate device candidates in VerifyRun. (#4084)
Enumerating device candidates can block (if the devices haven't been populated
yet), so make the whole thing async-based instead.
2018-05-15 09:01:16 -04:00
Rolf Bjarne Kvinge 3e6fef3c6f
[xharness] Wait for a build to finish even if it has started. (#4083)
Because build tasks might be shared between tests, and if another test already
started a build, this test must still wait until the build finishes.
2018-05-15 08:18:26 -04:00
Rolf Bjarne Kvinge 59c441c1a3
[xharness] Try for 30 seconds to edit TCC.db instead of just 5 seconds. (#4076)
Hopefully fixes an issue with monotouch-test randomly running into permission
dialogs because xharness failed to fix TCC.db because it gave up too early.
2018-05-11 15:21:43 +02:00
Rolf Bjarne Kvinge e3bb98fa7e
[xharness] Create better test clones when creating test variations. (#4062)
When creating test variations, make sure to set the 'IsUnitTest' field
properly, so that xharness produces better test reports.
2018-05-10 10:42:32 +02:00
Rolf Bjarne Kvinge fc9b799b86
[xharness] Improve logging of launched processes to not log the same information twice. (#4061)
Also log any added/changed/removed environment variables for launched
processes (compared to xharness itself).
2018-05-10 07:43:44 +02:00
Rolf Bjarne Kvinge 6bb8d3a69d
[xharness] 1 minute might not be enough to build our native test libraries, so bump to 10 minutes. (#4056) 2018-05-09 15:15:51 +02:00
Rolf Bjarne Kvinge 22db407b79
[tests] Make the msbuild-mac tests a normal NUnit test project. (#4042)
There's no need for this project to be a Xamarin.Mac project, which makes it
faster to build and easier to execute in the IDE.
2018-05-08 07:35:42 +02:00
Bernhard Urban 394117885c
[runtime] add support for mono runtime interpreter (#3058)
* [runtime] build interp-, icalltable- and ilgen libraries so they can be consumed in interpreter configuration

* [mtouch] add --interpreter configuration support

* [xharness] Add support for removing defines from cloned project files.

* [xharness] Run monotouch-test, mscorlib and mini tests with the interpreter too.

They're ignored for now, which means we won't run them on the bots.

* [mtouch] We're using dlsym when using the interpreter.

* [xharness] enable mini regression tests with interpreter on CI
2018-05-04 23:59:31 +02:00
Rolf Bjarne Kvinge 73f6bf0722
[tests] Determine at runtime instead of compile time whether LinkAll is enabled. Fixes #3812. (#4004)
* [tests] Determine at runtime instead of compile time whether LinkAll is enabled. Fixes #3812.

This way we can remove the LINKALL define, which also means nobody can forget
to define it when building using LinkAll.

https://github.com/xamarin/xamarin-macios/issues/3812
2018-04-27 13:11:27 +02:00
Alex Soto 76f0153e72
[build] Remove MT.D source build and replace it with a binary (#3957)
* [build] Remove MT.D source build and replace it with a binary

This commit removes MonoTouch.Dialog from our source build and
replaces it with a binary from c913506df2/MonoTouch.Dialog-Unified

The MT.D hash used in this commit is fixed to migueldeicaza/MonoTouch.Dialog@92c6e14

* Update .gitignore.

* MonoTouch.Dialog-1.dll doesn't need depend on the platform assembly anymore, because we're copying a binary instead of building.

* Add significant debug spew.

* More debugging.

* Remove debug spew.

* [compare-commits] Create directory before trying to create files in it.
2018-04-23 13:08:58 -05:00
Rolf Bjarne Kvinge 3e4b6bb9aa
[xharness] Fix typo in test variation name. (#3946) 2018-04-16 19:36:58 +02:00
Rolf Bjarne Kvinge bb79afdfb7
[xharness] Show html as html by serving the right mime type. (#3917) 2018-04-11 09:08:23 +02:00
Rolf Bjarne Kvinge b0f9d91b9d
[xharness] Fix creation of html reports for NUnit tests. (#3918)
Make sure to override all required TextWriter methods so that the LogFile gets
all the output any writers (such as the XmlWriter that is used to create the
html report for NUnit tests by applying an XSLT transform to the xml results)
writes.
2018-04-11 09:07:58 +02:00
Rolf Bjarne Kvinge 34af7112b5
Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes #3316. (#3877)
* Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes #3316.

If the Xamarin build is enabled, then build mlaunch from source, otherwise get it from macios-binaries.

Fixes #3316.

* Bump macios-binaries.

Commit list for xamarin/macios-binaries:

* xamarin/macios-binaries@69a9088 Bump mlaunch to xamarin/maccore@4505cd6f02 (#8)

Diff: e1e8bdf7a8...69a90882a0

* make msbuild/ after tools/.

Build msbuild/ after tools/, so that mlaunch builds when building from source,
since its build needs the MSBuild logic installed by msbuild/.

* Bump macios-binaries to get Xcode 9.3-capable mlaunch.
2018-04-10 09:53:11 +02:00
Vincent Dondain 2a84c116db Merge branch 'xcode9.3' 2018-04-03 14:14:07 -04:00
Vincent Dondain d34f1fbbd7 Merge branch 'xcode9.3' into d15-7-merge-xcode9.3 2018-03-29 19:31:01 -04:00
Rolf Bjarne Kvinge 4a0fd52fff Revert "Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes #3316. (#3800)" (#3805)
This reverts commit e6c45c2489.

It breaks the mlaunch build.
2018-03-22 21:27:19 -04:00
Rolf Bjarne Kvinge e6c45c2489
Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes #3316. (#3800)
* Put a binary copy of mlaunch in macios-binaries instead of in Azure. Fixes #3316.

If the Xamarin build is enabled, then build mlaunch from source, otherwise get it from macios-binaries.

Fixes #3316.

* Bump macios-binaries.

Commit list for xamarin/macios-binaries:

* xamarin/macios-binaries@69a9088 Bump mlaunch to xamarin/maccore@4505cd6f02 (#8)

Diff: e1e8bdf7a8...69a90882a0
2018-03-22 18:26:07 +01:00
Vincent Dondain 5158a3c001 [xharness] Don't count the skipped tests as failed (#3660)
Currently we consider the failed **and** the skipped tests as failures which is confusing. The tests didn't all "fail" because some were just not ran.

This PR updates the summary message to be a little more clear.
2018-03-09 14:55:42 -05:00
Rolf Bjarne Kvinge bb2041e5ce
[tests] Don't try to compile the Main function in F# extensions. (#3683)
The F# compiler complains that:

    /Users/xamarinqa/vsts/_work/52/s/tests/fsharp/Main.fs(13,9): error FS0433: A function labeled with the 'EntryPointAttribute' attribute must be the last declaration in the last file in the compilation sequence.

Main.fs is the last file in the project file, but the MSBuild tasks adds
another one at the end:

    obj/iPhone/Debug64-today-extension/Xamarin.iOS,Version=v1.0.AssemblyAttribute.fs

because we're building a library (in which case the MSBuild tasks assume that
there won't be any Main functions in the project, and as such it's safe to
append files to compile).

Work around this by excluding the Main function from F# extensions, it's not
needed anyway.
2018-03-07 22:13:13 +01:00
Rolf Bjarne Kvinge 636b2547c5
[xharness] Detect launch failures better and report them as such. (#3682) 2018-03-07 14:48:24 +01:00
Rolf Bjarne Kvinge f38d5bf816 [xharness] All variants of GuiUnit projects must be special-cased. (#3602) (#3662)
This fixes an issue where multiple copies of the GuiUnit project would have
the same output directory, causing random problems when building those
projects in parallel.
2018-03-05 17:18:20 -05:00
Sebastien Pouliot 29676a3662 Merge 15.6 into xcode9.3 2018-03-05 17:16:28 -05:00
Rolf Bjarne Kvinge 5558012e2d
[tests] Don't run mmp regression tests in parallel. Fixes maccore #645. (#3638) (#3641)
* [tests] Don't run mmp regression tests in parallel. Fixes maccore #645.

For some reason vstool becomes confused and starts erroring out if multiple
vstools are running at the same time.

So work around this by executing the MMP regression tests serialized.

Fixes https://github.com/xamarin/maccore/issues/645.

* [xharness] Upload the logs from the mmp regression tests.
2018-03-05 14:53:38 +01:00
Rolf Bjarne Kvinge 08f98e9cf8
[xharness] Add UI to expand/collapse all tests. (#3657) 2018-03-05 14:53:14 +01:00
Rolf Bjarne Kvinge f7369e652c
[xharness] Don't try to reload every second, instead wait a second after each reload to reload again. (#3656)
If the browser is slow to update the page, reloading every second can overload the browser.
2018-03-05 14:52:59 +01:00
Rolf Bjarne Kvinge 9de35b835a [xharness] Only run the install source tests if we're including either Mac or Simulator tests. (#3648)
This means we won't be running these tests when running device tests (which is
not needed).
2018-03-02 15:17:36 -05:00