The test run can fail even if all the tests pass: in case the process crashes
or deadlocks at exit for instance. Don't overwrite such failures with the unit
test results.
Use a private property (prefixed with underscore) for now, until we can decide
on a better/general name.
Also add a variation to xharness to build monotouch-test with CoreCLR
(building works fine, but it crashes at startup, which is expected at this
point).
Sometimes msbuild wants to restore packages during this process, which may
take a while.
Hopefully fixes errors like this:
Harness exception for 'Tests for C6353E8B-BFDA-4F54-93D9-FF0F838BF8EE': System.Exception: Unable to evaluate the property OutputPath.
at Xharness.AppBundleLocator.GetPropertyByMSBuildEvaluationAsync (System.Xml.XmlDocument csproj, System.String projectPath, System.String evaluateProperty, System.String dependsOnTargets, System.Collections.Generic.Dictionary`2[TKey,TValue] properties) [0x00327] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/AppBundleLocator.cs:109
at Xharness.AppBundleLocator.LocateAppBundle (System.Xml.XmlDocument projectFile, System.String projectFilePath, Microsoft.DotNet.XHarness.iOS.Shared.TestTarget target, System.String buildConfiguration) [0x000d2] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/AppBundleLocator.cs:49
at Microsoft.DotNet.XHarness.iOS.Shared.AppBundleInformationParser.ParseFromProject (System.String projectFilePath, Microsoft.DotNet.XHarness.iOS.Shared.TestTarget target, System.String buildConfiguration) [0x00147] in /_/src/Microsoft.DotNet.XHarness.iOS.Shared/AppBundleInformationParser.cs:71
at Xharness.AppRunner.InitializeAsync () [0x00046] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/AppRunner.cs:120
at Xharness.Jenkins.TestTasks.RunSimulator.SelectSimulatorAsync () [0x002df] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/Jenkins/TestTasks/RunSimulator.cs:108
at Xharness.Jenkins.TestTasks.AggregatedRunSimulatorTask.ExecuteAsync () [0x00335] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/Jenkins/TestTasks/AggregatedRunSimulatorTask.cs:63
at Xharness.Jenkins.TestTasks.TestTasks.RunInternalAsync () [0x00226] in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/xharness/Jenkins/TestTasks/TestTask.cs:283
* [tests] Add a .NET/macOS version of monotouch-test.
I made a macOS version of monotouch-test instead of a .NET version of xammac tests,
so that one day we might have only one test suite for all our API tests.
* Add a project file for .NET/macOS
* Fix some code to handle the fact that we're called 'monotouchtest' on macOS (but
only on .NET).
* Ignore exception marshalling tests, those aren't working yet.
* [xharness] Add support for .NET/macOS and add a macOS version of monotouch-test to our tests
* [dotnet-linker] Skip libSystem.Net.Security.Native and libSystem.Native when collecting native methods to preserve for .NET/macOS.
* [link sdk/link all] Adjust to compile on Mac Catalyst.
* [tests] Adjust the LinkAllRegressionTest.NoFatCorlib to work on Mac Catalyst.
* [tests] Add version checks to make link sdk green on Mac Catalyst.
* [tests] Make the LinkSdkRegressionTest.SpecialFolder test pass on Mac Catalyst.
* [xharness] link sdk and link all are green on Mac Catalyst now.
* Fix test build.
* [tests] Use the 'Apple Developer' code signing key instead of 'iPhone Developer' for xcframework-test.
* [msbuild] Fix resolving the XCFramework for Mac Catalyst.
* [xharness] Enable xcframework-test by default on Mac Catalyst.
Fixes this warning when running xharness:
Unknown file: fsharplibrary.fsproj (extension: .fsproj). There might be a default inclusion behavior for this file.
This is mainly to bring in this fix: dotnet/xharness@841114a
(two parameters were swapped and BundleIdentifier was set to AppPath)
Other changes contained:
We don't log the full XML when listing devices anymore, just the file size
SimulatorLoader has a new method that accepts retryCount
* [apitest] Sanitize files by adding missing eols.
* [apitest] Add #if __MACOS__ to all test files.
In preparation for the move into monotouchtest.
* [apitest] Move test files into monotouchtest.
* [tests] Remove the apitest project.
* [monotouch-test] Remove MessagingMac.cs, it's not needed.
* [xammac-tests] Add file PlatformInfo.cs to the build.
* [xammac-tests] Move files into monotouch-test.
* [monotouch-test] Rename test class to not clash with another test class of the same (Objective-C) name.
* [tests] How did this ever work?
Answer: it never did.
* [monotouch-test] Remove duplicated test code.
* [xammac-tests] Define DYNAMIC_REGISTRAR when we're using the dynamic registrar.
* [monotouch-test] Adjust the BundleTest.TestGetBundleId test to cope with having multiple apps for the same bundle id.
* [monotouch-test] Ignore a test that doesn't work with the static registrar (due to a bug in the static registrar).
* Install the Mac Catalyst versions of the mono libraries and BCL.
* The BCL is the same as the one for Xamarin.iOS, which means it has to be post-processed a bit to work with a Xamarin.MacCatalyst.dll
* Build our runtime for Mac Catalyst.
* Build a Xamarin.MacCatalyst.dll with the Mac Catalyst API (it compiles, but I haven't looked at the API surface at all). This PR assumes we're going to have a new TargetFrameworkIdentifier for Mac Catalyst, but a final decision has not been made (see https://github.com/dotnet/runtime/issues/44882), so this may change.
* Build a Xamarin.iOS.dll that contains type forwarders to Mac Catalyst for all the types that exist in both Mac Catalyst and Xamarin.iOS.
* Add support to xharness for running introspection on Mac Catalyst (there are a lot of failures because the API surface is wrong)
* Add support to our msbuild tasks and mtouch for building Mac Catalyst apps. This basically comes down to adding a new case in numerous places to either do things the iOS way or the macOS way, depending on each case.
* Add a __MACCATALYST__ define (which is in addition to the __IOS__ define).
* Bumps mono binaries to include x86_64 watchOS support
* Build runtime/registrar x86_64 slices
* Produce a 64 bit version of Xamarin.WatchOS.dll
* Allow building x86_64 for watch simulators in mtouch
* Let xharness know about x86_64
* [tests] Add x86_64 arch to test-libraries
* Make dotnet package aware of x64
* [ObjCRuntime] Fix computing if we're calling a stret function or not in a 64-bit watchOS simulator.
* [xharness] Re-enable some watchOS tests.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
New commits in mono/mono:
* mono/mono@ac596375c7 Add support for OP_FCONV_TO_I to mini-arm64.c. (#20548)
* mono/mono@392fe5b87b [2020-02][watchOS] Add simwatch64 support (#20552)
* mono/mono@a22ed3f094 Fix potential crash for Encoder.Convert (#20522)
* mono/mono@970783731f Bump to F# 5.0 (#20511)
* mono/mono@32ab5066f7 Bump msbuild to fix a build issue
* mono/mono@93a7fe77e8 Ensure special static slots respect alignment. (#20506)
* mono/mono@3db5b35841 [debugger] Switch to GC Unsafe in signal handler callbacks (#20495)
* mono/mono@af315f44c4 [2020-02][corlib] ThreadAbortException protection for ArraySortHelper (#20468)
* mono/mono@ca11fb0fd8 [2020-02] Bump ikvm-fork to include https://github.com/mono/ikvm-fork/pull/20 (#20452)
Diff: be2226b5a1..ac596375c7
This is done early so we can resolve the inner framework, inside the
xcframework, and let the existing framework support do most of the
work.
The resolving code has unit tests. Custom projects for "NoEmbedding"
exists for all supported platforms and executed by xharness.
A sample `xcframework` with tests projects is also available
[here](https://github.com/spouliot/xcframework).
The xcframework test case is based on Rolf's earlier/partial implementation.
https://github.com/rolfbjarne/xamarin-macios/commit/xcframework
Things to note:
Do not rename a framework (like XTest) to use it in an xcframework
(like XCTest). That will fail at codesign but won't give anything
useful. You might think signing the framework (instead of the inner
binary) would solve it. It does, as it codesign, but then the app
crash at startup. At some point you realize some symbols are still
using XTest (not XCTest) and then you can delete several other weird
workarounds (like for `ld`) because all of it was cause by this
never identified rename.
dSYM support (and tests) to be done in a separate PR.
* [watchOS] Add x86_64 simulator support
* Build runtime/registrar x86_64 slices
* Produce a 64 bit version of Xamarin.WatchOS.dll
* Allow building x86_64 for watch simulators in mtouch
* Let xharness know about x86_64
* [tests] Add x86_64 arch to test-libraries
* Make dotnet package aware of x64
* [ObjCRuntime] Fix computing if we're calling a stret function or not in a 64-bit watchOS simulator.
* [xharness] Re-enable some watchOS tests.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This involves a few changes:
* Change everything to reference net6.0 instead of net5.0
* Update various variables to be NET6* instead of NET5*
* Reorder build logic to account for that our targets are imported earlier in
the build process:
In the latest .NET 6, our Workloads.targets is imported earlier in the
build. This requires a few changes, because we still need to run most of
our logic later in the process, which we do by adding targets files we
want imported later to the AfterMicrosoftNETSdkTargets property.
What we're loading as soon as possible:
* Our version information (Xamarin.Shared.Sdk.Versions.targets)
* The supported OS versions
(Microsoft.<platform>.Sdk.SupportedTargetPlatforms.targets)
* The default OS version
(Xamarin.Shared.Sdk.TargetFrameworkInference.targets).
This is all information that the .NET build require early on.
Changes:
* Rename all files that are loaded early to *.props.
* Updated documentations to reflect these changes.
* Remove Microsoft.<platform>.TargetFrameworkInference.targets, these
files aren't used and don't contain anything useful.
* Move the logic to calculate _ComputedTargetFrameworkMoniker has been
delayed to later, because it needs TargetFrameworkMoniker set.
* Add a StoreAttributesStep to store attributes that are removed by the
linker, but that the static registrar needs.
In particular, in .NET 6 the linker removes the
System.Runtime.CompilerServices.ExtensionAttribute, which the static
registrar needs to handle category methods properly.
This involved copying and slightly modifying the RemoveAttributesBase
code.
This code isn't necessary in main, because we've removed any GuiUnit usage,
but in d16-8 we're still using GuiUnit, which means we still need to process
GuiUnit project references correctly.
This fixes numerous build failures when building Xamarin.Mac tests (in
particular when building Xamarin.Mac tests for older macOS versions).
Manually added `NoBindingEmbedding` to bindings .csproj and removed
`[LinkWith]` attributes.
Also added support for NativeReference to harness and updated the
workaround for watchOS (since it can't link against ModelIO)
Fixed/adapted unit tests wrt change
* [tests] Move the dotnet introspection test project to a new test directory.
A tvOS project will come soon too, this makes the separation nicer.
* [tests] Add a tvOS version of the dotnet introspection test project.
* Remove a lot of dead code.
* Use TestBase.CreateTask<T> to create tasks (and set required properties for all
tasks) instead of instantiating tasks directly. This required subclassing TestBase
in a few places, as well as making a few helper methods instance methods instead
of static methods.
* Bump to net472.
* A few other misc simplifications.
* [xharness] Add support for skipping more test variations.
* [xharness] Add the dotnet introspection project file to our list of projects, and mark it as a .NET project.
* [xharness] Clearly label .NET projects as such.
* [xharness] Disable project generation for .NET projects for now.
* [xharness] Create DotNetBuildTasks when we need to build .NET projects.
* [xharness] Improve project cloning to take into account default inclusion behavior for .NET projects.
.NET projects will include files from the current directory by default, which
means that if we clone the project file and write the cloned project file in a
different directory, we'll have to add an automatically included files into
the cloned file manually.
* [xharness] Add a GetDotNetExecutable callback and an MSBuildPath property to [I]ProcessManager.
* [xharness] Retrieve a .NET project's OutputPath by running MSBuild.
.NET projects are vastly simplified, which means that the OutputPath can't be
determined by reading the project file itself, it has to be calculated by
MSBuild.
So that's exactly what we do: we run MSBuild on the project file and get it to
print the property we're interested in.
* [xharness] Skip the old simulator variation for introspection/dotnet for now.
* [xharness] Try to not have log directories with spaces in them. (#8976)
The simulators won't write stdout/stderr to a path with a space.
* [xharness] Don't generate makefile targets for .NET projects.
They end up with the same make targets as non-.NET targets, which prints
warnings in the terminal when running make.
We're not using the makefile targets much anymore, so postpone implementing
them for .NET until we need them for some reason.
* [xharness] Add support for generating a tvOS version of .NET iOS projects. (#9032)
* [xharness] Add support for generating a tvOS version of .NET iOS projects.
And use it to run the tvOS version of introspection for .NET.
* [xharness] Change according to reviews.
* [xharness] Use Cecil to inspect assemblies, instead of Reflection. (#9082)
This is slightly faster - ~0.95s vs ~1.4s - (probably because reflection tries
to load a lot of other referenced assemblies, which may or may not exist,
causing exceptions (if they don't exist) or spend time loading them (which
Cecil won't)).
It also avoids a lot of exception details showing up when tracing xharness
execution.
* [xharness] Don't group simulators when running in server mode. (#9098)
Grouping simulators means we have to wait until we know which simulators are
available, because we group by the simulator's UDID.
When running in server mode, we don't need to group simulator test runs,
because we run the as the user chooses, not in any particular order.
This way the test list loads faster at startup, since we don't have to wait
until we've loaded the simulators before we can show the list.
* [xharness] Fix finding testable types in test assemblies. Fixes #xamarin/maccore@2267. (#9103)
Apparently a null namespace when using Reflection can be an empty namespace in Cecil.
Fixes https://github.com/xamarin/maccore/issues/2267.
* [xharness] Parse NUnit v2 xml reports correctly. (#9110)
* [xharness] De-asyncify template expansion. (#9104)
This avoids:
* Getting the resource stream for every template expansion
* Reading the resource stream
* A lot of async code.
Instead just read the resource stream once, into a string, and return that.
This speeds up the startup by ~10% (from ~0.96s to ~0.83s on my machine).
* [xharness] Improve code to not throw exceptions in the first place instead of catching and ignoring. (#9118)
* [xharness] Try harder to make lldb quit after trying to get a backtrace. (#9119)
When a process times out, we try to print a stack trace for all threads. This
involves executing "lldb --source <script>", where the script contains:
process attach --pid 1234
thread list
thread backtrace all
detach
quit
Basically we attach to the project, and ask lldb to print stack traces.
The problem:
16:09:02.9522580 Printing backtrace for pid=25276
16:09:02.9528060 /usr/bin/lldb --source /var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp
16:09:04.6127570 (lldb) command source -s 0 '/var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp'
16:09:04.6130020 Executing commands in '/var/folders/q7/mkzwrzcn7bzf3g2v38f3c1cw0000gn/T/tmp58e75d85.tmp'.
16:09:04.6130200 (lldb) process attach --pid 25276
16:09:05.6458750 error: attach failed: Error 1
16:09:05.7529100 25276 Execution timed out after 1200 seconds and the process was killed.
16:09:05.7588770 Execution timed out after 1200 seconds.
If any of those commands fail, the subsequent commands aren't executed. This
includes the final "quit" command, which means we end up waiting forever for
lldb to do its thing, when lldb doesn't think it needs to do anything at all.
The fix: pass two different scripts. It turns out subsequent scripts are
executed even if previous scripts fail, so we do the equivalent of:
lldb --source <attach script> --source <quit script>
And now lldb will quit no matter what the attach script does (it still works
even if the attach script succeeds, in which case we'll ask lldb to quit
twice).
* [xharness] Improve process killing a bit. (#9120)
* Log what we're doing.
* Wait a bit after sending SIGABRT, to allow for the OS to create any crash
reports.
* [xharness] Don't try to write cleaned xml into the same file as the dirty xml was. (#9122)
* [xharness] Don't replace existing files, instead add new files, when providing human-readable logs. (#9123)
This makes understanding what's going on much easier.
* [xharness] Fix BCL project generation for watchOS projects. (#9131)
Reading from a file named as the actual content of the watchOS template doesn't
_quite_ work. Instead just use the content directly.
* [xharness] Only try to parse the build log if we have a build log. (#9130)
* [xharness] Don't generate into the same directory/files for macOS Modern and macOS Full. (#9121)
* [xharness] Don't generate into the same directory/files for macOS Modern and macOS Full.
This meant that we were overwriting some generated files, which meant that we
were executing the Modern set of tests instead of the Full set of tests for at
least some configurations.
* Add a few ignored tests to the System.Configuration test now that we actually run it.
* [xharness] Disable default inclusion behavior for the introspection .NET tests. (#9137)
It gets too annoying to keep track of which files are where with the project
cloning we're doing, if files are implicitly included.
* [xharness] Generate BCL projects asynchronously. (#9083)
Split the BCL project generation in two: one part that figures out which
projects to generate, and which is done synchronously (because we need it to
compute the list of tests), then split out the actual generation and run it on
a background thread (since that doesn't have to happen until we want to
execute those tests).
This speeds up launching xharness in server mode significantly (from ~2s to
~0.2s).
* [xharness] Document the test loading process a little bit.
* [xharness] Generate projects into a subdirectory for each project.
Because it's cumbersome to keep the build working when the same directory has
multiple project files.
* [xharness] Remove a lot of special-casing for the mono-native tests.
It's not needed anymore.
* [xharness] Store attribute in local variable instead of fetching it twice.
* [xharness] Make ProjectFileExtensions.GetInfoPListInclude a bit clearer.
Add a comment to document what's supposed to happen, and simplify the code a little bit.
* [xharness] Add support for specifying whether variations should be generated for Xamarin.iOS projects.
* [xharness] Don't generate variations for BCL projects using the normal procedure, we generate them differently already.
This makes it possible to remove some dead code.
* [xharness] Add a TestPlatform value to TestProject, and use it to indicate that a TestProject might be of a specific platform already.
* [xharness] Make the templates path an input parameter for the BCL templates.
* [xharness] Make the BCL project generation generate each project file into its own subdirectory.
* [tests] Put the iOS and macOS mono-native templates in their own directory.
This makes it possible to remove a few conditions in the code.
* [xharness] Simplify the MonoNativeInfo class a bit by removing the custom Mac-specific subclass (there's almost no difference anymore).
* [tests] Move bindings-test[-mac].csproj to their own directory.
* [xharness] Fix compiler warning. (#9147)
* [xharness] Fix logic to not create makefile targets to run BCL tests. (#9148)
According to the existing code, we're already not supposed to create makefile
targets for BCL tests. However, the code to detect BCL tests wasn't quite
reliable, so we ended up creating makefile targets for some BCL tests.
So always use the same logic to detect BCL tests.
Also remove a lot of dead code to generate makefile code for BCL tests.
* [tests] Set the correct mono-native link mode and flavor defines in the csproj itself.
This makes it possible to simplify a lot of C# code.
* [xharness] Remove a redundant ToArray.
* [xharness] Allow for null environment variables to mlaunch. (#9140)
It's a valid scenario to set a null environment variable.
This happens when changing InCI to always return true (to test locally what's
done on the bots), in which case we try to forward BUILD_REVISION to mlaunch,
but BUILD_REVISION isn't necessarily set. Not forwarding it to mlaunch should
not cause problems in most tests, so just allow this scenario.
* [xharness] Generate makefile targets for non-executable projects. (#9158)
It turns out some Xamarin.Mac projects are library projects (for unit tests),
and we still need to generate makefile targets for those (because we use them
to build Xamarin.Mac tests to run on older macOS bots).
* [xharness] Backport Touch.Unit fix for command line parsing. (#9165)
Apply https://github.com/spouliot/Touch.Unit/pull/78 to the duplicated file.
* [tests] Info.plists should be inside LogicalName, not Link.
This makes it possible to simplify a bit more code inside xharness as well.
* Merge remote-tracking branch 'origin/main' into xharness-a-directory-per-project
* [xharness] Fix finding Info.plist, and add some tests too.
* [xharness] Fix how .NET projects are ignored. (#9168)
* [xharness] Fix how .NET projects are ignored.
Now they're actually ignored if .NET tests are not included.
* Fix compile error.
* Merge remote-tracking branch 'origin/main' into xharness-a-directory-per-project
* [xharness] If we're not generating projects in the Target class, the project path equals the template path.
* [xharness] Make restoring nugets the default for all test projects. (#9237)
But don't do it for .NET projects, since 'dotnet build' automatically restores.
* [xharness] Improve restoring NuGets for Xamarin.Mac projects in the makefile by depending on the csproj files in addition to the packages.conf files. (#9242)
Packages are listed in the csproj when we're using package references.
Also only list files in git, otherwise we pick up all the generated project files as well.
* [tests] Migrate the sample tests to use package references. (#9244)
* [tests] Switch msbuild-mac to use package references. (#9245)
And don't specify the solution path anymore, because we can restore using just
the csproj just fine now.
* [xharness] Mark the .NET tests project as a .NET project so that we don't try to call nuget restore on it. (#9251)
* [xharness] Mark the .NET tests project as a .NET project so that we don't try to call nuget restore on it.
* [xharness] Mark the .NET generator project as a .NET project as well.
* [xharness] Resolve relative project references using the referencing project's directory. (#9252)
Otherwise it would end up being resolved from the current directory, which is
not dependable.
* [xharness] Don't try to parse a build log that doesn't exist. (#9253)
* [xharness] Add support for generating/duplicating projects with a Touch.Client reference.
This also means removing support for generating/duplicating MonoTouch.NUnit references,
since we're not using those anymore.
* [tests] Remove dead code related to GuiUnit and the GuiUnit submodule.
* [xharness] Change how we run macOS unit tests to work with Touch.Client.
This is very similar to how we run iOS unit tests.
* [xharness] Add the Touch.Client projects to our solutions instead of MonoTouch.NUnitLite.
* [tests] The upstream NUnitLine isn't linker safe, so add an xml file to make sure linking doesn't remove too much.
In particular NUnit uses reflection to get a private method, and the linker removes
the corresponding private method:
1c680b4dc8/src/NUnitFramework/framework/Internal/TestExecutionContext.cs (L552)
So add an xml definition to keep this private method, and modify project files to
pass the xml definition to mtouch and mmp.
Some care needs to be taken to make sure xharness is still able to clone these project
files.
* [xharness] Add support for finding/setting the main Xamarin import among multiple imports.
This is required for the previous commit, where we added an additional import to
a few project files to pass an xml definition to mtouch/mmp.
* [xharness] Use LogFile instead of ResultFile in macOS BCL tests to match the Touch.Client usage.
* [tests] Restore projects before building them from the command line.
* [xharness] Implement Log.GetReader for all loggers. (#9257)
This makes it easier to write code that parses logs, since that code won't
have to worry about if a particular log supports GetReader or not.
* [xharness] Fix main log parsing by not requiring it to come from a file. (#9258)
Not all types of logs have files, but all types of logs have a reader (or will
have soon due to PR #9257).
* [xharness] Improve parsing of NUnitV2 xml reports to fix an issue with failures in parameterized tests. (#9259)
We weren't properly reporting all failures when there were multiple failures
in parameterized tests, because we'd incorrectly skip too many xml nodes when
we were looking for the next test failure.
* Merge remote-tracking branch 'origin/main' into HEAD
* Remove wrongly added dir. (#9301)
Someone (Manuel) uses an IDE that uses .idea to store user settings and
that got added by accident.
* [Harness] Fix some typos. (#9303)
* [xharness] Make it possible to include .NET tests by setting a label. (#9298)
This also means that setting 'run-all-tests' enables the .NET tests, which
means they'll now start running on internal Jenkins (as was the intention from
the beginning).
* [xharness] Look at the exact NUnit version an NUnit project is referencing to figure out how to run it in a makefile. (#9322)
* [xharness] Add support for adding PropertyGroups for unknown platform/configuration values in a csproj. Fixes xamarin/maccore@2277. (#9347)
Fixes https://github.com/xamarin/maccore/issues/2277.
* [xharness] Add support for getting the default AssemblyName for projects. (#9369)
In .NET projects there's a default value for most properties, which means that
there won't necessarily be an AssemblyName property in a csproj. We need to know the
AssemblyName, so calculate it from the csproj filename (which is how .NET does it).
This turned out slighly complicated, because we're pass an XmlDocument around,
and the XmlDocument doesn't know the file from where it was loaded, so we need
to keep that information separately.
* [xharness] Add the .NET version of interdependent-binding-projects to our lineup
* [xharness] Enhance ResolveAllPaths to understand $(RootTestsDirectory).
This requires passing the root directory around in multiple places, since ResolveAllPaths
doesn't have access to the static class where we define the root directory.
Also call ResolveAllPaths in a few more places to ensure paths everywhere are resolved.
* [xharness] Do not mark DeviceLoaders as loaded when loading fails (#9367)
* [xharness] Make sure MSBUILD_EXE_PATH is not set when executing dotnet.
Something ends up confused, and dotnet hangs.
* [xharness] Bump timeout for the .NET tests.
More tests take longer to run!
* [xharness] Don't clone the same project multiple times when reached through different project references.
Problem:
1. interdependent-binding-project references binding-test and bindings-test2.
2. bindings-test2 references bindings-test.
This means bindings-test is reached through 2 projects: both
interdependent-binding-projects and bindings-test2, and previously we'd
process each of these references separately, effectively making two separate
clones of the bindings-test project.
Instead keep track of all referenced projects from the leaf project, and if we
encounter a project we've already cloned, use that directly.
* [dotnet] Add support for consuming binding projects. (#9376)
* Port the interdependent-binding-projects test to .NET (it's the simplest
test project we have with binding projects).
* Add a lot of the shared source code for mtouch/mmp to dotnet-linker, and
make it compile. Most issues were fixed by adding a few stubbed out classes,
since there are large chunks of the mtouch/mmp code we're not using yet, so
stubbing out while things are being implemented works fine.
* Add a step in dotnet-linker for loading the linker output (the linked
assemblies) into our bundler code.
* Add another step in dotnet-linker to extract native resources from binding
libraries.
* Augment the build process to take into account the native resources we found
in any binding libraries.
* [xharness] Make it possible to make a project ignored by default from its initial definition.
Make it possible to make a project ignored by default from its initial definition,
and which takes precedence over any other selection criteria.
This way it's possible to make the .NET version of monotouch-test show up in the
web view, which makes it runnable locally (for testing), while at the same time it
will never be run on CI.
* [xharness] Add monotouch-test to the .NET lineup.
* [xharness] Improve resolving paths in project files.
* Properties may contain a list of files, separated by semi-colon, so if we find
any semi-colons, treat each entry as a separate path.
* Treat anything that starts with a variable as a full path, because either the
variable is a full path, or it will be updated according to the new project location
and resolve correctly.
* [xharness] Process '$(RootTestsDirectory)' before bailing due to any other variables when resolving paths in project files. (#9467)
While keeping '$(RootTestsDirectory)' as-is works fine when building the
project using MSBuild, xharness itself may end up encountering it (in
particular when listing referenced projects), and at that point will end up
confused. So just always resolve '$(RootTestsDirectory)' to the actual
directory to keep other parts of xharness happy.
Fixes this problem (as seen in PR #9460):
Harness exception for 'interdependent-binding-projects': System.IO.DirectoryNotFoundException: Could not find a part of the path "/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/external/Touch.Unit/Touch.Client/dotnet/iOS/Touch.Client-iOS.dotnet.csproj".
at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0015e] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.IO/FileStream.cs:223
at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean isAsync, System.Boolean anonymous) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.IO/FileStream.cs:149
at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2020-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.IO/FileStream.cs:86
at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess)
at Microsoft.DotNet.XHarness.iOS.Shared.Utilities.PListExtensions.LoadWithoutNetworkAccess (System.Xml.XmlDocument doc, System.String filename) [0x00001] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Utilities/PlistExtensions.cs:17
at Microsoft.DotNet.XHarness.iOS.Shared.TestProject.CreateCopyAsync (Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, Microsoft.DotNet.XHarness.iOS.Shared.Execution.IProcessManager processManager, Microsoft.DotNet.XHarness.iOS.Shared.Tasks.ITestTask test, System.String rootDirectory, System.Collections.Generic.Dictionary`2[TKey,TValue] allProjectReferences) [0x0010c] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/TestProject.cs:98
at Microsoft.DotNet.XHarness.iOS.Shared.TestProject.CreateCopyAsync (Microsoft.DotNet.XHarness.iOS.Shared.Logging.ILog log, Microsoft.DotNet.XHarness.iOS.Shared.Execution.IProcessManager processManager, Microsoft.DotNet.XHarness.iOS.Shared.Tasks.ITestTask test, System.String rootDirectory, System.Collections.Generic.Dictionary`2[TKey,TValue] allProjectReferences) [0x00811] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/TestProject.cs:168
at Microsoft.DotNet.XHarness.iOS.Shared.Tasks.TestTasks.RunInternalAsync () [0x00117] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xharness/Microsoft.DotNet.XHarness.iOS.Shared/Tasks/TestTask.cs:274
* [tests] Port dont link to .NET (#9488)
There are a few test failures, so it's ignored by default in xharness for now.
* [xharness] don't link is now green on .NET!
* [tests] Port link sdk to .NET
* [xharness] link sdk is now green on .NET!
At least the Debug configuration is, there's still a test failure in the Release
configuration, so skip that for now.
* [tests] Port link all to .NET (#9515)
* [tests] Port link all to .NET
* [tests] Exclude link all tests in .NET that reference OpenTK-1.0.dll
OpenTK-1.0.dll isn't supported yet.
* [tests] Exclude link all tests using API that isn't available anymore in .NET.
* [xharness] Use the local 'ignore' variable instead of fetching 'test.TestProject.Ignore' for every test when generating test variations. (#9518)
* [xharness] Enable monotouch-test/.NET by default, but ignore all variations. (#9526)
* [xharness] Enable monotouch-test/.NET by default, but ignore all variations.
The default (Debug) configuration is green now, but the rest of the variations aren't yet.
* [tests] Ignore LocaleTest.InitRegionInfo in .NET because it needs globalization data.
Ref: https://github.com/xamarin/xamarin-macios/issues/8906
I can't reproduce locally, because it's ignored for me:
[INCONCLUSIVE] InitRegionInfo : You can construct locale without countries
at MonoTouchFixtures.Foundation.LocaleTest.InitRegionInfo() in [...]/xamarin-macios/tests/monotouch-test/Foundation/LocaleTest.cs:line 47
but this should fix this monotouch-test test failure (I'm guessing that 'IV'
stands for 'Invariant', which is all the current .NET version ships with):
MonoTouchFixtures.Foundation.LocaleTest
[FAIL] InitRegionInfo : Name
String lengths are both 2. Strings differ at index 0.
Expected: "US"
But was: "IV"
-----------^
: at MonoTouchFixtures.Foundation.LocaleTest.InitRegionInfo() in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/monotouch-test/Foundation/LocaleTest.cs:line 49
* [xharness] Enable the .NET tests when a file containing 'dotnet' is modified. (#9510)
* [xharness] Enable the .NET tests when a file containing 'dotnet' is modified.
This also required adding support for using regex as file names when selecting
which tests to run.
* Remove redundant dotnet prefix.
* [xharness] Rename variables to match usage.
* [dotnet] Use net5.0-[ios|tvos|watchos|macos] TargetFrameworks. (#9532)
* [dotnet] Set TargetPlatformSupported when the right TargetPlatformIdentifier is used.
* [dotnet] Generate a list of valid OS versions for each platform, and add it to the SupportedTargetPlatform item group.
The generated files: https://gist.github.com/rolfbjarne/765c4e4b38b1b017380c9378d9666317
* [dotnet] Define and set the default platform version if it's not set in the TargetFramework variable.
* [dotnet] Switch to using the new-style TargetFramework values.
This includes bumping spouliot/Touch.Unit to get new-style TargetFramework values for Touch.Client.
* spouliot/Touch.Unit@89afaf7 [Touch.Client] Use the right TargetFrameworks for watchOS and macOS as well. (#92)
* spouliot/Touch.Unit@fd0e576 [Touch.Client] Use the right TargetFrameworks. (#91)
* spouliot/Touch.Unit@40f47db [Touch.Client] Add a macOS and watchOS version for .NET. (#90)
* spouliot/Touch.Unit@1d4b8c0 Add .gitignore for NuGet.config and global.json. (#89)
* spouliot/Touch.Unit@49441f3 Call `mlaunch` instead of `mtouch` (#88)
* spouliot/Touch.Unit@b157cf4 Fix a few markdown issues found by markdownlint. (#87)
Diff: d7f55a6167..89afaf7e05
* [dotnet] Document the script that generates the lists of the target platform versions a little bit better.
* [dotnet] Make the [Platform]SupportedTargetPlatform variables public.
This matches how Android and Windows do it:
* https://github.com/xamarin/xamarin-android/pull/5007
* 18ee4eac8b/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.WindowsSupportedTargetPlatforms.props
* [xharness] Update the TargetFramework value when creating project variations.
* [dotnet] Pass the Optimize flags from the extra bundler arguments to the linker configuration. (#9599)
* [dotnet] Pass the Optimize flags from the extra bundler arguments to the linker configuration.
Also call Application.InitializeCommon to initialize the application instance. The
important part here is that InitializeCommon calls Optimizations.Initialize to compute
the default optimizations. It also calls Set*ExceptionMode and sets the default EnableCoopGC
value (so we don't need to call/set those anymore), and it does a few other initialization
tasks which we don't need yet, but eventually will.
And finally remember to parse the bundler arguments before using them in the dotnet
build logic. How did this not cause problems before? 🤦
* [tests] Set the verbosity using the additional args instead of an internal variable.
The internal _BundlerVerbosity variable is overwritten now (with the verbosity
value from the additional args).
* [xharness] Disable tvOS generation for the introspection/.NET test, it incorrect and needs fixing.
* [dotnet-linker] Add OptimizeGeneratedCodeSubStep into the pipeline. (#9608)
Fixes these linkall tests:
Linker.Shared.OptimizeGeneratedCodeTest
[FAIL] IsARM64CallingConvention : optimized: no ldsfld instruction
Expected: 0
But was: 1
at Linker.Shared.BaseOptimizeGeneratedCodeTest.IsARM64CallingConvention() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 527
[FAIL] SetupBlockPerfTest : At least 6x speedup
Expected: greater than 6
But was: 1.0876440665344851d
at Linker.Shared.BaseOptimizeGeneratedCodeTest.SetupBlockPerfTest() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 120
And linkall is now green for .NET/Debug.
* [xharness] Add support for adding PropertyGroups if none are applicable when adding a property. (#9616)
Some .NET test projects (monotouch-test) require this to set properties for
some test variations, and .NET projects are quite minimal, making it likely
that they won't have a particular property that needs to be set/modified.
This makes it possible to execute the various monotouch-test variations for
.NET in xharness (some of the variations still fail though, so they're still
ignored by default).
* [tests] Ignore a few link sdk tests in .NET for now. (#9614)
There are easier fish to fry right now, so implementing the linker support for
these tests to pass (metadata reduction and code removal) can wait.
This makes the link sdk/Release variation green for .NET.
Ref: https://github.com/xamarin/xamarin-macios/issues/9612
Ref: https://github.com/xamarin/xamarin-macios/issues/9613
* [xharness] The linkall tests are green for .NET/Release now. (#9629)
* [xharness] Skip the 'dont link / AssemblyBuildTarget: SDK framework (release)' variation by default for tvOS. Fixes #xamarin/maccore@2282. (#9659)
This test variation ends up being too big (1.5 GB), so it doesn't install
properly and the app crashes at startup.
So just skip it.
Fixes https://github.com/xamarin/maccore/issues/2282.
* [src] Build the .NET version of our product assemblies using a .NET 5 BCL. (#9637)
* [src] Build the .NET version of our product assemblies using a .NET 5 BCL.
We're not shipping the .NET product assemblies in any stable release, so we
can use a preview version of the .NET 5 BCL.
Also:
* Add all the nuget feeds we need to the top-level NuGet.config, even for .NET
5/6, there shouldn't be any conflicts with stable feeds since we use exact
version numbers.
* Generate a top-level global5.json which is copied to every directory that
needs a .NET 5 global.json (overriding the .NET 3.1 global.json in the root
directory).
* Use the expected dotnet binary during our local build.
* [tests] Fix the bgen tests to use .NET 5.
* [xharness] Set the current directory to the project directory when running .NET tests.
This way we end up using the dotnet version that's configured in global.json for the tests.
* [xharness] Fix typo in test case creating two identical test cases. (#9702)
* [xharness] Use XmlReaderSettings to ignore stuff we don't care about when parsing xml. (#9703)
At the very least this makes stepping through code while debugging much
faster, since stepping through whitespace and comments is useless.
* [xharness] Fix rendering human-readable results for parameterized tests. Fixes#9400. (#9704)
We weren't probably parsing nested test-suite elements in the xml, so
implement parsing of test-suite elements using recursion, which is easier and
less error-prone (and turns out to work fine too).
Also use TextReader/Writer instead of StreamReader/Writer in a few places.
This makes it easier to write tests, since TextReader/Writer can use in-memory
streams (no need to mess with temporary files).
Fixes https://github.com/xamarin/xamarin-macios/issues/9400.
* [dotnet-linker] Add a CollectUnmarkedMembersStep that will keep linked away types around for the static registrar. (#9722)
The static registrar may need access to types that have been linked away, so
store unmarked types so that the static registrar can access them later.
This also makes all the monotouch-test variations green, so enable them all.
Fixes this monotouch-test when all optimizations are enabled:
MonoTouchFixtures.ObjCRuntime.RegistrarTest
[FAIL] TestProtocolRegistration : UIApplicationDelegate/17669
Expected: True
But was: False
at MonoTouchFixtures.ObjCRuntime.RegistrarTest.TestProtocolRegistration() in xamarin-macios/tests/monotouch-test/ObjCRuntime/RegistrarTest.cs:line 1350
* [xharness] Only run xammac variations if macOS tests are included. (#9729)
This fixes a build error with a xammac test variation that is executed even if
the macOS build is disabled.
* [xharness] The dotnet version of introspection/iOS 10.3 works fine now. (#9728)
* [xharness] Disable iOS 32-bit simulator tests, they don't work on Xcode 12+. (Re)fixes #xamarin/maccore@2284. (#9732)
Fixes https://github.com/xamarin/maccore/issues/2284.
* Undo does changes that ship and we do not want to bring with us.
* Fix issues that do not allow compilation.
* Fix mtouch tests.
* Ignore generated files.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Merge DotNetBuildTask into MSBuildTask to simplify creation of build tasks -
these tasks are very similar, and we instantiate MSBuildTasks in over a dozen
places, and this way we don't have to update every instantiation to check if
an MSBuildTask or a DotNetBuildTask should be created.
The end result is that code that does 'new MSBuildTask (...)' will just work
without updates for .NET projects.
Split the iOS msbuild tests in two:
* Xamarin.MacDev.Tasks.Tests: contains in-process unit tests for tasks.
* Xamarin.MacDev.Tasks: contains larger tests that either invoke targets or a complete
build. These are currently in-process, but will become out-of-process soon to make
it possible to run them with dotnet.
Also make the new projects non-iOS-specific, because the macOS msbuild tests will
be moved here as well soon.
There is some duplicated code between these two test projects now (all files
that show up as new are copies of existing files), this will be cleaned up in
later pull requests.
* [msbuild] Move msbuild/tests to tests/msbuild to put all the tests together.
* [tests] Move test projects for Xamarin.Mac to tests/common/TestProjects
* [tests] Move test projects for Xamarin.iOS to tests/common/TestProjects
Manuel de la Pena
Rolf Bjarne Kvinge
Přemek Vysoký
Alex Soto
Sebastien Pouliot