7756d957f9
Also add a linker descriptor to the monotouch-test/.NET project file to preserve the parts of nunit.framework.dll that aren't linker safe. This fixes an unhandled exception that causes a crash during the monotouch-test test run when the linker is enabled: Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object Thread 11 Crashed:: tid_15403 Dispatch queue: com.apple.root.default-qos 0 libsystem_kernel.dylib 0x00007fff5220c33a __pthread_kill + 10 1 libsystem_pthread.dylib 0x00007fff522b6e60 pthread_kill + 430 2 libsystem_c.dylib 0x00007fff5219bb7c abort + 120 3 libxamarin-debug.dylib 0x000000010c93979f xamarin_unhandled_exception_handler + 47 4 libmonosgen-2.0.dylib 0x000000010cc04980 mono_invoke_unhandled_exception_hook + 131 5 libmonosgen-2.0.dylib 0x000000010cb40166 mono_handle_exception_internal + 5915 6 libmonosgen-2.0.dylib 0x000000010cb3ea47 mono_handle_exception + 11 7 libmonosgen-2.0.dylib 0x000000010cb8ea06 mono_amd64_throw_exception + 158 8 ??? 0x000000010d4125b0 0 + 4517340592 9 libxamarin-debug.dylib 0x000000010c9394fb xamarin_process_managed_exception + 379 (runtime.m:2361) 10 libxamarin-debug.dylib 0x000000010c939377 xamarin_process_managed_exception_gchandle + 55 (runtime.m:1144) 11 libxamarin-debug.dylib 0x000000010c939333 xamarin_ftnptr_exception_handler + 19 (runtime.m:1133) 12 libmonosgen-2.0.dylib 0x000000010cb3feeb mono_handle_exception_internal + 5280 13 libmonosgen-2.0.dylib 0x000000010cb3ea47 mono_handle_exception + 11 14 libmonosgen-2.0.dylib 0x000000010cb8f2dd handle_signal_exception + 67 15 ??? 0x000070000fd730d8 0 + 123145568071896 16 ??? 0x000000011a3ab99b 0 + 4735023515 17 ??? 0x000000011a3aa643 0 + 4735018563 18 libdispatch.dylib 0x00007fff52090a0f _dispatch_queue_override_invoke + 1071 19 libdispatch.dylib 0x00007fff5209dcc5 _dispatch_root_queue_drain + 350 20 libdispatch.dylib 0x00007fff5209e46a _dispatch_worker_thread2 + 102 21 libsystem_pthread.dylib 0x00007fff522b39f7 _pthread_wqthread + 220 22 libsystem_pthread.dylib 0x00007fff522b2b77 start_wqthread + 15 |
||
---|---|---|
.. | ||
BundledResources | ||
EmbeddedResources | ||
api-shared | ||
apitest | ||
bcl-test | ||
bgen | ||
bindings-framework-test | ||
bindings-test | ||
bindings-test2 | ||
cecil-tests | ||
common | ||
dotnet | ||
framework-test | ||
fsharp | ||
fsharplibrary | ||
generator | ||
interdependent-binding-projects | ||
introspection | ||
linker | ||
mac-binding-project | ||
mini | ||
misc | ||
mmp-regression | ||
mmptest | ||
mono-native | ||
monotouch-test | ||
msbuild-mac | ||
mtouch | ||
qa-regression | ||
sampletester | ||
scripted | ||
scriptingbridge | ||
templates | ||
test-libraries | ||
xammac_tests | ||
xharness | ||
xtro-sharpie | ||
.gitignore | ||
HtmlTransform.xslt | ||
Makefile | ||
README.md | ||
launch-manual-device-tests.sh | ||
launch-manual-sample-tests.sh | ||
nunit.framework.targets | ||
nunit.framework.xml | ||
package-mac-tests.sh | ||
test-dependencies.sh | ||
tests-mac.sln | ||
tests.sln |
README.md
Various tests to be executed prior to releases
Test solutions
Many of the test solutions and test projects are generated, and will
only be available after running make
once.
- tests.sln: This is the base test solution for Xamarin.iOS, which targets iOS using the Unified API. Not generated.
- tests-tvos.sln: All the TVOS test projects. Generated.
- tests-watchos.sln: All the WatchOS test projects. Generated.
- tests-mac.sln: This is the base test solution for Xamarin.Mac. Not generated.
Test solution/project generation
The tool that generates the test solutions / projects is called xharness, and lives in the xharness subdirectory.
Types of Tests
Unit Tests
Most of the projects are using NUnit[Lite] and looks like unit tests. They are meant to be executed on the target: simulator, devices, OSX.
In reality most of them are regression tests - but that does not change the need to execute and continually expand them to cover new code.
Introspection Tests
Introspection tests are executed on target (both simulator and device for iOS) or a specific version of OSX. The application proceed to analyze itself using:
System.Reflection
for managed code; and- the ObjectiveC runtime library for native code
and compare the results. E.g. if using .NET reflection it can see a binding
for a NSBundle
type then it should be able to find a native NSBundle
type using the ObjC runtime functions. Otherwise an error is raised...
Since the application analyze itself it must contains everything we wish
to test. That's why the introspection tests are part of the dontlink.app
application (for iOS) and the dontlink-mac project (for OSX).
Pros
- They always tell the truth, which can differ from documentation
Cons
- Incomplete - Not everything is encoded in the metadata / executable;
- Too complete - Not every truth is good to be known (or published)
Extrospection Tests
Extrospection tests takes data from some outside sources and see if our implementation match the information, e.g.
- Header files from the SDK;
- Rules, like Gendarme or FxCop;
Since this is done externally there's no need to run them on the devices, simulator or even a specific version of OSX.
Pro
- There is more data available, e.g. information lost when compiling
Con
- The data might not represent the truth (errors, false positives...)
Xamarin.Mac
Many tests when run for macOS use a integration hack which helps handle a number of issues:
- Allowing command line arguments to tests while excluding "psn" arguments passed in while debugging with Visual Studio for Mac
- Invoking
_exit
to work around a number of post-test hangs. See the bug for details. - Add a number of "default" excludes for mono BCL tests
One very useful "hack" this support adds is the ability to run a single test from the command line via the XM_TEST_NAME
environmental variable. For example
XM_TEST_NAME=MonoTouchFixtures.Security.KeyTest.CreateRandomKeyWithParametersTests make run-mac-unified-xammac_tests
Test Suites
*-tests : where * is the assembly name, e.g. monotouch
Use the project defaults for linking, i.e.
-
"Don't link" for simulator
-
"Link SDK assemblies only" for devices
dontlink
-
regression testing without using the linker
-
both simulator and devices are set to "Don't link"
linkall
-
regression testing using the linker on the all assemblies
-
"Link all assemblies" for both simulator/devices
linksdk
-
regression testing using the linker on the SDK assemblies
-
"Link SDK assemblies only" for both simulator/devices
bcl-test
These are the Mono BCL test suite tweaked to run on the mobile profile. It reuse the files directly from mono's repository (linking, not copying).
As other unit tests the configuration is set to mimick normal apps, e.g.
-
"Don't link" for simulator
-
"Link SDK assemblies only" for devices
Common make targets
Run every test in both the simulator and on device, using both the compat and the new profile (for the simulator both in 32 and 64bit mode).
$ make run
Run every test in the simulator, using both the compat and the new profile (both 32 and 64bit simulators).
$ make run-all-sim
Run every test on device, using both the compat and the new profile
$ make run-all-dev
Detailed make targets
-
Main target
- run-what-where-project: Builds, installs (if applicable) and runs the specified test project on the specified platform. This is the most common target to use.
- build-what-where-project: Will build the specified test project for the specified platform and target.
- install-what-where-project: Will install the specified test project on a connected device. There's currently no way to select the device, so ensure you've only one connected (if many devices are connected, it's random which will used).
- exec-what-where-project: Will run the specified test project in the simulator or on a device.
-
What
- -ios-: iOS.
- -tvos-: TVOS.
- -watchos-: WatchOS
If "What" is skipped, all variations are executed sequentially.
-
Where
- -simclassic-: Simulator using the Classic API. Only applicable when platform is iOS.
- -simunified-: Simulator using Unified API. The build will contain both an i386 and an x86_64 binary. Only applicable to the build-* target, while the -sim32- and -sim64- are only applicable to the exec-* targets. Only applicable when the platform is iOS.
- -sim32-: 32bits iOS simulator using the Unified API. Not applicable to other platforms.
- -sim64-: 64bits iOS simulator using the Unified API. Not applicable to other platforms.
- -sim-:
- iOS: Both the -simclassic- and -simunified- versions.
- WatchOS/TVOS: The WatchOS/TVOS simulator.
- -devclassic-: Device using the Classic API. Only applicable when the platform is iOS.
- -devunified-: Device using the Unified API. The build will contain both an armv7 and an arm64 binary. It's not possible to select a 32/64bit version, you'll run what your device supports. Only applicable when the platform is iOS.
- -dev-:
- iOS: Both the -devclassic- and -devunified- versions.
- WatchOS/TVOS: A Watch or TV device.
-
Examples
$ make run-ios-sim32-monotouchtest: This will run
monotouch-test
using the Unified API in a 32-bit simulator. $ make run-tvos-dev-dont\ link: This will rundont link
on an Apple TV device.
Utility run-* targets
These targets will build, install (if applicable) and run the specified project(s).
-
Simulator
- run-sim-project: Builds and runs the specified test project in the simulator in compat, 32 and 64bit mode.
- run-sim: Builds and runs all the non-bcl test projects in the simulator in compat, 32 and 64bit mode.
-
Device
- run-dev-project: Builds and runs the specified non-bcl test project on a device in compat and native mode (if it's 32 and 64bit depends on the device; 64bit devices will run in 64bit mode and the same for 32bit devices).
- run-devcompat: Run all the non-bcl test projects on device, in compat mode.
- run-devdual: Run all the non-bcl test projects on device, in native mode (if it's 32 and 64bit depends on the device; 64bit devices will run in 64bit mode and the same for 32bit devices).
- run-dev: Run all the non-bcl test projects on device, in both compat and native mode.