xamarin-macios/tests
Alex Soto 97f46ea685 Various Xcode 9 API improvements and also fixes bug 59433 (#2676)
https://bugzilla.xamarin.com/show_bug.cgi?id=59433

While fixing bug 59433 I noticed some additional issues outlined below:

AVDepthData:

* Renamed **non** static `Create` methods because `Create` only
  makes sense with the **Static** method in this context. Also
  by renaming the methods we are now closer to the names that
  swift uses.

* Kept descriptive method names in favor of self-documenting code.
  `Convert`, `Apply` and `Replace` do not fully give us the intent
  of the method.

* Added a convenience `Create` static method that takes a
  `CGImageAuxiliaryDataInfo`.

* AvailableDepthDataTypes is now an array of `CVPixelFormatType` instead
  of a `NSNumber` array (The aactual fix for bug 59433).

ImageIO:

* Refactored `CGImageAuxiliaryDataInfo` to be a `StrongDictionary` in order
  to avoid most of the manual code and also to avoid reimplementing
  the `ToDictionary` method (which contained a subtle bug).

* Adjusted code to reflect the above change.

* Added missing `.ctor (IntPtr, bool)` to `CGImageMetadata` so the class
  is able to be created by our `Runtime.Get*`.

* Simplified `CopyAuxiliaryDataInfo` method by using `CGImageAuxiliaryDataInfo`
  as a `DictionaryContainer`.

Tests:

* Added `xamarinmonkey.heic` which is an image that contains depth data needed
  to test above changes.

* Adds `AVDepthDataTests` that touches most of the changes listed here.
2017-09-13 09:58:08 -05:00
..
BundledResources [tests] Add support for building device tests for 32-bit or 64-bit. 2016-10-11 19:52:36 +02:00
EmbeddedResources [tests] Explicitly state the resource name for resources. 2017-02-10 16:39:38 +01:00
api-shared [registrar] Use metadata tokens instead of strings to find types and methods. (#1085) 2016-11-01 14:34:56 -04:00
apitest [58851] Disable NSMenuView in AllItemsWithNSMenuShouldAllowNull (#2504) 2017-08-23 12:04:07 -08:00
bcl-test [Do not merge yet] Update to mono 2017-04 branch (#1960) 2017-05-29 18:39:29 +02:00
bindings-framework-test [tests] Since libtest.a need ModelIO now, the corresponding LinkWith attributes need to state that too. 2017-09-01 11:14:46 +02:00
bindings-test [tests] Since libtest.a need ModelIO now, the corresponding LinkWith attributes need to state that too. 2017-09-01 11:14:46 +02:00
bindings-test2 [tests] Add test for bug #43689. 2017-02-10 14:46:29 +01:00
common Add missing [MarshalDirective] attributes. (#2558) 2017-08-29 14:59:24 +02:00
dontlink-mac [tests] Don't create test packages by default. (#1941) 2017-03-30 18:01:15 +02:00
framework-test [tests] Fix framework-test to actually work. (#1297) 2016-12-05 18:35:53 +01:00
fsharp [test][fsharp] Add F# test for bug 52866 (#1892) 2017-03-22 07:47:57 -05:00
fsharplibrary [tests] Add support for building device tests for 32-bit or 64-bit. 2016-10-11 19:52:36 +02:00
generator [generator] Fix smart enum FieldAttribute LibraryName generation (#2376) 2017-07-26 18:09:42 -05:00
interdependent-binding-projects [tests] Change ProjectGuid of a few projects to make sure they're unique. (#1943) 2017-03-30 14:33:26 +02:00
introspection [AVFoundation] Set the wrongly added API to be obsolete and ensure it returns logical values. (#2650) 2017-09-13 08:21:59 -04:00
linker-ios Merge d15-3 (#2322) 2017-07-13 08:16:25 -04:00
mac-api-sharpie [tests] Make mdtool verbose on CI, and fix a few xbuild invocation to honor XBUILD_VERBOSITY. (#131) 2016-06-06 13:01:24 +02:00
mac-binding-project Framework tests were still binding non-linked Simple class which errors now (#2216) (#2236) 2017-06-21 16:41:33 +02:00
mini [tests] Add support for building device tests for 32-bit or 64-bit. 2016-10-11 19:52:36 +02:00
misc [tests] Import 2016-05-26 15:06:52 +02:00
mmptest [macos] Reenable static registrar since Xcode9b6 un-broke SDK headers (#2544) 2017-08-24 15:52:21 -05:00
monotouch-test Various Xcode 9 API improvements and also fixes bug 59433 (#2676) 2017-09-13 09:58:08 -05:00
msbuild-mac [tests] Fix msbuild-mac project to use a project reference to GuiUnit. (#1950) 2017-03-31 11:19:00 +02:00
mtouch Merge latest d15-3 fixes into xcode9 (#2660) 2017-09-12 13:08:55 -04:00
no-mmp [tests] Import 2016-05-26 15:06:52 +02:00
qa-regression [tests] Import 2016-05-26 15:06:52 +02:00
scripted [mtouch] Don't allow building for 32-bit when deployment target is >= 11. Fixes #57966. (#2303) 2017-07-11 08:54:50 +02:00
scriptingbridge [tests] Import 2016-05-26 15:06:52 +02:00
templates [tests] The container app for watchOS tests should be 64-bit. (#2463) 2017-08-10 07:26:13 +02:00
test-libraries [tests] Make sure test symbols don't clash between different test libraries. 2017-09-01 11:11:06 +02:00
xammac_tests Merge pull request #2571 from rolfbjarne/xcode9-matrix 2017-09-05 15:33:02 +02:00
xharness [tests] Add all test project variations to tests-mac.sln (#2579) 2017-09-11 23:10:45 +02:00
xtro-sharpie [xtro] The right platform for macOS is 'macos', not 'osx'. (#2677) 2017-09-13 08:33:35 -04:00
.gitignore [tests] Convert mmp tests to a standard NUnit test library. (#2067) 2017-05-08 13:20:01 +02:00
HtmlTransform.xslt [tests] Move HtmlTransform.xslt here from maccore. (#16) 2016-04-28 15:07:26 -04:00
Makefile [tests] Clean up after tests on wrench to minimize the peak diskspace requirements. (#2047) 2017-04-27 14:30:04 +02:00
README.md [tests] Update README. (#1207) 2016-11-17 14:18:39 +01:00
common.mk [tests] Import 2016-05-26 15:06:52 +02:00
package-mac-tests.sh [tests] Add scripts to install system dependencies to the XM tests package. (#847) 2016-09-19 10:02:10 -04:00
test-dependencies.sh [system-dependencies] Add logic to auto-provision cmake. (#940) 2016-10-04 10:28:41 +02:00
tests-mac.sln [tests] Add all test project variations to tests-mac.sln (#2579) 2017-09-11 23:10:45 +02:00
tests.sln [tests] Add support for building device tests for 32-bit or 64-bit. 2016-10-11 19:52:36 +02:00

README.md

Various tests to be executed prior to releases

Test solutions

Many of the test solutions and test projects are generated, and will only be available after running make once.

  • tests.sln: This is the base test solution for Xamarin.iOS, which targets iOS using the Unified API. Not generated.
  • tests-tvos.sln: All the TVOS test projects. Generated.
  • tests-watchos.sln: All the WatchOS test projects. Generated.
  • tests-mac.sln: This is the base test solution for Xamairn.Mac. Not generated.

Test solution/project generation

The tool that generates the test solutions / projects is called xharness, and lives in the xharness subdirectory.

Types of Tests

Unit Tests

Most of the projects are using NUnit[Lite] and looks like unit tests. They are meant to be executed on the target: simulator, devices, OSX.

In reality most of them are regression tests - but that does not change the need to execute and continually expand them to cover new code.

Introspection Tests

Introspection tests are executed on target (both simulator and device for iOS) or a specific version of OSX. The application proceed to analyze itself using:

  • System.Reflection for managed code; and
  • the ObjectiveC runtime library for native code

and compare the results. E.g. if using .NET reflection it can see a binding for a NSBundle type then it should be able to find a native NSBundle type using the ObjC runtime functions. Otherwise an error is raised...

Since the application analyze itself it must contains everything we wish to test. That's why the introspection tests are part of the dontlink.app application (for iOS) and the dontlink-mac project (for OSX).

Pros

  • They always tell the truth, which can differ from documentation

Cons

  • Incomplete - Not everything is encoded in the metadata / executable;
  • Too complete - Not every truth is good to be known (or published)

Extrospection Tests

Extrospection tests takes data from some outside sources and see if our implementation match the information, e.g.

  • Header files from the SDK;
  • Rules, like Gendarme or FxCop;

Since this is done externally there's no need to run them on the devices, simulator or even a specific version of OSX.

Pro

  • There is more data available, e.g. information lost when compiling

Con

  • The data might not represent the truth (errors, false positives...)

Test Suites

*-tests : where * is the assembly name, e.g. monotouch

Use the project defaults for linking, i.e.

  • "Don't link" for simulator

  • "Link SDK assemblies only" for devices

  • regression testing without using the linker

  • both simulator and devices are set to "Don't link"

linkall

  • regression testing using the linker on the all assemblies

  • "Link all assemblies" for both simulator/devices

linksdk

  • regression testing using the linker on the SDK assemblies

  • "Link SDK assemblies only" for both simulator/devices

bcl-test

These are the Mono BCL test suite tweaked to run on the mobile profile. It reuse the files directly from mono's repository (linking, not copying).

As other unit tests the configuration is set to mimick normal apps, e.g.

  • "Don't link" for simulator

  • "Link SDK assemblies only" for devices

Common make targets

Run every test in both the simulator and on device, using both the compat and the new profile (for the simulator both in 32 and 64bit mode).

$ make run

Run every test in the simulator, using both the compat and the new profile (both 32 and 64bit simulators).

$ make run-all-sim

Run every test on device, using both the compat and the new profile

$ make run-all-dev

Detailed make targets

  • Main target

    • run-what-where-project: Builds, installs (if applicable) and runs the specified test project on the specified platform. This is the most common target to use.
    • build-what-where-project: Will build the specified test project for the specified platform and target.
    • install-what-where-project: Will install the specified test project on a connected device. There's currently no way to select the device, so ensure you've only one connected (if many devices are connected, it's random which will used).
    • exec-what-where-project: Will run the specified test project in the simulator or on a device.
  • What

    • -ios-: iOS.
    • -tvos-: TVOS.
    • -watchos-: WatchOS

    If "What" is skipped, all variations are executed sequentially.

  • Where

    • -simclassic-: Simulator using the Classic API. Only applicable when platform is iOS.
    • -simunified-: Simulator using Unified API. The build will contain both an i386 and an x86_64 binary. Only applicable to the build-* target, while the -sim32- and -sim64- are only applicable to the exec-* targets. Only applicable when the platform is iOS.
    • -sim32-: 32bits iOS simulator using the Unified API. Not applicable to other platforms.
    • -sim64-: 64bits iOS simulator using the Unified API. Not applicable to other platforms.
    • -sim-:
      • iOS: Both the -simclassic- and -simunified- versions.
      • WatchOS/TVOS: The WatchOS/TVOS simulator.
    • -devclassic-: Device using the Classic API. Only applicable when the platform is iOS.
    • -devunified-: Device using the Unified API. The build will contain both an armv7 and an arm64 binary. It's not possible to select a 32/64bit version, you'll run what your device supports. Only applicable when the platform is iOS.
    • -dev-:
      • iOS: Both the -devclassic- and -devunified- versions.
      • WatchOS/TVOS: A Watch or TV device.
  • Examples

    $ make run-ios-sim32-monotouchtest: This will run monotouch-test using the Unified API in a 32-bit simulator. $ make run-tvos-dev-dont\ link: This will run dont link on an Apple TV device.

Utility run-* targets

These targets will build, install (if applicable) and run the specified project(s).

  • Simulator

    • run-sim-project: Builds and runs the specified test project in the simulator in compat, 32 and 64bit mode.
    • run-sim: Builds and runs all the non-bcl test projects in the simulator in compat, 32 and 64bit mode.
  • Device

    • run-dev-project: Builds and runs the specified non-bcl test project on a device in compat and native mode (if it's 32 and 64bit depends on the device; 64bit devices will run in 64bit mode and the same for 32bit devices).
    • run-devcompat: Run all the non-bcl test projects on device, in compat mode.
    • run-devdual: Run all the non-bcl test projects on device, in native mode (if it's 32 and 64bit depends on the device; 64bit devices will run in 64bit mode and the same for 32bit devices).
    • run-dev: Run all the non-bcl test projects on device, in both compat and native mode.