Up to this commit the test dlls used for the bcl tests were from iOS. Since
the Mono SDK provides the dlls for both missing platforms (TvOS and
WatchOS) we can now use the correct path for the dlls.
There is a small trick to minimize the project generation, since there
is a simple stirng.Replace, the logic now checks the platform under
test and does:
* TvOS - Goes from monotouch_TEST_NAME to monotouch_tv_TEST_NAME
* WatchOS - Gores from monotouch_TEST_NAME to monotouch_watch_TEST_NAME
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
```
/Users/poupou/git/master/xamarin-macios/tools/linker/MonoTouch.Tuner/RemoveBitcodeIncompatibleCodeStep.cs(14,7): warning CS0105: The using directive for 'Xamarin.Linker' appeared previously in this namespace [/Users/poupou/git/master/xamarin-macios/tools/mtouch/mtouch.csproj]
```
This also centralize other interpreter checks and options in the same
location (making it easier to read / update).
* Warn and switch the REPL if the interpreter is enabled on simulator
Why ? It's confusing to build the same code using different options for
simulator and devices. This is what happens if you try to use features
like `dynamic` or `System.Reflection.Emit`.
So instead of an error, we warn that the interpreter is not supported
and switch to the existing REPL mode.
The JIT remains the only option for the simulator but it allows testing
features without a device.
* Fail early if the interpreter is used on 32bits [1]
The current interpreter only works on 64 bits (so ARM64). However the
error won't be reported, back to the developer, until deployment time.
This temporary [1] fix spot the condition very early and report an error
```
error MT0099 : Internal error : The interpreter is currently only available for 64 bits.
```
instead of the current one at deploy time
```
IncorrectArchitecture: Failed to find matching arch for 32-bit Mach-O input file /private/var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.tNKDlx/extracted/X.app/X
error MT1006: Could not install the application 'X.app' on the device 'Mercure': AMDeviceSecureInstallApplicationBundle returned: 0xe8000087 (kAMDIncorrectArchitectureError).
Application could not be uploaded to the device.
```
[1] https://github.com/mono/mono/issues/9871
* [tests] Fix/renumbered MT0138
The test was using simulator + interpreter which is not _really_
possible, we use REPL in that case - so we're now checking if
assemblies were specified with `--interpreter` to cover both cases.
Also 0138 was already used by `mmp` and the warning was **not**
registered or documented in the errors documents. To avoid
confusion it has been renumbered to 0142 and documented.
* [Foundation] Add an NSArray.FromNSObjects overload that can take an array of INativeObjects.
* [runtime] Use mono_array_setref instead of mono_array_set.
Otherwise the GC won't know about the assignment, and the assigned value can
be freed if it's no longer referenced anywhere else.
Some optimizations are not safe to apply when dynamically loading code
at runtime. This is not possible when ahead-of-time (AOT) compilation
is used, so those options were always enabled in the past. The
interpreter is changing the situation so those optimizations are now
disabled, when the interpreter is enabled.
The disabled optimizations are:
* 'remove-dynamic-registrar' when using the interpreter
This avoid the following exception when unknown (at build time) code
tries to register (at runtime)
```
Unhandled Exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> ObjCRuntime.RuntimeException: Can't register the class SubclassDemo.CustomView when the dynamic registrar has been linked away.
```
* 'inline-isdirectbinding' when using the interpreter
This avoid crashing (at runtime) when types are subclasses by the
interpreter (code unknown to `mtouch` at build time).
* 'register-protocols' when the interpreter is enabled
Dynamically loaded code can include protocols. When the optimization is
enabled then they won't be registered and won't work as expected.
Note: It is possible to re-enable the optimization(s), either one or all,
if the interpreter is not used to dynamically load code at runtime.
In both cases we use a different binaries for a few assemblies. They
include support for SRE and a few other things that are not normally
usable on iOS. That's totally fine and not something that can be fixed
(unless you stop using the feature). So this PR simply ignore that
specific case so we don't warn about things that can't be changed (and
are not a problem)
E.g.
```
Warning MT0109: The assembly 'mscorlib.dll' was loaded from a different path than the provided path (provided path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/Xamarin.iOS/mscorlib.dll, actual path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/repl/mscorlib.dll). (MT0109) (XXX)
Warning MT0109: The assembly 'System.Core.dll' was loaded from a different path than the provided path (provided path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/Xamarin.iOS/System.Core.dll, actual path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/repl/System.Core.dll). (MT0109) (XXX)
Warning MT0109: The assembly 'System.dll' was loaded from a different path than the provided path (provided path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/Xamarin.iOS/System.dll, actual path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/repl/System.dll). (MT0109) (XXX)
Warning MT0109: The assembly 'System.Xml.dll' was loaded from a different path than the provided path (provided path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/lib/mono/Xamarin.iOS/System.Xml.dll, actual path: /Users/poupou/git/intr/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/repl/System.Xml.dll). (MT0109) (XXX)
```
E.g. `MetalPerformanceShaders` is unusable in 10.3.1 on the simulator so
trying to build with the static registrar cause simlauncher to be rebuilt
with a **strong** (and not _weak_) reference to the framework. This cause
a crash then the application start
```
Dyld Error Message:
Library not loaded: /System/Library/Frameworks/MetalPerformanceShaders.framework/MetalPerformanceShaders
Referenced from: /Users/USER/Library/Developer/CoreSimulator/Devices/95679F95-CCE7-4733-8376-35E91E97039C/data/Containers/Bundle/Application/0F665B2D-ADAC-4CA0-BD04-33E968B3F268/FailerApp.app/FailerApp
Reason: no suitable image found. Did find:
/System/Library/Frameworks/MetalPerformanceShaders.framework/MetalPerformanceShaders: mach-o, but not built for iOS simulator
```
ref: https://github.com/xamarin/xamarin-macios/issues/5753
document it and adjust the optimization tests.
This allows the optimization to be:
* disabled (if ever needed) on fully AOT'ed applications
* re-enabled for the interpreter (at your own risk)
Fix comparison so the XM profiles do not report _inexistent_ changes for `_Attribute`, `_EventInfo` and `_Exception`. That noise makes it harder to review the diffs (for us) and is incorrect in our published API diffs (for release notes).
Also
* Delete .unzip.stamp on clean
* Skip classic XM diff (binaries won't change)
Reenable the tests and ensure that the failing ones on Modern are
ignored. The bcl tests on mac os x can now make a diff between the
version (Moder/Full) to ignore files so that tests that work in a
version are not ignored in the other one.
Fixes https://github.com/xamarin/maccore/issues/1200
* MachO.cs: Support reading LC_BUILD_VERSION
Newer SDKs set this instead of LC_VERSION_MIN_*
* MachO.cs: Add support for reading Mach-O files inside ar archives.
* [tests] Augment ProductTests.MinOSVersion to test static libraries as well.
* Adjust enum field names to match our naming scheme.
The removal of the XML files broke the comparison with the previous
commit. It did NOT fail the original PR because the targets were
in the previous commit.
And it will fail this PR because the previous commit still does not
have the targets - but it _should_ be fine, once merged, for all
future PR.
Fixes https://github.com/xamarin/maccore/issues/1452
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes#5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
Instead download the assemblies (part of existing bundle.zip) of the
current release and generate the XML locally (no storage in github)
before doing the diff against the newer, just built, assemblies.
reference: https://github.com/xamarin/xamarin-macios/issues/4891
* We only need to do a runtime check on watchOS.
* On watchOS assume we're on a ARM64-based CPU as long as we're not on a
ARMv7k CPU.
Fixes an issue where we failed to detect a 64-bit iOS CPU as ARM64-based (in
particular iPhone X's ARM64v8).
The Mono AOT compiler maintains a set of signatures of compiled methods.
Those signatures are necessary to emit wrappers to enable the transition
from interpreter->AOT code. Thus, they must be collected for each
assembly.
* Improve the behavior when we fail to download the .dvtdownloadableindex file
to report a "Not Found" if we get a "Forbidden" (403) response, since that's
what it really is.
* Improve the behavior when we fail to download the .dvtdownloadableindex file
to check if the requested simulators are already installed instead of always
failing. This makes it possible to manually install the required simulators
if needed (Apple might not have published simulators for the Xcode version
we're using), and this would allow any checks to pass if the required
simulators are already installed.
* Improve the code to modify the PackageInfo XML to not use simple string
replacement (it's too fragile), use proper XML parsing instead.
* Fix "time left" math, I have no idea what my previous math was doing.
Clang will automatically compile .bc files into object code if needed, but
it's done serially. If we do the compilation ourselves, it'll be parallelized.
This makes the dontlink tests build in half the time when building for watchOS
/ Release (it drops from ~15 minutes to ~7 minutes).
The AOT compiler uses the 'outfile' as the base for a temporary .bc file. If
it's not set, the path to the assembly is used as the base instead. This does
not work if we compile the same assembly to multiple architectures (for
instance armv7k+arm64_32), because the temporary file will clash between those
AOT compilations.
This is not a problem for iOS (armv7+armv7s) because we don't compile to
bitcode (.bc files) on iOS.
* [mtouch] Don't use the native linker to create fat executables.
Don't use the native linker to create fat executables, instead link each
architecture separately, and then manually lipo everything together at the
end. This requires a few changes since we need to keep track of the linker
flags per architecture.
The problem is that bitcode files (.bc) do not correspond with a particular
architecture, so the linker can't distinguish between .bc files for armv7k and
.bc files for arm64_32. So if we pass all together to the linker, the linker
will add all .bc files to both architectures, thus duplicating everything (and
the linking fails with duplicate symbols errors).
* [mtouch] Fix building symlinked simulator executables.
* [mtouch] Fix several assumptions about each Target only producing a single executable.
The current code did not consider that overrides could be swept later,
if unused/unmarked by the linker, so we were missing opportunities to
make methods as final.
Also change output to use the full path to files as nodes, instead of just the
filename, and instead use a label to set what's shown to just the filename.
This makes the graph correct when we have multiple files with the same name,
but different paths.
We need our 32-bit and 64-bit assemblies to be identical so that we can avoid
duplicating the .dll in fat apps.
One difference used to be that the .dll contained the full path to the
corresponding .pdb ([1]), but we changed cecil to only write the filename
([2]). Unfortunately this change breaks something else, so it has to be
reverted ([3]).
This implements a different solution: we provide a custom symbol writer to
Cecil, which only writes the filename of the pdb in the .dll, not the full
path.
[1]: https://bugzilla.xamarin.com/show_bug.cgi?id=54578
[2]: https://github.com/jbevain/cecil/issues/372
[3]: https://github.com/jbevain/cecil/pull/554
(cherry picked from commit 53874c863996656eaba43a5582731b93eb6f53b7)
# Conflicts:
# tools/mtouch/mtouch.csproj
Fixes this warning:
tools/bcl-test-importer/BCLTestImporter/BCLTestProjectGenerator.cs(19,17): warning CS0414: The field 'BCLTestProjectGenerator.iOSReleasePattern' is assigned but its value is never used
* Add a Runtime.IsARM64CallingConvention property.
Determining whether we should use the ARM64 calling convention in P/Invokes
gets more complicated with ARM64_32 (which for our purposes is a 32-bit
architecture).
So add a property on the Runtime class to avoid code duplication, and teach
the linker to optimize any calls to this property to a constant value whenever
possible (and the method is marked as optimizable).
Also change our code to use this new property, and make the corresponding
methods optimizable.
Some shuffling in mmp was required, which meant a little bit more code is now
shared between mtouch and mmp.
* Coding style.
* Test tweaks.
* Improve comment.
* Document new optimization
* Move ILReader to shared linker test code location.
* Disable inlining on armv7k.
* Change IsARM64CallingConvention to a read-only field.
We can give the AOT compiler a constant value for a read-only field, so that
the AOT compiler optimizes away the call to the field by using the constant
value.
This commit does not implement this change for the AOT compiler, but using a
read-only field makes it easy to implement it in the future.
* [linker] Remove non-bitcode compatible code, and show a warning.
Remove code not currently compatible with bitcode and replace it with an
exception instead (otherwise we'll assert at runtime).
Also show a warning when we detect this.
This is quite helpful when looking at watch device test runs to filter out
failures we already know about.
This fixes point #2 in #4763.
* Improve documentation.
* Simplify linker code by using a substep.
* Fix whitespace issues.
* Improve reporting.
* Add support for reporting more than one MT2105 at the same time when making
the errors instead of warnings.
* Only report MT2105 for methods that haven't been linked away.
* Format the error message nicer for properties.
* Tweak a bit for warning tests to pass.
* Use ExceptionalSubStep to provide better error information.
* Adjust where linker warnings/errors are reported from to avoid a NullReferenceException.
* Make PreserveSmartEnumConversionsSubStep use error codes that are not
already used elsewhere: this isn't obvious at first, but all
ExceptionalSubStep errors use a range of error numbers (10 numbers from NNN0
to NNN9), so we need to reserve that entire range. In this case there were
other errors using some of the numbers of the range for PreserveSmartEnumConversionsSubStep.
* Make sure there are anchor links for all the variations of each ExceptionalSubStep error.
Rework BCL test importation to generate projects that won't compile instead of
giving up completely in case of generation failure.
This means that xharness can still be used for non-BCL tests even if the
generation of the BCL tests fail.
This is usually not a problem, because these variables are already set when
running from the command-line or xharness. However, it shows up when running
tests directly from VSfM.
Sometimes we just Log a string without any format arguments. This works fine,
until the string comes from the user, and happen to contain braces, in which
case an invalid format exception is thrown.
Adding Log overloads that doesn't take format arguments (nor formats its
input) avoids this problem.
Alexander Köplinger
Manuel de la Pena
Sebastien Pouliot
Rolf Bjarne Kvinge
Bernhard Urban