* [runtime] Clean up public symbols. Fixes#5124.
Clean up public symbols, by:
* Symbols that don't need to be public (most of them), can be private.
* Prefix all public symbols with `xamarin_`.
* Add a test to ensure we don't introduce new public symbols.
* Use C symbols instead of mangled C++ symbols, since those are easier to
handle in the test.
This minimizes the chance of getting into a symbol clash with another native library.
Fixes https://github.com/xamarin/xamarin-macios/issues/5124.
* Some test fixes.
watchOS extensions are top-level containers in our build (because we ignore
watchOS apps entirely in mtouch), so treat them as such when computing the
protocol member map.
Fixes issue https://github.com/xamarin/xamarin-macios/issues/3930.
* [linker] Optimize calls to BlockLiteral.SetupBlock to inject the block signature.
Optimize calls to BlockLiteral.SetupBlock[Unsafe] to calculate the block
signature at build time, and inject it into the call site.
This makes block invocations 10-15x faster (I've added tests that asserts at
least an 8x increase).
It's also required in order to be able to remove the dynamic registrar code in
the future (since calculating the block signature at runtime requires the
dynamic registrar).
* [mtouch/mmp] Add support for reporting errors/warnings that point to the code line causing the error/warning.
Add support for reporting errors/warnings that point to the code line causing
the error/warning by adding ErrorHelper overloads that take the exact
instruction to report (previously we defaulted to the first line/instruction
in a method).
* [tests] Add support for asserting filename/linenumber in warning messages.
* Make all methods that manually create BlockLiterals optimizable.
* [tests] Create a BaseOptimizeGeneratedCodeTest test that's included in both XI's and XM's link all test.
* [tests] Add link all test (for both XI and XM) to test the BlockLiteral.SetupBlock optimization.
* [tests] Add mtouch/mmp tests for the BlockLiteral.SetupBlock optimization.
* [tests][linker] Make the base test class abstract, so tests in the base class aren't executed twice.
* [tests][linker] Don't execute linkall-only tests in linksdk.
The optimization tests only apply when the test assembly is linked, and that
only happens in linkall, so exclude those tests in linksdk.
* [tests][mmptest] Update test according to mmp changes.
Fixes these test failures:
1) Failed : Xamarin.MMP.Tests.MMPTests.MM0132("inline-runtime-arch")
The warning 'MM0132: Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.' was not found in the output:
Message #1 did not match:
actual: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
Message #2 did not match:
actual: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
2) Failed : Xamarin.MMP.Tests.MMPTests.MM0132("foo")
The warning 'MM0132: Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.' was not found in the output:
Message #1 did not match:
actual: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
Message #2 did not match:
actual: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
* [tests][linker] Fix typo.
Fixes this test failure:
1) SetupBlock_CustomDelegate (Linker.Shared.BaseOptimizeGeneratedCodeTest.SetupBlock_CustomDelegate)
Counter
Expected: 1
But was: 2
* [registrar] Minor adjustment to error message to match previous (and better) behavior.
Fixes this test failure:
1) Failed : Xamarin.Registrar.GenericType_WithInvalidParameterTypes
The error 'MT4136: The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<U>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)'' was not found in the output:
Message #1 did not match:
actual: 'The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<Foundation.NSObject>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)''
expected: 'The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<U>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)''
* [docs] mmp shows MM errors/warnings.
* [docs] Improve according to reviews.
* [tests] Fix merge failure causing test duplication.
* [tests] Share supporting code between the mtouch and mmp tests.
Create a new class 'BundlerTool', which now contains most of the code in
MTouchTool that's also applicable to mmp (and the new MmpTool class).
This will make it easier to share tests between the mtouch and mmp tests.
Some tweaks are still probably required, but this should get us most of the
way.
* [tests] Fix generator tests after changes in shared test code.
* [tests] Add new file to the MSBuild/XM tests.
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
* [mtouch][mmp] Report invalid debug symbols files. Fixes#3200
Try to read the assembly with symbols and, if that fails, warn and
fallback to loading them without symbols.
This fixes cases were it's not easy to update or delete (e.g. nuget)
bad symbols files - so this cannot be an error without causing a lot
of pain.
However it needs to be reported, otherwise it wont be fixed (by the
publisher) and it can limit the debugability of the application and
the usefulness of the stacktraces.
Finally merge most of the resolver's code between mtouch and mmp so
we don't have to fix such issue twice anymore.
note: this needs to be slightly updated once we get a version of cecil
that can give us a more precise error message.
Also bring Rolf's tests from
https://github.com/xamarin/xamarin-macios/pull/3079
reference:
https://github.com/xamarin/xamarin-macios/issues/3200
* [mtouch/mmp/bgen] Add support for response files.
This is the first part of the fix for #56501.
https://bugzilla.xamarin.com/show_bug.cgi?id=56501
* [tests] Make sure no single argument starting with a '@' is passed to mtouch unless it's a response file.
--assembly-build-target takes arguments starting with '@', for instance:
--assembly-build-target @all=framework
which does not work anymore, because that's interpreted as a response file
(mtouch tries to read the file '@all=framework', which obviously doesn't
exist).
The fix is simple, don't put a space between the two arguments:
--assembly-build-target=@all=framework
* Add --root-assembly to mtouch/mmp and make the MSBuild tasks use this new option.
This makes it possible to pass root assemblies starting with `@` to mtouch/mmp
without getting mistaken for response files.
* [msbuild] Always use the command-line option that takes an equals or colon.
Always use the command-line option that takes an equals or colon instead of a
space.
Do either of these:
--foo=something
--foo:something
instead of this:
--foo something
so that `something` can start with an at (`@`) sign without being mistaken for
a response file.
* [msbuild] Fix tests according to recent task changes.
* [tests][mtouch] Convert MT4162 to new syntax.
This also means adding support for custom usings in test code, and adding
support for asserting filename/linenumber with error patterns, and custom
pattern syntax.
* [tests][mtouch] Add test case for bug #59617.
This amounts to running the MT4162 test with the linker enabled.
https://bugzilla.xamarin.com/show_bug.cgi?id=59617
* [msbuild] Re-added wildcard (*) expandsion for application-identifier in Entitlements.plist (#2186)
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=57119
* Bump mono (#2213)
* Framework tests were still binding non-linked Simple class which errors now (#2216) (#2218)
- Improve Makefile to rebuild when projects build with errors
* Bump mono to get cecil fix for bug #56808. (#2222)
https://bugzilla.xamarin.com/show_bug.cgi?id=56808
* [msbuild] Use @(ReferencePath) instead of @(ResolvedFiles) (#2188) (#2214)
This allows things to work on both xbuild and msbuild.
In xbuild, both lists are exactly the same and on msbuild,
only @(ReferencePath) exists.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=55147
* NSActivityOptions.IdleDisplaySleepDisabled had wrong value (#2232) (#2239)
This was due to an integer overflow. The original value was based on Int32
1 << 40 == 256
The correct value should be based on a UInt64.
1UL << 40 == 1099511627776
* [tests] Fix bug 57699 - [iOS]InternalsTest failure (Linkall) tests on device (#2243)
Strip native debugging symbols should not be checked for debug builds
* Bump mono to get fix for bug #57780.
https://bugzilla.xamarin.com/show_bug.cgi?id=57780
* Update .gitmodules
Change branch to d15-3 branch of mono
* Bump maccore to get fix for bug #55064.
https://bugzilla.xamarin.com/show_bug.cgi?id=55064
* [mono] Bump mono to get the head of cecil/mono-2017-04 and fix IsComObject #57919
Also fix#58789 [1], the typo in tools/mtouch/Tuning.cs showing in MT0000 errors
instead MT2102. That's already in master and d15-4
mono bump includes:
[2] commit 2a6502cee0df9de5198eafe7c8b5f6ac25106f34 (HEAD -> d15-3, origin/d15-3)
Merge: 02457c20fcf 5e05cafc6f1
Author: Luis Aguilera <luis.aguilera@xamarin.com>
Date: Fri Aug 18 10:04:06 2017 -0400
Merge pull request #5401 from marek-safar/com
[Marshal.IsComObject] Make this predicate return false instead of thr…
[3] commit 02457c20fcf57c0610e844d638eb1da82b5d1eb0
Merge: da80840ea55 73fd9a1b82e
Author: Luis Aguilera <luis.aguilera@xamarin.com>
Date: Fri Aug 18 09:59:06 2017 -0400
Merge pull request #5400 from spouliot/bump-cecil-58834-d15-3
[cecil] Bump to the head of the mono-2017-04 branch and pick the fix for bug #58834
References
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=58789
[2] https://bugzilla.xamarin.com/show_bug.cgi?id=57919
[3] https://bugzilla.xamarin.com/show_bug.cgi?id=58834
* [mtouch] Put 'mono_profiler_startup_log' in the symbol list. Fixes#58778. (#2501)
We need the 'mono_profiler_startup_log' symbol when profiling is enabled, so
make sure to add the symbol to the correct list of symbols we need.
Previously we were passing `-u _mono_profiler_startup_log` to clang directly,
which is fine, but not complete, since it does not write the symbol to the
symbollist file (--symbollist=file), which means it wouldn't be preserved when
the MSBuild tasks strip the executable.
https://bugzilla.xamarin.com/show_bug.cgi?id=58778
* Bump versions for SR3
https://trello.com/c/EVze08ei
* Bump mono to include HttpClientHandler fix#44027https://trello.com/c/jYFXadH8/8-systemnethttp-close-request-stream-when-httpclienthandlerhttps://bugzilla.xamarin.com/show_bug.cgi?id=44027
We need the 'mono_profiler_startup_log' symbol when profiling is enabled, so
make sure to add the symbol to the correct list of symbols we need.
Previously we were passing `-u _mono_profiler_startup_log` to clang directly,
which is fine, but not complete, since it does not write the symbol to the
symbollist file (--symbollist=file), which means it wouldn't be preserved when
the MSBuild tasks strip the executable.
https://bugzilla.xamarin.com/show_bug.cgi?id=58778
* [mtouch] Don't allow building for 32-bit when deployment target is >= 11. Fixes#57966.
Also bump maccore to get an mlaunch error for launching a 32-bit app in a 64-bit-only simulator.
https://bugzilla.xamarin.com/show_bug.cgi?id=57966
* [tests][msbuild] Make sure all Info.plists have deployment targets.
Otherwise we get different behavior (32-bit allowed or not) depending on which
Xcode is used to build.
* [mtouch] Default to 64-bit arch if not specified and targeting iOS 11+.
* [tests] Tweak tests to either specify a deployment target < 11 or not build a 32-bit arch.
When we process P/Invokes to add support for exception marshaling, we may
change P/Invokes to be __Internal. This means that we need to move the check
for __Internal P/Invokes to after processing P/Invokes for exception
marshaling.
https://bugzilla.xamarin.com/show_bug.cgi?id=57833
* [mtouch] Improve how we make sure native symbols aren't stripped away. Fixes#51710 and #54417.
* Refactor required symbol collection to store more information about each
symbol (field, function, Objective-C class), and in general make the code
more straight forward.
* Implement support for generating source code that references these symbols,
and do this whenever we can't ask the native linker to keep these symbols
(when using bitcode). Additionally make it possible to do this manually, so
that the source code can be generated for non-bitcode platforms too (which
is useful if the number of symbols is enormous, in which case we might
surpass the maximum command-line length).
* Also make it possible to completely ignore native symbols, or ignore them on
a per-symbol basis. This provides a fallback for users if we get something
right and we try to preserve something that shouldn't be preserved (for
instance if it doesn't exist), and the user ends up with unfixable linker
errors.
* Don't collect Objective-C classes unless they're in an assembly with
LinkWith attributes. We don't need to preserve Objective-C classes in any
other circumstances.
* Implement everything for both Xamarin.iOS and Xamarin.Mac, and share the
code between them.
* Remove previous workaround for bug #51710, since it's no longer needed.
* Add tests.
https://bugzilla.xamarin.com/show_bug.cgi?id=54417https://bugzilla.xamarin.com/show_bug.cgi?id=51710
* [mtouch] Make sure to only keep symbols from the current app when code sharing.
This fixes a build problem with the interdependent-binding-projects test when
testing in Today Extension mode.
If both an extension and the container app (or multiple extensions) reference
the same binding assembly for a framework, then we'd error out with an
internal error when trying to copy the framework from both locations to the
container app.
So instead detect when we're trying to copy multiple identical (by comparing
the on-disk contents) frameworks, and only copy one of them.
We'll still show an error if the frameworks are different, but now a nice
MT1035 error with a proper error description instead of an internal error.
https://bugzilla.xamarin.com/show_bug.cgi?id=56635
* [mmp/mtouch/ObjCRuntime] Calculate the path to the runtime-options.plist using NSBundle's ResourcePath.
The anatomy of apps and frameworks differ between iOS and macOS:
* iOS: we put runtime-options.plist in the root directory of the app.
* macOS:
* for apps we put runtime-options in foo.app/Contents/Resources
* for frameworks we put runtime-options in foo.framework/Versions/Current/A/Resources
Luckily NSBundle's ResourcePath property returns exactly this path, so change
our logic to use this property.
Also calculate the NSBundle using an exported type we know we have (using the
main bundle won't work when we're a framework).
* [tests] Add mmp/mtouch tests to verify the default HttpClientHandler according to build arguments.
* [ObjCRuntime] Use a custom class for finding the bundle.
Use a self-defined custom class to find the bundle where our resources are.
Using the internal NSObject.NSObject_Disposer class doesn't work when this
file (RuntimeOptions.cs) is compiled into System.Net.Http.dll.
Of particular importance is if we're building for LLVM or not: this fixes a
bug where we wouldn't pass --llvm to the AOT compiler when compiling
assemblies to frameworks (which we do when sharing code).
https://bugzilla.xamarin.com/show_bug.cgi?id=55555
Allow creating temporary apps/extensions multiple times without re-creating temporary directories.
This makes it possible for tests to call CreateTemporaryApp|Extension multiple
times with different code, and have only the actual managed assembly change
between each time (which is useful for tests that verifies that cached builds
are used properly).
Warn if mtouch loads an assembly from a different location than requested
(which might be because there are multiple assemblies with the same name).
Also rework the MT0023 check a bit by explicitly loading the root assembly
first, and then detecting if any loaded assemblies matches the root assembly.
This results in code that's a bit more obvious, and it also works correctly
with extensions (previously the entire MT0023 check was skipped for
extensions).
Create a custom AssemblyCollection class that contains a dictionary with
assembly identity (name) -> Assembly mapping.
This also means that we can detect if we end up loading multiple assemblies
with the same identity, and show an error in that case (even if that case
should never happen since we cache assemblies based on the identity, it's nice
to have code that ensures it).
Performance tests
-----------------
This is for a new watchOS extension project, built for release.
* The default (currently -O2) optimizations: 41s ( baseline ) 30.027.060 bytes ( baseline )
* All optimizations disabled (`--llvm-opt=all=`): 17s (-24s = -59%) 32.978.312 bytes (+2.951.252 = +10%)
* Optimized for size (`--llvm-opt=all=-Os`): 36s ( -5s = -12%) 28.617.408 bytes (-1.409.652 = -5%)
* Optimized for more size (`--llvm-opt=all=-Oz`): 35s ( -6s = -15%) 28.601.016 bytes (-1.426.044 = -5%)
* Optimized slightly (`--llvm-opt=all=-O1`): 35s ( -6s = -15%) 28.666.556 bytes (-1.360.504 = -5%)
* Optimized a lot (`--llvm-opt=all=-O3`): 41s ( 0s = 0%) 30.403.996 bytes (+ 376.936 = +1%)
Conclusions
-----------
* The fastest build by far (less than twice as fast) is if optimizations are
disabled, but this adds a 10% size penalty (~3 MB in this test case),
compared to the baseline, and 15% size penalty (4.3 MB) compared to -Oz.
* -Oz seems to have the best overall results: at least as fast as any other
optimized build, and the smallest app as well.
Caveats
-------
Some optimizations might not work the AOT compiled code. The resulting
binaries have not been tested.
The BundleId property is used by the code that generates the mSYM directory,
but its value was always the default value 'com.yourcompany.sample' instead of
looked up in the app's Info.plist.
So fix the BundleId property to do the expected.
Also fix the mSYM test (SymbolicationData) to actually test mSYM stuff (it was
partially disabled when we disabled automatic mSYM generation for C8, and
never re-enabled), and port it to the new and better test syntax, and add a
few more asserts to check the manifest.xml generation.
Earlier versions of Xamarin Studio stored an invalid http message handler in
watchOS project files, which would cause a build error. In addition Xamarin
Studio removed the UI to set the http message handler (since only one value is
valid), which meant that the user had to edit the project file by hand to get
around this build error.
So make it a warning instead (and document what the user has to do to fix the
warning).
https://bugzilla.xamarin.com/show_bug.cgi?id=46552
The container app may not reference the same third-party frameworks as
extensions, which means that we must make sure the extension's frameworks are
also included in the app bundle.
So when building extensions save a list of all third-party frameworks, and
then read that list and include those frameworks when building the main app.
https://bugzilla.xamarin.com/show_bug.cgi?id=45800
- BuildAndLaunchTime is replacing AppLaunchTime as it calculates both
build time and launch time for different linker modes.
* [mtouch/tests] Add RegistrarTime test
- Also fix MTouchTool --registrar:dynamic
Rolf Bjarne Kvinge
Sebastien Pouliot
Sebastien Pouliot
Timothy Risi
Vincent Dondain