* [MMP] Revert recursive search dirs changes
* [MMP] Allow resolving assemblies to the ones passed in command line args
This is what actually makes the MonoMacResolver actually resolve to
assemblies given as arguments.
This mirrors the behaviour of Pack, which is called on the other
code-path (that's not --runregistrar)
3113c5d2b5/tools/mmp/driver.cs (L513)
* [static registrar] Optimize creation of delegates for blocks.
Optimize creation of delegates for blocks so that it doesn't require the
dynamic registrar.
This is done by getting the metadata token for the Create method that creates
the delegate, and embed that metadata token in the generated code from the
static registrar.
Also add tests, since this scenario was not covered by tests already.
* [mmptest] Fix test after recent changes.
* [test-libraries] Avoid duplicate symbols.
* [tests] Update according to changes.
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
* [tests] Fix relative paths in mmptest and make cloned XM nunit projects work properly.
Fixes this build problem with mmptest:
/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/mmptest/CustomBuildActions.targets: error : Command 'make bin/SimpleClassDylib.dylib bin/SimpleClassStatic.a bin/Mobile-static/MobileBinding.dll' exited with code: 255.
Fixes https://github.com/xamarin/maccore/issues/633.
* [mmp] Improve Driver.IsDefaultMarshalingManagedExceptionMode to work before MarshalManagedException has been initialized.
Fixes this test failure:
1) Failed : Xamarin.MMP.Tests.MMPTests.UnifiedDebugBuilds_ShouldLinkToPartialStatic_UnlessDisabled
Debug build should use partial static registrar
Expected: True
But was: False
This regressed in 0561618460, where we wouldn't
properly select the partial static registrar because the initialization order
changed and we'd do it before selecting the default exception marshaling mode.
Changing Driver.IsDefaultMarshalingManagedExceptionMode to work before
MarshalManagedException has been initialized to its default value fixes the
problem.
There's only one static registrar now, so there's no need for code to select
which one, just create the one and only.
Also unify this code between mtouch and mmp.
You were the preprocessor we wished C# had natively
Removing PMCS requires these changes:
* Remove XamCore from src/
* Remove XamCore from tools/
* Remove XamCore from runtime/
* nint/nuint enum conversion
* _compat_ enum conversion
* NSAction conversion
* Hand fix single API incorrectly converted by PMCS to unbreak compatibility
- Due to a bug in PMCS, the nuint was incorrectly converted in this API.
- However, as that ship as sailed, we must "fix" it until XAMCORE_4_0
* Update readme
* Bump macios-binaries
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
* [mtouch] Improve how we make sure native symbols aren't stripped away. Fixes#51710 and #54417.
* Refactor required symbol collection to store more information about each
symbol (field, function, Objective-C class), and in general make the code
more straight forward.
* Implement support for generating source code that references these symbols,
and do this whenever we can't ask the native linker to keep these symbols
(when using bitcode). Additionally make it possible to do this manually, so
that the source code can be generated for non-bitcode platforms too (which
is useful if the number of symbols is enormous, in which case we might
surpass the maximum command-line length).
* Also make it possible to completely ignore native symbols, or ignore them on
a per-symbol basis. This provides a fallback for users if we get something
right and we try to preserve something that shouldn't be preserved (for
instance if it doesn't exist), and the user ends up with unfixable linker
errors.
* Don't collect Objective-C classes unless they're in an assembly with
LinkWith attributes. We don't need to preserve Objective-C classes in any
other circumstances.
* Implement everything for both Xamarin.iOS and Xamarin.Mac, and share the
code between them.
* Remove previous workaround for bug #51710, since it's no longer needed.
* Add tests.
https://bugzilla.xamarin.com/show_bug.cgi?id=54417https://bugzilla.xamarin.com/show_bug.cgi?id=51710
* [mtouch] Make sure to only keep symbols from the current app when code sharing.
This fixes a build problem with the interdependent-binding-projects test when
testing in Today Extension mode.
Replace https://github.com/xamarin/xamarin-macios/pull/1973 expect that
the test parts are still needed.
* Add XM SDK + LinkSkip test
* [macos] Add platform linking support to msbuild
* [macos] Add full SDK test
* [macios] Diable classic from using linkplatform
- Extended test infrastructure change to allow classic projects that include bundling
- Setting linkplatform in MonoBundlingExtraArgs since we don't even read project setting LinkMode - Platform for classic
Don't use the global command line arguments to determine input, because that's
not the input we use for app extensions anymore.
Instead explicitly pass the input arguments when creating the cache.
Implement support for sharing both code and resources between app extensions
and their container app:
* AOT-compiled code. Each shared assembly is only AOT-compiled once, and if
the assembly is built to a framework or dynamic library, it will also only
be included once in the final app (as a framework or dynamic library in the
container app, referenced directly by the app extension). If the assemblies
are built to static objects there won't be any size improvements in the app,
but the build will be much faster, because the assemblies will only be AOT-
compiled once.
* Any resources related to managed assemblies (debug files, config files,
satellite assemblies) will be put in the container app only.
Since these improvements are significant, code sharing will be enabled by
default.
Test results
============
For an extreme test project with 7 extensions (embedded-frameworks)[1]:
with code sharing cycle 9 difference
build time 1m 47s 3m 33s -1m 46s = ~50% faster
app size 26 MB 131 MB -105 MB = ~80% smaller
For a more normal test project (MyTabbedApplication)[2] - this is a simple application with 1 extension:
with code sharing cycle 9 difference
build time 0m 44s 0m 48s -4s = ~ 8% faster
app size 23 MB 37 MB -15 MB = ~40% smaller
Another tvOS app with one extension also show similar gains (MyTVApp)[3]:
with code sharing cycle 9 difference
build time 0m 22s 0m 48s -26s = ~54% faster
app size 22 MB 62 MB -40 MB = ~65% smaller
[1]: https://github.com/rolfbjarne/embedded-frameworks
[2]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTabbedApplication
[3]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTVApp
Cecil cannot save an assembly more than once without reloading its
debugging symbols. There was code for this, in the special case where
an assembly could be saved more than once. However the order of the
operations changed and the code to reload symbols was now located too
late.
https://bugzilla.xamarin.com/show_bug.cgi?id=51667
So that there can be multiple caches in the same process (which we'll have
once mtouch can compile extensions and the container app in the same process).
* [mtouch] Remove most static state and put it on an instance.
Remove most static state, and put it on an instance (of Application) instead.
Soon we'll build multiple apps (app extensions) in the same process, and we
need to make sure we're not inadvertedly (and incorrectly) sharing state
between apps, so remove as much static state as possible to avoid any
problems.
* [mtouch] Rename GetIsUsingClang -> IsUsingClang.
Managed exception marshaling interferes with the debugger, because it adds
exception handlers to executing code, which makes the Mono runtime think an
exception is handled when logically it's not (although technically it is).
The consequence is that the IDEs will only be notified when we re-throw the
exception after catching it, making it impossible for the IDEs to stop when
the exception is thrown (they will instead stop when we re-throw the
exception).
So disable managed exception marshaling (unless the user changed the default
behavior) when a debugger is attached.
This is the same behavior as Xamarin.Android.
https://bugzilla.xamarin.com/show_bug.cgi?id=45116