A fix (thanks Rolf!) in how we do SDK version checks is needed
because for some reason `new Version (3, 2, 0)` isn't the same to
`new Version (3, 2)` and we end up with a MT4134 when building the
watchOS static registrar.
This cuts down another group of conditional compilation sections, paving the
way for an IKVM-based generator.
This makes it required to pass --target-framework for to generator executables
(previously only required for Xamarin.Mac/Unified to distinguish between the
different Xamarin.Mac/Unified variants), but it should be invisible to users
since we'll automatically pass the correct --target-framework argument from
the corresponding scripts (btouch/btv/bwatch/bmac) and the MSBuild targets.
This will only break somebody who is executing the managed executables
directly, but nobody should do that in the first place (it's not a supported
scenario).
Generated diff: https://gist.github.com/rolfbjarne/1674be6625632446dba774a305951981
Besides the obvious reasons, this is also useful when testing warnings, since
by making warnings errors, mtouch/mmp will exit a lot faster (and the tests
will finish faster).
Change cache invalidation so that if any app extension's cache is invalid,
then invalidate the cache for the container app and all other app extensions.
This is the safest option when we're sharing code.
Don't use the global command line arguments to determine input, because that's
not the input we use for app extensions anymore.
Instead explicitly pass the input arguments when creating the cache.
Since neither mtouch nor mmmp is mkbundled anymore, the installed binary is in
fact a shell script.
This means that it's quite useless to check if the shell script has been
modified; instead check if the executing assembly has been modified (which
works now that we're not mkbundled anymore).
Implement support for sharing both code and resources between app extensions
and their container app:
* AOT-compiled code. Each shared assembly is only AOT-compiled once, and if
the assembly is built to a framework or dynamic library, it will also only
be included once in the final app (as a framework or dynamic library in the
container app, referenced directly by the app extension). If the assemblies
are built to static objects there won't be any size improvements in the app,
but the build will be much faster, because the assemblies will only be AOT-
compiled once.
* Any resources related to managed assemblies (debug files, config files,
satellite assemblies) will be put in the container app only.
Since these improvements are significant, code sharing will be enabled by
default.
Test results
============
For an extreme test project with 7 extensions (embedded-frameworks)[1]:
with code sharing cycle 9 difference
build time 1m 47s 3m 33s -1m 46s = ~50% faster
app size 26 MB 131 MB -105 MB = ~80% smaller
For a more normal test project (MyTabbedApplication)[2] - this is a simple application with 1 extension:
with code sharing cycle 9 difference
build time 0m 44s 0m 48s -4s = ~ 8% faster
app size 23 MB 37 MB -15 MB = ~40% smaller
Another tvOS app with one extension also show similar gains (MyTVApp)[3]:
with code sharing cycle 9 difference
build time 0m 22s 0m 48s -26s = ~54% faster
app size 22 MB 62 MB -40 MB = ~65% smaller
[1]: https://github.com/rolfbjarne/embedded-frameworks
[2]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTabbedApplication
[3]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTVApp
Warn if mtouch loads an assembly from a different location than requested
(which might be because there are multiple assemblies with the same name).
Also rework the MT0023 check a bit by explicitly loading the root assembly
first, and then detecting if any loaded assemblies matches the root assembly.
This results in code that's a bit more obvious, and it also works correctly
with extensions (previously the entire MT0023 check was skipped for
extensions).
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
This makes dylibs automatically have the correct dylib id, which means no
fixups are required.
For instance: we'd build libpinvokes.armv7.dylib from libpinvokes.armv7.m,
which by default ends up with a dylib id of "libpinvokes.armv7.dylib". With
this fix no change is required, since we now build armv7/libpinvokes.dylib
from armv7/libpinvokes.m.
Create a custom AssemblyCollection class that contains a dictionary with
assembly identity (name) -> Assembly mapping.
This also means that we can detect if we end up loading multiple assemblies
with the same identity, and show an error in that case (even if that case
should never happen since we cache assemblies based on the identity, it's nice
to have code that ensures it).
Cecil cannot save an assembly more than once without reloading its
debugging symbols. There was code for this, in the special case where
an assembly could be saved more than once. However the order of the
operations changed and the code to reload symbols was now located too
late.
https://bugzilla.xamarin.com/show_bug.cgi?id=51667
Resource files for mac should be copied to the app bundle
* Move the Satellite code used by mac to tools/common/Assembly.cs
* Add EmbeddedResources test to xammac_tests
Most projects building to bitcode (any kind of bitcode, this includes the
marker-only version as well), will fail to link when linking with third-party
libraries and incremental builds are enabled.
So automatically disable incremental builds when we detect this scenario.
This is only a workaround until we can make this scenario build correctly.
https://bugzilla.xamarin.com/show_bug.cgi?id=51710
Use @rpath instead of @executable_path in dylibs, since it allows us to be
more flexible when placing dylibs in the app.
In particular with this change it's trivial to put libmonosgen-2.0.dylib in
the container app, and reference it from extensions.
Event sequence:
* mtouch is executed with the linker disabled.
* The linker pipeline copies all input assemblies (since the linker is
disabled the assemblies don't change) into the PreBuild directory. This will
keep the original timestamps of the input assemblies.
* mtouch is executed again, when none of the input assemblies changed.
* The linker pipeline will re-execute, because it will see that at least one
of the input assemblies (at least the .exe) is newer than at least one of
the assemblies in the PreBuild directory (usually a framework assembly,
because those have the original timestamp from their install location).
Fix:
Touch all the assemblies in the PreBuild directory after the linker pipeline
executes the first time. This way the second time mtouch is executed, it will
find that all assemblies in the PreBuild directory have timestamps later than
all the input assemblies, so it will load the cached linked assemblies,
instead of re-executing the linker pipeline.
The native linker treats object files (.o) and static libraries (.a files,
which are archives of .o files) differently.
The native linker will always include object files into the executable:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ld foo.o -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
0000000000000fe0 T _xxx
However, if the object file is inside a static library:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ar cru foo.a foo.o
$ ld foo.a -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
<no output>
This means that our testing library (libtest.a) which is a fat library of
_object files_, do not show the problems reported in bug #51548.
So:
a) I've fixed the creation of libtest.a to be a fat library of _static
libraries_. This causes the `FastDev_LinkWithTest` test to fail exactly
like in bug #51548.
b) I've made mtouch pass `-u <native symbol>` to the native linker, for every
native symbol referenced in a managed assembly, when creating a dylib.
Amazingly this seems to work fine even with symbols to Objective-C classes
(`_OBJC_CLASS_$_<class name>`).
c) This also required adding support for collecting the Objective-C names of
all managed types registered with Objective-C to the linker. The
information is already available in the static registrar, but that would
require us to make sure the static registrar is executed before compiling
dylibs, which means those two tasks won't be able to run in parallel (also
there's no guarantee we'll even run the static registrar).
https://bugzilla.xamarin.com/show_bug.cgi?id=51548
Alex Soto
Rolf Bjarne Kvinge
Sebastien Pouliot
Timothy Risi
Vlad Brezae