* [mtouch] Only iOS has version 8 (and earlier OS versions that don't support frameworks).
* [mtouch] Update logging to be less confusing when logging about WatchKit extensions.
* [mtouch] Don't compile P/Invoke wrappers for extensions that are sharing code.
The container app already has the P/Invoke wrappers.
* Use Visual Studio instead of Xamarin Studio.
* VS doesn't have mdtool, it has vstool.
Also there's no need to manually invoke the mdtool.exe executable anymore
(which we did because the mdtool executable had a min macOS version of 10.9,
and we used to build tests on older macOS versions [1]), since now we only run
tests on older macOS versions, we don't build those tests there.
[1] a1932b0ccd
We want to copy the aot data for both the 32-bit and the 64-bit versions of an
assembly even if the 32-bit and 64-bit versions of the assembly are identical.
https://bugzilla.xamarin.com/show_bug.cgi?id=54499
- Before this mmp was not adding -framework, -weak_framework consistently on non-static registrar use cases
- GatherFrameworks was previously not ported from mtouch, and did not work as DeploymentTarget was unset in mmp
- Added verbose prints so users can determine why various framework linkages are added
- Fixed an issue where duplicate were being added due to HandleFramework shoving args by hand
- Tested with auto test and https://github.com/chamons/xm-version-regression-test manual test
Previously we copied any equivalent .dylib and ran install_name_tool on the
library to change the library id to make it a framework.
Unfortunately this does not work when the library contains bitcode, because
bitcode embeds linker flags (-install_name for instance), and
install_name_tool does not change those linker flags.
This means that we need to create frameworks by linking with the proper
arguments, since it's much more difficult to fixup the embedded bitcode linker
flags as well.
So change how be build Mono.framework, Xamarin.framework, and any frameworks
built from assemblies to:
* Always link instead of fixup a dylib. For Mono.framework this means
extracting all the object files from libmonosgen-2.0.a and linking those,
for Xamarin.framework this means linking the object files we've already
built.
* Make sure the library is correctly named when linked (once again: bitcode
contains embedded linker flags, so renaming the executable later breaks
stuff as well).
I've also extracted the logic that creates Mono.framework from
libmonosgen-2.0.a to a separate shell script, to deduplicate this logic.
This required a minor change in the mono builds: we need the Mono.framework
when building the `all` target, so make sure that happens.
https://bugzilla.xamarin.com/show_bug.cgi?id=53813
* [builds] Improve mono/llvm dependencies.
* Create a list of all the files in the mono and llvm repositories, and save
these lists as a Make variable (in a generated Makefile - .deps.*.mk). We
don't list _all_ the files in each repository, because there are quite a few
(55k for mono), and Make measurably takes a while to check all of them, so
try to limit it to a sane subset, without risking missing changes to files
that actually matters.
* Always create stamp files when we're done with mono builds.
* Modify the mono/llvm builds to depend on all the files in their
repositories.
* Explicitly list the corresponding .stamp-build-* files as dependencies for
various files that are produced by the mono builds, so that make knows how
to build these files.
* Rewrite the *-facade-check targets to depend on the corresponding
*_BCL_TARGETS, so that we can avoid running a submake to the same Makefile
to execute the facade checks.
It now takes a little while (less than a second on my machine, which is
fine) for make to list all dependencies and get their timestamps, but if
executing multiple submakes this adds up to a multi-second timewaste.
So avoid the timewaste by not doing submakes, but instead use dependencies
to enforce the required target execution ordering.
* Don't depend on nicely named intermediate targets, since won't prevent
rebuilds:
build-cross64: setup-cross64
Since the `setup-cross64` file doesn't exist, `build-cross64` will always
execute. Instead depend on the stamp file:
build-cross64: .stamp-configure-cross64
And now `build-cross64` will only rebuild if needed.
* Don't try to list all intermediate files as .SECONDARY dependencies, instead
list none at all, which works as if all files were listed as dependencies.
* Some targets had to move later in the file, since variables used in dependencies:
foo: $(VARIABLE)
must be defined before that point in the file, as opposed to variables used in recipes:
foo:
$(MAKE) $(VARIABLE)
can be defined anywhere in the Makefile.
* Simplify the targets that sign assemblies significantly.
There are a few end results:
* It's now possible to do `make install`, without doing `make all` first. This
might seem weird, but that also ensures the more common `make all install`
works properly.
* Remakes (without any mono/llvm changes) in build/ are much faster, because
we now won't recurse into every mono build:
$ time make all -C builds/ -j8
[...]
real 0m1.873s
This even means that we might be able to make it a habit to remake in the
root directory, which doesn't take forever now:
$ time make all -j8
[...]
real 0m4.521s
Unfortunately adding `make install` to the mix still does some useless
stuff, and it ends up taking ~30 seconds to complete a full build:
$ time make all install -j8
[...]
real 0m32.542s
* [msbuild] Don't verify the xml syntax of targets files unless the files change.
* [build] Don't depend on installed files.
Don't depend on installed files, because that causes a rebuild when installing
to a different directory (i.e. package creation).
* Bump maccore to get build improvements.
Rebuilds are now very fast:
$ make all install -j8
$ time make all install -j8
real 0m5.735s
Less than 6s to figure out that nothing needs to be done.
And strangely flushing the disk cache doesn't make it much slower:
$ sudo purge
$ time make all install -j8
real 0m7.309s
Which probably means that Make mostly reads file metadata, and not actual file
contents (which is good).
This fixes a file sharing exception:
> MTOUCH: error MT1009: Could not copy the assembly '[...]/msbuild/tests/MyActionExtension/bin/iPhone/Debug/MyActionExtension.dll' to '[...]/msbuild/tests/MyTabbedApplication/obj/iPhone/Debug/mtouch-cache/32/Link/MyActionExtension.dll': Sharing violation on path [...]/msbuild/tests/MyActionExtension/bin/iPhone/Debug/MyActionExtension.pdb
This should stop errors like below to happen on wrench
```
Xamarin.Bundler.MonoMacException: The type 'AVFoundation.IAVContentKeyRecipient' (used as a parameter in AVFoundation.AVContentKeySession.Add) is not available in macOS 10.12.2 (it was introduced in macOS 10.12.4). Please build with a newer macOS SDK (usually done by using the most recent version of Xcode).
```
A fix (thanks Rolf!) in how we do SDK version checks is needed
because for some reason `new Version (3, 2, 0)` isn't the same to
`new Version (3, 2)` and we end up with a MT4134 when building the
watchOS static registrar.
When using debug simulator we don't generate main.m so we were not passing the gc options.
The MONO_GC_PARAMS variable is not in app.EnvironmentVariables (which only contains environment variables passed to mtouch using --setenv), which is why the above condition does not trigger.
No frameworks should be bundled in WatchKit 1 extensions, they should be
bundled in the container (iOS) app.
This broke when merging the equivalent fix for master into the framework-sdk
branch (i.e. a broken merge).
- Update comments on XM45.targets file
- Remove unnecessary AssemblySearchPaths hack causing issues using nugets with same name as Facades
- Note: MSBuild with XM 4.5 is still broken for now
This cuts down another group of conditional compilation sections, paving the
way for an IKVM-based generator.
This makes it required to pass --target-framework for to generator executables
(previously only required for Xamarin.Mac/Unified to distinguish between the
different Xamarin.Mac/Unified variants), but it should be invisible to users
since we'll automatically pass the correct --target-framework argument from
the corresponding scripts (btouch/btv/bwatch/bmac) and the MSBuild targets.
This will only break somebody who is executing the managed executables
directly, but nobody should do that in the first place (it's not a supported
scenario).
Generated diff: https://gist.github.com/rolfbjarne/1674be6625632446dba774a305951981
Besides the obvious reasons, this is also useful when testing warnings, since
by making warnings errors, mtouch/mmp will exit a lot faster (and the tests
will finish faster).
The cached linker results can have multiple identical input assemblies (for
assemblies that show up in both the app and any app extensions), so make sure
we don't load those more than once.
Since the linker can process multiple apps/appex'es at the same time, it also
means it will put together all the required symbols found in _all_ assemblies.
This means that we need to filter out required symbols for other
apps/appex'es.
Change cache invalidation so that if any app extension's cache is invalid,
then invalidate the cache for the container app and all other app extensions.
This is the safest option when we're sharing code.
Don't use the global command line arguments to determine input, because that's
not the input we use for app extensions anymore.
Instead explicitly pass the input arguments when creating the cache.
Since neither mtouch nor mmmp is mkbundled anymore, the installed binary is in
fact a shell script.
This means that it's quite useless to check if the shell script has been
modified; instead check if the executing assembly has been modified (which
works now that we're not mkbundled anymore).
We must build each appex bundle before the container bundle, so that we can
compute the frameworks each appex the needs before bundling the container app.
Also there's no need to store the list of frameworks appex's need in a file,
since everything is now done in the same mtouch process.
Implement support for sharing both code and resources between app extensions
and their container app:
* AOT-compiled code. Each shared assembly is only AOT-compiled once, and if
the assembly is built to a framework or dynamic library, it will also only
be included once in the final app (as a framework or dynamic library in the
container app, referenced directly by the app extension). If the assemblies
are built to static objects there won't be any size improvements in the app,
but the build will be much faster, because the assemblies will only be AOT-
compiled once.
* Any resources related to managed assemblies (debug files, config files,
satellite assemblies) will be put in the container app only.
Since these improvements are significant, code sharing will be enabled by
default.
Test results
============
For an extreme test project with 7 extensions (embedded-frameworks)[1]:
with code sharing cycle 9 difference
build time 1m 47s 3m 33s -1m 46s = ~50% faster
app size 26 MB 131 MB -105 MB = ~80% smaller
For a more normal test project (MyTabbedApplication)[2] - this is a simple application with 1 extension:
with code sharing cycle 9 difference
build time 0m 44s 0m 48s -4s = ~ 8% faster
app size 23 MB 37 MB -15 MB = ~40% smaller
Another tvOS app with one extension also show similar gains (MyTVApp)[3]:
with code sharing cycle 9 difference
build time 0m 22s 0m 48s -26s = ~54% faster
app size 22 MB 62 MB -40 MB = ~65% smaller
[1]: https://github.com/rolfbjarne/embedded-frameworks
[2]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTabbedApplication
[3]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTVApp
Warn if mtouch loads an assembly from a different location than requested
(which might be because there are multiple assemblies with the same name).
Also rework the MT0023 check a bit by explicitly loading the root assembly
first, and then detecting if any loaded assemblies matches the root assembly.
This results in code that's a bit more obvious, and it also works correctly
with extensions (previously the entire MT0023 check was skipped for
extensions).
Allow the assembly build target name for frameworks to end with '.framework',
so that the following:
--assembly-build-target=@sdk=framework=Xamarin.Sdk.framework
doesn't end up creating Xamarin.Sdk.framework.framework.
Store the location of every assembly that can't be deduced at runtime (i.e.
all assemblies that are build to frameworks, since there can be multiple
assemblies in each framework, and the framework name can be customized).
Detect when assemblies have native dependencies between them (which can happen
when there are multiple binding projects, and the native libraries in those
binding projects have dependencies between them), and add the proper link
arguments (this is only required when building to dynamic libraries or
frameworks, since otherwise everything is linked to one big binary and there
are no dependency problems).
https://bugzilla.xamarin.com/show_bug.cgi?id=43689
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
The AOT-compilation occurs in the AOT-task now, and then we compile the result
using CompileTask.
This means that the error message in CompileTask was slightly incorrect, so
rectify it.
This makes dylibs automatically have the correct dylib id, which means no
fixups are required.
For instance: we'd build libpinvokes.armv7.dylib from libpinvokes.armv7.m,
which by default ends up with a dylib id of "libpinvokes.armv7.dylib". With
this fix no change is required, since we now build armv7/libpinvokes.dylib
from armv7/libpinvokes.m.
Compute the dependency map for assemblies earlier, and store the results.
In a later commit we'll need to know if a dependency map was successfully
computed when determining if a task is up-to-date or not.
Rework the code that copies assemblies and their related files to the app
bundle to take into account that we might be building to frameworks now.
Also strip the assemblies when they're copied (if they must be stripped),
which removes the need for custom logic to copy files related to stripped
assemblies.
Additionally change how we handle duplicated assemblies by checking for
duplication before copying them to the app bundle. This allows us to copy
assemblies to the root directory (not the .monotouch-[32|64] subdirectory) if
the 32-bit and 64-bit versions are identical, which also means we won't need
symlinks anymore.
Make the architecture a suffix instead of infix for aotdata filenames so that
it's easier to compute the filename from the assembly name without passing
printf-style format strings around.
Create a custom AssemblyCollection class that contains a dictionary with
assembly identity (name) -> Assembly mapping.
This also means that we can detect if we end up loading multiple assemblies
with the same identity, and show an error in that case (even if that case
should never happen since we cache assemblies based on the identity, it's nice
to have code that ensures it).
Rolf Bjarne Kvinge
Sebastien Pouliot
Alex Soto
Chris Hamons
Timothy Risi
Vincent Dondain
Marek Safar
Vlad Brezae
Zoltan Varga