This avoids the following compiler warning:
/Applications/Xcode83.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/OpenGL.framework/Headers/gl.h:5:2: warning: gl.h and gl3.h are both included. Compiler will not invoke errors if using removed OpenGL functionality. [-W#warnings]
I found the magic symbol by looking at the headers.
If both an extension and the container app (or multiple extensions) reference
the same binding assembly for a framework, then we'd error out with an
internal error when trying to copy the framework from both locations to the
container app.
So instead detect when we're trying to copy multiple identical (by comparing
the on-disk contents) frameworks, and only copy one of them.
We'll still show an error if the frameworks are different, but now a nice
MT1035 error with a proper error description instead of an internal error.
https://bugzilla.xamarin.com/show_bug.cgi?id=56635
[registrar] Support 'out' parameters from NULL pointers. Fixes#54919.
Native code doesn't have the 'out' and 'ref' distinction C# has, and passes
NULL around left and right.
So make sure the generated code from the static registrar doesn't write to such NULLs.
https://bugzilla.xamarin.com/show_bug.cgi?id=54919
Replace https://github.com/xamarin/xamarin-macios/pull/1973 expect that
the test parts are still needed.
* Add XM SDK + LinkSkip test
* [macos] Add platform linking support to msbuild
* [macos] Add full SDK test
* [macios] Diable classic from using linkplatform
- Extended test infrastructure change to allow classic projects that include bundling
- Setting linkplatform in MonoBundlingExtraArgs since we don't even read project setting LinkMode - Platform for classic
This makes registrar.h compilable as Objective-C (as opposed to
Objective-C++), because the __monoObjectGCHandle field (whose type is an C++
type) isn't there anymore.
When gathering frameworks, gather into the product assembly's frameworks, not
the global frameworks.
This is necessary when building a non-linked dylib/framework in embeddor mode,
because we link the registrar code into the framework, and that code will
require linking with all the frameworks all the assemblies require.
- Before this mmp was not adding -framework, -weak_framework consistently on non-static registrar use cases
- GatherFrameworks was previously not ported from mtouch, and did not work as DeploymentTarget was unset in mmp
- Added verbose prints so users can determine why various framework linkages are added
- Fixed an issue where duplicate were being added due to HandleFramework shoving args by hand
- Tested with auto test and https://github.com/chamons/xm-version-regression-test manual test
This fixes a file sharing exception:
> MTOUCH: error MT1009: Could not copy the assembly '[...]/msbuild/tests/MyActionExtension/bin/iPhone/Debug/MyActionExtension.dll' to '[...]/msbuild/tests/MyTabbedApplication/obj/iPhone/Debug/mtouch-cache/32/Link/MyActionExtension.dll': Sharing violation on path [...]/msbuild/tests/MyActionExtension/bin/iPhone/Debug/MyActionExtension.pdb
A fix (thanks Rolf!) in how we do SDK version checks is needed
because for some reason `new Version (3, 2, 0)` isn't the same to
`new Version (3, 2)` and we end up with a MT4134 when building the
watchOS static registrar.
This cuts down another group of conditional compilation sections, paving the
way for an IKVM-based generator.
This makes it required to pass --target-framework for to generator executables
(previously only required for Xamarin.Mac/Unified to distinguish between the
different Xamarin.Mac/Unified variants), but it should be invisible to users
since we'll automatically pass the correct --target-framework argument from
the corresponding scripts (btouch/btv/bwatch/bmac) and the MSBuild targets.
This will only break somebody who is executing the managed executables
directly, but nobody should do that in the first place (it's not a supported
scenario).
Generated diff: https://gist.github.com/rolfbjarne/1674be6625632446dba774a305951981
Besides the obvious reasons, this is also useful when testing warnings, since
by making warnings errors, mtouch/mmp will exit a lot faster (and the tests
will finish faster).
Change cache invalidation so that if any app extension's cache is invalid,
then invalidate the cache for the container app and all other app extensions.
This is the safest option when we're sharing code.
Don't use the global command line arguments to determine input, because that's
not the input we use for app extensions anymore.
Instead explicitly pass the input arguments when creating the cache.
Since neither mtouch nor mmmp is mkbundled anymore, the installed binary is in
fact a shell script.
This means that it's quite useless to check if the shell script has been
modified; instead check if the executing assembly has been modified (which
works now that we're not mkbundled anymore).
Implement support for sharing both code and resources between app extensions
and their container app:
* AOT-compiled code. Each shared assembly is only AOT-compiled once, and if
the assembly is built to a framework or dynamic library, it will also only
be included once in the final app (as a framework or dynamic library in the
container app, referenced directly by the app extension). If the assemblies
are built to static objects there won't be any size improvements in the app,
but the build will be much faster, because the assemblies will only be AOT-
compiled once.
* Any resources related to managed assemblies (debug files, config files,
satellite assemblies) will be put in the container app only.
Since these improvements are significant, code sharing will be enabled by
default.
Test results
============
For an extreme test project with 7 extensions (embedded-frameworks)[1]:
with code sharing cycle 9 difference
build time 1m 47s 3m 33s -1m 46s = ~50% faster
app size 26 MB 131 MB -105 MB = ~80% smaller
For a more normal test project (MyTabbedApplication)[2] - this is a simple application with 1 extension:
with code sharing cycle 9 difference
build time 0m 44s 0m 48s -4s = ~ 8% faster
app size 23 MB 37 MB -15 MB = ~40% smaller
Another tvOS app with one extension also show similar gains (MyTVApp)[3]:
with code sharing cycle 9 difference
build time 0m 22s 0m 48s -26s = ~54% faster
app size 22 MB 62 MB -40 MB = ~65% smaller
[1]: https://github.com/rolfbjarne/embedded-frameworks
[2]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTabbedApplication
[3]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTVApp
Warn if mtouch loads an assembly from a different location than requested
(which might be because there are multiple assemblies with the same name).
Also rework the MT0023 check a bit by explicitly loading the root assembly
first, and then detecting if any loaded assemblies matches the root assembly.
This results in code that's a bit more obvious, and it also works correctly
with extensions (previously the entire MT0023 check was skipped for
extensions).
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
This makes dylibs automatically have the correct dylib id, which means no
fixups are required.
For instance: we'd build libpinvokes.armv7.dylib from libpinvokes.armv7.m,
which by default ends up with a dylib id of "libpinvokes.armv7.dylib". With
this fix no change is required, since we now build armv7/libpinvokes.dylib
from armv7/libpinvokes.m.
Create a custom AssemblyCollection class that contains a dictionary with
assembly identity (name) -> Assembly mapping.
This also means that we can detect if we end up loading multiple assemblies
with the same identity, and show an error in that case (even if that case
should never happen since we cache assemblies based on the identity, it's nice
to have code that ensures it).
Cecil cannot save an assembly more than once without reloading its
debugging symbols. There was code for this, in the special case where
an assembly could be saved more than once. However the order of the
operations changed and the code to reload symbols was now located too
late.
https://bugzilla.xamarin.com/show_bug.cgi?id=51667
Resource files for mac should be copied to the app bundle
* Move the Satellite code used by mac to tools/common/Assembly.cs
* Add EmbeddedResources test to xammac_tests
Most projects building to bitcode (any kind of bitcode, this includes the
marker-only version as well), will fail to link when linking with third-party
libraries and incremental builds are enabled.
So automatically disable incremental builds when we detect this scenario.
This is only a workaround until we can make this scenario build correctly.
https://bugzilla.xamarin.com/show_bug.cgi?id=51710
Use @rpath instead of @executable_path in dylibs, since it allows us to be
more flexible when placing dylibs in the app.
In particular with this change it's trivial to put libmonosgen-2.0.dylib in
the container app, and reference it from extensions.
Event sequence:
* mtouch is executed with the linker disabled.
* The linker pipeline copies all input assemblies (since the linker is
disabled the assemblies don't change) into the PreBuild directory. This will
keep the original timestamps of the input assemblies.
* mtouch is executed again, when none of the input assemblies changed.
* The linker pipeline will re-execute, because it will see that at least one
of the input assemblies (at least the .exe) is newer than at least one of
the assemblies in the PreBuild directory (usually a framework assembly,
because those have the original timestamp from their install location).
Fix:
Touch all the assemblies in the PreBuild directory after the linker pipeline
executes the first time. This way the second time mtouch is executed, it will
find that all assemblies in the PreBuild directory have timestamps later than
all the input assemblies, so it will load the cached linked assemblies,
instead of re-executing the linker pipeline.
The native linker treats object files (.o) and static libraries (.a files,
which are archives of .o files) differently.
The native linker will always include object files into the executable:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ld foo.o -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
0000000000000fe0 T _xxx
However, if the object file is inside a static library:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ar cru foo.a foo.o
$ ld foo.a -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
<no output>
This means that our testing library (libtest.a) which is a fat library of
_object files_, do not show the problems reported in bug #51548.
So:
a) I've fixed the creation of libtest.a to be a fat library of _static
libraries_. This causes the `FastDev_LinkWithTest` test to fail exactly
like in bug #51548.
b) I've made mtouch pass `-u <native symbol>` to the native linker, for every
native symbol referenced in a managed assembly, when creating a dylib.
Amazingly this seems to work fine even with symbols to Objective-C classes
(`_OBJC_CLASS_$_<class name>`).
c) This also required adding support for collecting the Objective-C names of
all managed types registered with Objective-C to the linker. The
information is already available in the static registrar, but that would
require us to make sure the static registrar is executed before compiling
dylibs, which means those two tasks won't be able to run in parallel (also
there's no guarantee we'll even run the static registrar).
https://bugzilla.xamarin.com/show_bug.cgi?id=51548
Simplify code a bit to avoid constant null checking and also take advantage of
the fact that HashSet.Add returns if the value was added or not (to avoid a
Contains check).
So that there can be multiple caches in the same process (which we'll have
once mtouch can compile extensions and the container app in the same process).
Fixes this test failure:
Xamarin.Registrar.MT4161 : Unexpected error/warning with --registrar:static:
error MT0000: Unexpected error - Please file a bug report at http://bugzilla.xamarin.com
* [mtouch] Remove most static state and put it on an instance.
Remove most static state, and put it on an instance (of Application) instead.
Soon we'll build multiple apps (app extensions) in the same process, and we
need to make sure we're not inadvertedly (and incorrectly) sharing state
between apps, so remove as much static state as possible to avoid any
problems.
* [mtouch] Rename GetIsUsingClang -> IsUsingClang.
Add support for specifying the profile with the target framework,
and support using "Version=" before the version:
Xamarin.Mac,Version=v2.0,Profile=Mobile.
-lsqlite3 is a linker flag, not a file to be linked with, so when
automatically determining that we need to pass -lsqlite3 we need to put it in
the right list of linker information.
Otherwise we may end up passing `-force_load -lsqlite3` to the linker (if the
assembly's ForceLoad flag is set), which won't compile.
https://bugzilla.xamarin.com/show_bug.cgi?id=49220
Use metadata tokens instead of strings to find types and methods.
This makes the code to find methods more compact (a lot less strings in the
executable, and additionally in most cases a compact representation (32-bit
integer) of the corresponding metadata token and additional information can be
used, which results in less executable code (fewer parameters to methods,
etc)), resulting in smaller executables.
Size savings are around 200kb for dont link apps, and 20-60kb for linked apps
(this obviously varies a lot depending on how much has to registered by the
registrar).
| | Before | After | Diff |
|----------------|--------------:|--------------:|------------------:|
| dontlink/32bit | 102.810.144 | 102.609.456 | -200.688 = -0,20% |
| dontlink/64bit | 107.420.576 | 107.221.792 | -198.784 = -0,19% |
| linksdk/32bit | 40.957.296 | 40.936.864 | -20.432 = -0,05% |
| linksdk/64bit | 43.113.136 | 43.093.936 | -19.200 = -0,04% |
| linkall/32bit | 38.410.032 | 38.348.288 | -61.744 = -0,16% |
| linkall/64bit | 40.315.200 | 40.267.344 | -47.856 = -0,12% |
Additionally I've removed the `lazy_map` dictionary, which we populated at
startup and was used to map between Class instances and the corresponding
managed type's FullName, and instead iterate over a native array of Class ->
metadata token mappings whenever we need to look up the managed type for a
certain Class instance.
This is slightly slower for each type we need to look up (for a non-linked app
there might be a 2000-3000 entries in the native array, which would be
iterated instead of using a hashtable lookup), but it's only done once per
type and there's a significant startup memory improvement.
For a non-linked test app I get the following using the Xamarin profiler:
| | Before | After | Diff |
|-------------------|--------:|--------:|----------------:|
| Memory allocated | 2,8 MB | 2,4 MB | -0,4 MB = -14 % |
| Objects allocated | 43678 | 38463 | -5215 = -12 % |
| Private bytes | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
| Working set | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
Right now the logic exists in a few places, both in and outside the
linker. We recently began to use part of the linker pipeline in normal /
all builds so it's easier to share (and unify) the code now.
The real gain is to avoid copying assemblies, in particular large ones,
more than strictly needed while building.
E.g. a build including a very large 1.3GB assembly, with several
native libraries embedded, save a lot of time avoiding the rewrites
mtouch (before)
Total time: 64202 ms
mtouch (after)
Total time: 34840 ms
* Add XM support for RemoveUserResourcesSubStep
* Tests supplied by @chamons
* Comment is wrong: the code is never executed in parallel
* If reached then RemoveResources will be called and already check for
symbols (and load them when needed). Just a simplification (it won't
really save time as it's not loaded twice)
* [mtouch] Add missing HasParameters check inside linker and static registrar
Without the checks new, empty collections can be allocated and their
whole and only purpose will be to iterate up to 0 (nop).
The checks saves a small amount of memory (collections) and time.
* [registrar] Fix method comparison when they have no parameters
Managed exception marshaling interferes with the debugger, because it adds
exception handlers to executing code, which makes the Mono runtime think an
exception is handled when logically it's not (although technically it is).
The consequence is that the IDEs will only be notified when we re-throw the
exception after catching it, making it impossible for the IDEs to stop when
the exception is thrown (they will instead stop when we re-throw the
exception).
So disable managed exception marshaling (unless the user changed the default
behavior) when a debugger is attached.
This is the same behavior as Xamarin.Android.
https://bugzilla.xamarin.com/show_bug.cgi?id=45116
This makes it possible to set linker flags per assembly:
[assembly: LinkWith (LinkerFlags = "-lsqlite3")]
Which is required when incremental builds is enabled and a particular assembly
needs special linker flags (because we don't propagate the global -gcc_flags
to each dylib we build when doing incremental builds).
Also add an option to set the dlsym mode for an assembly (using the LinkWith
attribute).
Duration before: 1,60s
Duration after: 1,54s
Difference: -0,06s = -3,8%
Memory usage hardly changed (-21 kb of 540 MB), but the number of method calls
shrunk significantly.
Method calls before: 86.720.379
Method calls after: 74.390.061
Difference: -12.330.318 = -14,2%
The call to `GetSystemVoidType` was #2 on the list of method calls (whens
sorted by 'self'), called 1072 times taking 1429 ms each time. After this
change it's only called once (and obviously pushed way down the list).
Fix NullReferenceException in PInvoke wrapper generation when incremental
builds are enabled and the linker didn't run because cached results were
found.
https://bugzilla.xamarin.com/show_bug.cgi?id=44763
* [tests] Remove Classic SDK tests.
* Remove XI/Classic support.
This also means we can remove support for the legacy registrars.
* [monotouch-test] Remove legacy registrar tests.
* [tests/mtouch] Remove Classic tests (and legacy registrar logic).
* [tests/scripted] Fix tests to reference Xamarin.iOS.dll.
We need to compile the generated P/Invoke wrappers to a dylib, and link the
dylib for the product assembly (Xamarin.WatchOS.dll) with the generated
P/Invoke wrappers.
Since there might be P/Invokes in any assembly, just link in the P/Invoke
wrapper dylib for every assembly.
https://bugzilla.xamarin.com/show_bug.cgi?id=44048
note: AVFoundation is commented on watchOS 3 as this breaks the static
registrar .a helper built for watchOS.
AVFoundation does a file check to enable some types - but the watchSimulator
include that file, leading to compilation errors later (missing CMTime.h)
#if TARGET_OS_WATCH
#if ! __has_include(<AVFoundation/AVAnimation.h>)
#define AVF_IS_WATCHOS_SDK 1
#endif
#endif
* Bump [watch-]mono to master to get fix for #43658.
https://bugzilla.xamarin.com/show_bug.cgi?id=43658
* [mtouch/mmp] Fix build after breaking cecil update in mono.
Also use mono's cecil instead of our own cecil submodule for mtouch.
* Bump [watch-]mono to get compilation fixes after cecil bump in mono.
* Remove cecil submodule, we only use the one in mono now.
* Enable some SceneKit-related WatchKit API
* Enable some SceneKit-related SpriteKit API
* Enable some SceneKit-related Foundation API
* Fix generator to include `using SceneKit;` on watchOS
* Adjust xtro tests since watchOS headers include some stuff that's not available in reality
* Lots of [Watch (3,0)] attributes
This is only applicable to tvOS 10 (i.e. the xcode8 branch),
but applying the patch already helps avoiding merge failures
for other fixes in the same area.
Category methods are exposed like extension methods, and the first parameter
specifies the class, which means we need to skip the first type when generating
the ObjC signature.
https://bugzilla.xamarin.com/show_bug.cgi?id=42489
There can be circular dependencies between Objective-C classes,
so make sure we don't fail compilation when that occurs by
forward declaring any Objective-C classes/protocols.
The test case in question does not contain a circular dependency,
but the same issue occurs due to types not being generated in the
correct order (a correct order could be constructed for the test
case, but there's no general solution since circular dependencies
can exist).
https://bugzilla.xamarin.com/show_bug.cgi?id=42454
Category methods are exposed like extension methods, and the first parameter
specifies the class, which means we need to skip the first type when generating
the ObjC signature.
https://bugzilla.xamarin.com/show_bug.cgi?id=42489
There can be circular dependencies between Objective-C classes,
so make sure we don't fail compilation when that occurs by
forward declaring any Objective-C classes/protocols.
The test case in question does not contain a circular dependency,
but the same issue occurs due to types not being generated in the
correct order (a correct order could be constructed for the test
case, but there's no general solution since circular dependencies
can exist).
https://bugzilla.xamarin.com/show_bug.cgi?id=42454
Fixes NSCharacterSetTest.NSMutableCharacterSet_TestStaticSets when
running with the P/Invoke wrapper (for exceptions) enabled (i.e.
watchOS), since otherwise the wrapper would truncate char parameters
to byte.
* Added Speech Framework bindings
* Ensured introspection test pass
* Ensured we link against Speech framework
* FIXME: SFSpeechRecordingRecognitionRequest is not in the public api
filled radar://26799291 https://trello.com/c/s6s6YKua
* Added CallKit Bindings
* Ensure CallKit.framework is linked
* Ensure CallKit passes introspection tests
* FIXME: https://trello.com/c/afWXDZ3A
Headers says CallKit is available on macOS 10.12
but uses AVAudioSession and it is iOS only
Opened Radar awaiting response.
There's a clang bug [2] where if a selector is marked as unavailable,
it's marked as unavailable for every class, not just the class where
the unavailable selector is.
This means that we can't do `[super initWithCoder:x]` anywhere,
because `initWithCoder:` is marked as unavailable for UIActivityViewController.
So instead rewrite the call to super to call objc_msgSendSuper
directly, circumventing clang's broken availability checks.
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=41319
[2] https://llvm.org/bugs/show_bug.cgi?id=28058
mtouch only uses Xamarin.Mac to read plists, so change to use
our purely managed plist reader in Xamarin.MacDev instead.
That makes us able to change mtouch to be a normal command-line
executable (and project).
Which makes it logical to not mkbundle mtouch anymore,
it executes just fine with the system mono (and there's
no code to protect anymore either).
And since mmp and mtouch share some files, do the same
for mmp.
Rolf Bjarne Kvinge
Sebastien Pouliot
Chris Hamons
Alex Soto
Vincent Dondain
Marek Safar
Timothy Risi
Vlad Brezae
Sebastien Pouliot
Manuel de la Pena
Alex Soto
Timothy Risi