Put simulator assemblies in MonoBundle/simulator for frameworks, so that we
can have a single framework that contains both device and simulator
assemblies without assemblies conflicting between device and simulator.
The device assemblies continue in the same place, in the MonoBundle directory,
so no additional checks are needed on device.
Previously we copied any equivalent .dylib and ran install_name_tool on the
library to change the library id to make it a framework.
Unfortunately this does not work when the library contains bitcode, because
bitcode embeds linker flags (-install_name for instance), and
install_name_tool does not change those linker flags.
This means that we need to create frameworks by linking with the proper
arguments, since it's much more difficult to fixup the embedded bitcode linker
flags as well.
So change how be build Mono.framework, Xamarin.framework, and any frameworks
built from assemblies to:
* Always link instead of fixup a dylib. For Mono.framework this means
extracting all the object files from libmonosgen-2.0.a and linking those,
for Xamarin.framework this means linking the object files we've already
built.
* Make sure the library is correctly named when linked (once again: bitcode
contains embedded linker flags, so renaming the executable later breaks
stuff as well).
I've also extracted the logic that creates Mono.framework from
libmonosgen-2.0.a to a separate shell script, to deduplicate this logic.
This required a minor change in the mono builds: we need the Mono.framework
when building the `all` target, so make sure that happens.
https://bugzilla.xamarin.com/show_bug.cgi?id=53813
Fixes bug #52575: Missing ModelIO API
(https://bugzilla.xamarin.com/show_bug.cgi?id=52575)
We turn the MDLMesh constructors into static ones because we don't want to lose the shape name.
Also, the signatures of these constructors differ so it would not be possible to use an enum to differentiate the shapes.
* Bump maccore to get fix for launching the simulator for app extensions.
* [runtime] Don't look in shared memory for debug data in normal apps.
Don't look in shared memory for debug data in normal apps, because it
interferes when debugging extensions from the same solution as a container
app:
Example, for a keyboard extension:
1. Run extension in XS.
2. Manually launch the extension's container app (which contains a textbox
that can be used to launch the keyboard).
3. The container app reads the debug data in shared memory which was intented
for the extension, and takes over the debugger.
4. Launch keyboard, which will not be able to connect to the IDE because the
container app already connected to the IDE.
Clang's static analyzer tought me something new today: 'copy' properties are
not automatically deallocated 😒:
monotouch-debug.m:237:1: warning: 'XamarinHttpConnection' lacks a 'dealloc' instance method but must release '_completion_handler' and others
@implementation XamarinHttpConnection
^
This warning:
shared.m:216:3: warning: Incorrect decrement of the reference count of an object that is not owned at this point by the caller
[pool release];
^~~~~~~~~~~~~~
is somewhat incorrect, because we're using NSAutoreleasePools in uncommon ways
(at the same time it's not entirely incorrect either, because we're not
following Apple's documentation about how to use NSAutoreleasePools).
Luckily calling [NSAutoreleasePool drain] is equivalent to [NSAutoreleasePool
release], so just replace the latter with the former to silence the warning,
since clang doesn't know those two are equivalent.
At the beginning of dawn, not long after the big bang, we used to vibrate the
phone if there was a problem when connecting to the debugger in the IDE, and
the number of times the phone vibrated indicated a certain error condition.
This innovative debugging technique was at some point replaced with more
intuitive (albeit less innovative) error reporting, however the *_connect_*
family of functions continued returning the number of times the phone should
vibrate, even if the caller was now soundlessly ignoring this information.
After many years someone had the wonderful idea of running a static analyzer
on the code, and now the unused return value is not so silent anymore:
monotouch-debug.m:663:5: warning: Value stored to 'rv' is never read
rv = monotouch_connect_usb ();
^ ~~~~~~~~~~~~~~~~~~~~~~~~
monotouch-debug.m:665:5: warning: Value stored to 'rv' is never read
rv = monotouch_connect_wifi (hosts);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
monotouch-debug.m:667:5: warning: Value stored to 'rv' is never read
rv = xamarin_connect_http (hosts);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So fix the warning by removing the return value.
This fixes an issue clang's static analyzer found: if an exception occured in
a call to managed code, we wouldn't process the list of objects to dispose:
trampolines-invoke.m:549:2: warning: Potential leak of memory pointed to by 'dispose_list'
MONO_THREAD_DETACH; // COOP: This will switch to GC_SAFE
^~~~~~~~~~~~~~~~~~
./xamarin/runtime.h:306:11: note: expanded from macro 'MONO_THREAD_DETACH'
} while (0)
^
Fix this by always processing the list of objects to dispose, even if
exceptions occur.
trampolines.m:75:22: warning: Result of 'malloc' is converted to a pointer of type 'id', which is incompatible with sizeof operand type 'void *'
id *buf = (id *) malloc (sizeof (void *) * length);
~~~~ ^~~~~~ ~~~~~~~~~~~~~~~
trampolines-i386.m:126:19: warning: Assigned value is garbage or undefined
it->state->edx = v[1];
^ ~~~~
trampolines-x86_64.m:410:16: warning: Assigned value is garbage or undefined
*reg_ptr = v [stores];
^ ~~~~~~~~~~
trampolines-x86_64.m:447:14: warning: Assigned value is garbage or undefined
*reg_ptr = v [stores++];
^ ~~~~~~~~~~~~
Getting a time zone instance in two different code paths, one allocating the
time zone instance, and the other not allocating, is problematic when we're
unconditionally freeing the time zone at the end.
xamarin-support.m:58:2: warning: Incorrect decrement of the reference count of an object that is not owned at this point by the caller
[tz release];
^~~~~~~~~~~~
So fix this by autoreleasing the allocated timezone.
If an extension is sharing code with the container app, then assemblies may be
placed in:
* The container app's Framework directory (for assemblies whose code is
shared, and the build action is 'framework'). We already handle this case.
* The container app's root directory (for assemblies whose code is shared, and
the build action is not 'framework'). This case we didn't handle, and we're
now fixing it.
* In the extension (for assemblies whose code is not shared). We already
handle this case.
Add support for finding assembly-related resources (debug files, config files,
satellite assemblies, aotdata files, etc) in multiple locations:
* The bundle's root directory.
* A pointer-size specific size directory (.monotouch-32|64)
* A framework's MonoBundle directory (and the framework may not be named after
the assembly, which means it's not necessarily possible to deduce the
framework's name at runtime).
* In a container bundle.
Some files may also have an arch-specific suffix.
Make the architecture a suffix instead of infix for aotdata filenames so that
it's easier to compute the filename from the assembly name without passing
printf-style format strings around.
* [XM] Seperate parsing from compilation in AOT code
- Parsing command line options was too entangled with actually compiling
- Refactor test code to centralize Compile, reducing some complexity of tests
- Groundwork for hybrid vs non-hybrid AOT work
* Add hybrid/standard AOT support
* Add xammac_tests to makefile/jenkins
* Add flag to force AOT
* Disable MonoTouchFixtures.Contacts.ContactStoreTest.GetUnifiedContacts on XM due to prompts on Jenkins
Use @rpath instead of @executable_path in dylibs, since it allows us to be
more flexible when placing dylibs in the app.
In particular with this change it's trivial to put libmonosgen-2.0.dylib in
the container app, and reference it from extensions.
This is a series of fixes to the dynamic libraries we build / create to remove
any unnecessary bloat (unused architectures, bitcode).
A brand new watchOS app with no changes goes from 35MB to 11MB with these
fixes (with incremental builds disabled, the app size is 10MB).
--------------------------------------------------------------------------------
* [runtime] Split list of architectures into simulator and device-specific lists.
* [runtime] Build separate dylibs for device and simulator.
Build separate dylibs for device and simulator, since we already install these
into different locations.
This makes both the simulator and device builds slightly faster (since the
respective dylibs are smaller, and less data to copy around).
For watchOS apps, this saves ~430kb.
* [runtime] Strip bitcode from dylibs. Fixes#51352.
We know that dylibs will never be shipped to the App Store, so we'll never
need them to have bitcode.
So just strip the bitcode from all our dylibs, since this makes apps with
fastdev significantly smaller (and thus much faster to upload to watch
devices).
For watchOS apps this is a very significant improvement: a branch new watchOS
app without any changes goes from 35MB to 17MB.
https://bugzilla.xamarin.com/show_bug.cgi?id=51352
* [mtouch] Fix dylib compilation to not embed full bitcode.
Facts
=====
a. The output from the AOT compiler is an assembly (.s) file.
b. Clang's assembler does not support -fembed-bitcode-marker (only -fembed-
bitcode), so when we ask clang to -fembed-bitcode-marker, the assembler
receives a -fembed-bitcode argument.
c. This means that the assembled object file does not contain the
__LLVM/__bitcode and __LLVM/__cmdline sections (it does however contain an
__LLVM/__asm section).
d. The native linker will create a bitcode assembly section if none of the
object files passed to the linker contain a __LLVM/__bitcode section and
there's an __LLVM/__asm section present.
e. The end result is that when we build to a dylib, we end up (unexpectedly,
because we ask Clang to -fembed-bitcode-marker) including both armv7k and
bitcode in the dylib, thus bloating the dylib size significantly.
Solution
========
We manually add the __LLVM/__bitcode and __LLVM/__cmdline sections to the .s
file Mono's AOT compiler generated. This way the .o file will have the magic
sections, and the linker will not include bitcode (only the bitcode marker) in
the final library.
An empty watchOS extension with incremental builds is now 6MB smaller (down
to 11MB from 17MB).
The completion handler block must be copied to the XamarinHttpConnection
instance, otherwise we'll just store a pointer to a stack block, and once we
invoke the block we'll access stack memory (probably from another thread),
which is really not what we want to do.
https://bugzilla.xamarin.com/show_bug.cgi?id=44568
The length argument for `strncpy` specifies how many characters to copy, not
the length of the target string, which makes our usage incorrect.
Fix our usage, and use the `strlcpy` variant instead, which takes the length
of the target string.
And use `strlcat` instead of `strcat`.
Use metadata tokens instead of strings to find types and methods.
This makes the code to find methods more compact (a lot less strings in the
executable, and additionally in most cases a compact representation (32-bit
integer) of the corresponding metadata token and additional information can be
used, which results in less executable code (fewer parameters to methods,
etc)), resulting in smaller executables.
Size savings are around 200kb for dont link apps, and 20-60kb for linked apps
(this obviously varies a lot depending on how much has to registered by the
registrar).
| | Before | After | Diff |
|----------------|--------------:|--------------:|------------------:|
| dontlink/32bit | 102.810.144 | 102.609.456 | -200.688 = -0,20% |
| dontlink/64bit | 107.420.576 | 107.221.792 | -198.784 = -0,19% |
| linksdk/32bit | 40.957.296 | 40.936.864 | -20.432 = -0,05% |
| linksdk/64bit | 43.113.136 | 43.093.936 | -19.200 = -0,04% |
| linkall/32bit | 38.410.032 | 38.348.288 | -61.744 = -0,16% |
| linkall/64bit | 40.315.200 | 40.267.344 | -47.856 = -0,12% |
Additionally I've removed the `lazy_map` dictionary, which we populated at
startup and was used to map between Class instances and the corresponding
managed type's FullName, and instead iterate over a native array of Class ->
metadata token mappings whenever we need to look up the managed type for a
certain Class instance.
This is slightly slower for each type we need to look up (for a non-linked app
there might be a 2000-3000 entries in the native array, which would be
iterated instead of using a hashtable lookup), but it's only done once per
type and there's a significant startup memory improvement.
For a non-linked test app I get the following using the Xamarin profiler:
| | Before | After | Diff |
|-------------------|--------:|--------:|----------------:|
| Memory allocated | 2,8 MB | 2,4 MB | -0,4 MB = -14 % |
| Objects allocated | 43678 | 38463 | -5215 = -12 % |
| Private bytes | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
| Working set | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
Managed exception marshaling interferes with the debugger, because it adds
exception handlers to executing code, which makes the Mono runtime think an
exception is handled when logically it's not (although technically it is).
The consequence is that the IDEs will only be notified when we re-throw the
exception after catching it, making it impossible for the IDEs to stop when
the exception is thrown (they will instead stop when we re-throw the
exception).
So disable managed exception marshaling (unless the user changed the default
behavior) when a debugger is attached.
This is the same behavior as Xamarin.Android.
https://bugzilla.xamarin.com/show_bug.cgi?id=45116
Setting this option prints this to stdout:
> Mono Warning: option gen-compact-seq-points is deprecated.
and it's ignored, so just don't set this option.
Example code:
public override void ViewDidLoad ()
{
throw new Exception ("USELESS");
}
Current output with unhandled exceptions that are marshaled:
Unhandled Exception:
System.Exception: USELESS
at (wrapper managed-to-native) AppKit.NSApplication:NSApplicationMain (int,string[])
at AppKit.NSApplication.Main (System.String[] args) [0x00041] in /work/maccore/master/xamarin-macios/src/AppKit/NSApplication.cs:98
at UselessExceptions.MainClass.Main (System.String[] args) [0x00007] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/Main.cs:10
[ERROR] FATAL UNHANDLED EXCEPTION: UselessExceptions.CustomException: USELESS
at (wrapper managed-to-native) AppKit.NSApplication:NSApplicationMain (int,string[])
at AppKit.NSApplication.Main (System.String[] args) [0x00041] in /work/maccore/master/xamarin-macios/src/AppKit/NSApplication.cs:98
at UselessExceptions.MainClass.Main (System.String[] args) [0x00007] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/Main.cs:10
Note how the managed frame where the exception was thrown does not show up.
This is because we technically catch exceptions when marshaling them, and then
later we call mono_raise_exception to raise the same exception. Unfortunately
that leads to overwriting the initial stack trace, and we end up with a stack
trace that does not include the location where the original exception was
thrown.
So instead of calling mono_raise_exception to rethrow the same exception, we
use ExceptionDispatchInfo, which is able to capture the stack trace for both
the original exception and the new exception, producing the following output:
System.Exception: USELESS
at UselessExceptions.ViewController.ViewDidLoad () [0x0000c] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/ViewController.cs:27
--- End of stack trace from previous location where exception was thrown ---
at (wrapper managed-to-native) AppKit.NSApplication:NSApplicationMain (int,string[])
at AppKit.NSApplication.Main (System.String[] args) [0x00041] in /work/maccore/master/xamarin-macios/src/AppKit/NSApplication.cs:98
at UselessExceptions.MainClass.Main (System.String[] args) [0x00007] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/Main.cs:10
[ERROR] FATAL UNHANDLED EXCEPTION: UselessExceptions.CustomException: USELESS
at UselessExceptions.ViewController.ViewDidLoad () [0x0000c] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/ViewController.cs:27
--- End of stack trace from previous location where exception was thrown ---
at (wrapper managed-to-native) AppKit.NSApplication:NSApplicationMain (int,string[])
at AppKit.NSApplication.Main (System.String[] args) [0x00041] in /work/maccore/master/xamarin-macios/src/AppKit/NSApplication.cs:98
at UselessExceptions.MainClass.Main (System.String[] args) [0x00007] in /Users/rolf/Downloads/filed-bug-test-cases-master/Xamarin/bxc45742/Main.cs:10
Incidently this is how Xamarin.Android does it [1].
[1]: 9387f2fe16https://bugzilla.xamarin.com/show_bug.cgi?id=45742
* [tests] Remove Classic SDK tests.
* Remove XI/Classic support.
This also means we can remove support for the legacy registrars.
* [monotouch-test] Remove legacy registrar tests.
* [tests/mtouch] Remove Classic tests (and legacy registrar logic).
* [tests/scripted] Fix tests to reference Xamarin.iOS.dll.
At startup we send a request to all the IP addresses we have,
so we must make sure to not get confused if we get responses
from more than one of them.
This fix also requires an updated mlaunch.
https://bugzilla.xamarin.com/show_bug.cgi?id=44568
We've used the dynamic registrar for years now when selecting the IL registrar
(and I've never heard about anybody doing it), so just remove all the related
(and in fact dead) code.
* [runtime] Use printf on watchOS, NSLog doesn't shown up (by default).
* [runtime] Use a wrapper function for logging.
So that we can chose between printf and NSLog at runtime,
depending on where we're running.
The watchOS device has limited networking support; in particular
it does not allow inbound/output network connections using 'bind'
(kernel-level sandbox restrictions).
This means that we can't use BSD sockets to connect to the debugger
in the IDE on the desktop. Instead we create an http tunnel that
knows how to convert socket send/recv data into http requests on
both sides.
https://bugzilla.xamarin.com/show_bug.cgi?id=41554
Launching a watchOS extension on device with the managed debugger
attached is slow, which means that the launch watchdog will kick in
and kill the app before it has launched.
So we attach the native debugger as well, which prevents the launch
watchdog from killing the app. Incidentally it also makes watchOS
not background the app.
We're using private API to determine whether a native debugger is
attached, but it's only in debug code, and as such would not be
included in release builds for customer apps. Also the code is
currently limited to watchOS since it's not needed on other
platforms for now.
The watchOS device has limited networking support; in particular
it does not allow inbound/output network connections using 'bind'
(kernel-level sandbox restrictions).
This means that we can't use BSD sockets to connect to the debugger
in the IDE on the desktop. Instead we create an http tunnel that
knows how to convert socket send/recv data into http requests on
both sides.
https://bugzilla.xamarin.com/show_bug.cgi?id=41554
Launching a watchOS extension on device with the managed debugger
attached is slow, which means that the launch watchdog will kick in
and kill the app before it has launched.
So we attach the native debugger as well, which prevents the launch
watchdog from killing the app. Incidentally it also makes watchOS
not background the app.
We're using private API to determine whether a native debugger is
attached, but it's only in debug code, and as such would not be
included in release builds for customer apps. Also the code is
currently limited to watchOS since it's not needed on other
platforms for now.
64-bit iOS 9+ devices have a pagesize of 16kb (even when running 32-bit apps).
This means all executable code must have sectors aligned to 16kb when running
on such devices.
The native linker uses the min iOS version to determine the sector alignment;
if min iOS version is < 8.0, then the sector alignment is 4k, otherwise 16k.
This means that when we create our 32-bit dylibs we must link them with a min
iOS version of 8.0, or they won't work on a 64-bit device in a 32-bit app (bug
So change our makefiles to link armv7[s] dylibs with min iOS version 8.0.
[1] https://bugzilla.xamarin.com/show_bug.cgi?id=43102
Runtime
- Added support for Vector2d, Vector3d and Vector4d SIMD types
- Added support for GKBox, GKQuad and GKTriangle SIMD types
GameplayKit
- Apple introduced GKQuadTreeNode and GKQuadTree in Xcode 7.1 and
removed those types in a later Xcode (7.2?) but we kept them around.
Now apple introduced again both types but renamed them to GKQuadtreeNode
and GKQuadtree (lowercase t), this is a breaking change and since we do
care about binary compat we are just changing the registration name and
keeping the same managed name deprecating old methods and introducing
the new ones.
- Added tests for new SIMD types
In bug #43592 the following occurs:
* App calls an API that takes a block.
* We create a stack-based ObjC block based on the delegate the app provided.
This block has a pointer to a block description, describing the block
in question (including the signature of the block, as an ObjC-type
encoded string). We allocate a new block description for every block.
* Apple's API stores the pointer to the signature string somewhere.
* Apple calls _Block_copy to get a heap-based block.
* We create a heap-based block, and copy the entire description into the
new heap-based block (including a copy of the signature).
* Apple returns from the API, and then we free the stack-based block
(and the descriptor, and thus the signature string in the descriptor).
* Apple uses the pointer to the signature stored previously to investigate
the signature of the block, and crashes because this signature has been
freed.
The assumption in Apple's code is that the description will never be freed,
which is true for any Xcode project (clang will always be able to create the
block description at compile-time and emit it in the binary, which means the
memory will never be freed). We could potentially do the same thing in the
static registrar, but we'd still need a solution when using the dynamic
registrar.
To fix this instead of copying the entire description structure when creating
a heap-based block from the stack-based block, we make the description ref-
counted, and just use the same description in the heap-based block.
The signature will now stay in memory until both the heap-based and stack-
based blocks have been freed, and we hope Apple doesn't have any API that
needs the signature after all the blocks for that signature have been freed.
https://bugzilla.xamarin.com/show_bug.cgi?id=43592
* [spritekit] Update for iOS 10 beta 1
* [spritekit] Add missing MarshalDirective
* [runtime] Add missing simd function signatures
- Add matrix_float2x2 and matrix_float3x3 natives types.
* [introspection] Fix ApiSignatureTest
- We now avoid checking {?=[X]} encoded signatures.
- We now calculate the right size for matrix_float2x2, matrix_float3x3 and matrix_float4x4.
- Fix SKWarpGeometryGrid wrong bindings revealed by tests (IntPtr instead of Vector2).
* [monotouch-test] Add SKUniform and SKWarpGeometryGrid tests
The [NSURL URLByAppendingPathComponent:] will try and check if a file:///
url is point to a directory or not, and in that process it will read metadata
about the file, which strangely enough makes macOS create a new thread.
Just after calling URLByAppendingPathComponent: we will call setenv to set
MONO_REGISTRY_PATH to the calculated path.
At the same time (sometimes _exactly_ at the same time) the new thread will
read the current environment variables.
Since getenv/setenv isn't thread-safe (!!!), there's a race condition here
that may once in a while cause a crash (bug #37794).
So instead of calling URLByAppendingPathComponent: we call
URLByAppendingPathComponent:isDirectory:, thus telling the API
if the resulting path is a directory or not, which means it doesn't
have to check (and won't create any new threads).
So with this change Xamarin.Mac apps are effectively single-threaded
until we're done setting environment variables.
https://bugzilla.xamarin.com/show_bug.cgi?id=37794
The base directory and config file name is normally set automatically
when we ask Mono to execute the Main function, but in the case of extensions,
there is no Main function, so these values are not set, causing some
features (reflection-only assembly load) to not work correctly.
https://bugzilla.xamarin.com/show_bug.cgi?id=42784
The base directory and config file name is normally set automatically
when we ask Mono to execute the Main function, but in the case of extensions,
there is no Main function, so these values are not set, causing some
features (reflection-only assembly load) to not work correctly.
https://bugzilla.xamarin.com/show_bug.cgi?id=42784
* [XM] Dynamically look up NSExtensionMain to unbreak pre-10.10 XM apps
- So having a simple reference to NSExtensionMain is enough for /usr/lib/libextension.dylib to get added as a reference to libxammac.dylib
- If you have a reference to /usr/lib/libextension.dylib then any XM app (including our tests) bomb out on startup. This is suboptimal.
- So dlopen/dysm our way to victory.
* Fix spacing
Support binding NSObjects as IntPtr. This is usually not
a problem, because when we fetch the ObjC signature for a
method, we usually get the signature as exported by us,
(in which case a parameter bound as 'IntPtr' would be treated
as 'void *' in the dynamic registrar) *except* when the
selector corresponds with a protocol the type implements,
in which case we get the signature as defined in the protocol.
https://bugzilla.xamarin.com/show_bug.cgi?id=41132
* Restore the mono_jit_thread_attach signature to how it's in
the latest stable.
* Don't use the new attach/detach methods when using the dynamic
mono runtime, since the methods aren't available in any released
version of mono. Also XM doesn't support coop yet, so it's not
needed.
* And finally don't use the new attach/detach methods for anything
that isn't building using mono/master (iow only watchOS). This
should be changed once the rest of the products starts using
mono/master again.
Rolf Bjarne Kvinge
Sebastien Pouliot
Vincent Dondain
Alex Soto
Chris Hamons
Sebastien Pouliot
Zoltan Varga