According to Vlad it's not necessary to set MONO_GC_PARAMS during AOT-
compilation, since all MONO_GC_PARAMS options can be changed at runtime:
Rolf Kvinge [16:35] @vlad.brezae is this true for all the different options MONO_GC_PARAMS take: https://github.com/xamarin/xamarin-macios/pull/1546#discussion_r97318092?
Rolf Kvinge [16:36] I remember this: https://bugzilla.xamarin.com/show_bug.cgi?id=35414#c14
Rofl Kvinge [16:36] which apparently you changed here, so that it can be changed at runtime: https://bugzilla.xamarin.com/show_bug.cgi?id=35414#c27
Rolf Kvinge [16:36] but I don't know if this is true for all the options you can pass using MONO_GC_PARAMS
Vlad Brezae [16:41] yes, it should be true for all of them, that was a bug
Rolf Kvinge [16:41] ok, that's great news 😄
HFS timestamp resolution is 1 second, which means that we can't distinguish
files modified again within 1 second. This means that this test will fail more
often the faster we make mtouch, so add a forced sleep to make sure we don't
do things faster than the file system can keep track of.
Performance tests
-----------------
This is for a new watchOS extension project, built for release.
* The default (currently -O2) optimizations: 41s ( baseline ) 30.027.060 bytes ( baseline )
* All optimizations disabled (`--llvm-opt=all=`): 17s (-24s = -59%) 32.978.312 bytes (+2.951.252 = +10%)
* Optimized for size (`--llvm-opt=all=-Os`): 36s ( -5s = -12%) 28.617.408 bytes (-1.409.652 = -5%)
* Optimized for more size (`--llvm-opt=all=-Oz`): 35s ( -6s = -15%) 28.601.016 bytes (-1.426.044 = -5%)
* Optimized slightly (`--llvm-opt=all=-O1`): 35s ( -6s = -15%) 28.666.556 bytes (-1.360.504 = -5%)
* Optimized a lot (`--llvm-opt=all=-O3`): 41s ( 0s = 0%) 30.403.996 bytes (+ 376.936 = +1%)
Conclusions
-----------
* The fastest build by far (less than twice as fast) is if optimizations are
disabled, but this adds a 10% size penalty (~3 MB in this test case),
compared to the baseline, and 15% size penalty (4.3 MB) compared to -Oz.
* -Oz seems to have the best overall results: at least as fast as any other
optimized build, and the smallest app as well.
Caveats
-------
Some optimizations might not work the AOT compiled code. The resulting
binaries have not been tested.
Current mono master is versioned as `409001455` in it's updateinfo
which (technically) violates the format. Rewrite the parser to take
an arbitrary number of digits in the final slot instead of assuming
it's always going to be 3.
Now you can provision mono/master:
```
$ cat /Library/Frameworks/Mono.framework/Home/updateinfo
964ebddd-1ffe-47e7-8128-5ce17ffffb05 409001455
$ cat /Library/Frameworks/Mono.framework/Home/updateinfo | cut -d' ' -f2 | cut -c6- | awk '{print(int($0))}'
1455
```
This can happen when a parameter of the method being [Async]-ified has
the same name as one of the delegate parameter.
E.g.
* Binding code
```
delegate void MTKTextureLoaderCallback ([NullAllowed] IMTLTexture texture, [NullAllowed] NSError error);
[Async]
void FromTexture (MDLTexture texture, [NullAllowed] MTKTextureLoaderOptions options, MTKTextureLoaderCallback completionHandler);
```
* Generated code
```
public unsafe virtual Task<global::Metal.IMTLTexture> FromTextureAsync (global::ModelIO.MDLTexture texture, NSDictionary options)
{
var tcs = new TaskCompletionSource<global::Metal.IMTLTexture> ();
> FromTexture(texture, options, (texture, error) => {
if (error != null)
tcs.SetException (new NSErrorException(error));
else
tcs.SetResult (texture);
});
return tcs.Task;
}
```
* Errpr
```
MTKTextureLoader.g.cs(397,35): error CS0136: A local variable named `texture' cannot be declared in this scope because it would give a different meaning to `texture', which is already used in a `parent or current' scope to denote something else
```
Note: we are appending, not prepending, a '_' character
Because:
* we don't want to pick up a fight with GetUniqueParamName that already
prepend a '_' in some cases;
* Pre-fixing '_' does not work for parameters names starting with a `@`
Event sequence:
* mtouch is executed with the linker disabled.
* The linker pipeline copies all input assemblies (since the linker is
disabled the assemblies don't change) into the PreBuild directory. This will
keep the original timestamps of the input assemblies.
* mtouch is executed again, when none of the input assemblies changed.
* The linker pipeline will re-execute, because it will see that at least one
of the input assemblies (at least the .exe) is newer than at least one of
the assemblies in the PreBuild directory (usually a framework assembly,
because those have the original timestamp from their install location).
Fix:
Touch all the assemblies in the PreBuild directory after the linker pipeline
executes the first time. This way the second time mtouch is executed, it will
find that all assemblies in the PreBuild directory have timestamps later than
all the input assemblies, so it will load the cached linked assemblies,
instead of re-executing the linker pipeline.
* [xharness] Show test xml files in the html report.
* [xharness] Force links to xml files to show up in a new top-level window.
Jenkins shows the html report in an iframe, and links will load in that
iframe. This means that the browser renders the contents of the link as if it
was html, and rendering xml as html results in a blank page.
So set the target attribute on links to xml files, so that those xml files
open as top-level pages, and then the browser will rendering the xml like xml.
Switch to a new API introduced in Cecil added new API to get all the
attributes in an assembly, without having to traverse (and load) every type
and member.
This makes mtouch significantly faster when computing the list of referenced
assemblies, which is noticable when mtouch later determines nothing else needs
to be done (because nothing changed).
A few tests show that mtouch is now approximately twice as fast:
* A customer test project goes from 2s to 1s.
* A simple test app goes from 1.3s to 0.6.
https://bugzilla.xamarin.com/show_bug.cgi?id=49087
Fixing the compilation was not enough [1] as it would still require to implement the async variant when subclassing. That could already be done outside the bindings (no saving).
The good news is that the generated `*Async` method does not need to be `abstract` even if it calls an `abstract` method. We already have enough information (from the `abstract` signature) to generated the async wrapper method.
Ideally (too late for the current profile) the `*Async` method should not be virtual, the async wrapper does not need to be overridden, just the binding call -> XAMCORE_4_0.
reference:
https://bugzilla.xamarin.com/show_bug.cgi?id=43165
[1] it was missing a `;` - but also caused other compilation issues
Originally, the IpaPackagePath would default to something
like:
bin/Debug/iPhone/MyProjectName $(DateTime)/MyProjectName.ipa
It was recently changed to not have the timestamp, but I
seem to have forgotten to do away with the subdirectory
completely. In other words, w/o this fix, we'd get:
bin/Debug/iPhone/MyProjectName/MyProjectName.ipa
When what we *really* want is:
bin/Debug/iPhone/MyProjectName.ipa
The native linker treats object files (.o) and static libraries (.a files,
which are archives of .o files) differently.
The native linker will always include object files into the executable:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ld foo.o -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
0000000000000fe0 T _xxx
However, if the object file is inside a static library:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ar cru foo.a foo.o
$ ld foo.a -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
<no output>
This means that our testing library (libtest.a) which is a fat library of
_object files_, do not show the problems reported in bug #51548.
So:
a) I've fixed the creation of libtest.a to be a fat library of _static
libraries_. This causes the `FastDev_LinkWithTest` test to fail exactly
like in bug #51548.
b) I've made mtouch pass `-u <native symbol>` to the native linker, for every
native symbol referenced in a managed assembly, when creating a dylib.
Amazingly this seems to work fine even with symbols to Objective-C classes
(`_OBJC_CLASS_$_<class name>`).
c) This also required adding support for collecting the Objective-C names of
all managed types registered with Objective-C to the linker. The
information is already available in the static registrar, but that would
require us to make sure the static registrar is executed before compiling
dylibs, which means those two tasks won't be able to run in parallel (also
there's no guarantee we'll even run the static registrar).
https://bugzilla.xamarin.com/show_bug.cgi?id=51548
TL&DR: This is *how* it should be done and tested, it's not complete
(single, simple case) nor the most interesting case ;-)
The trick is to make sure each case is covered by tests so a mono
_bump_ won't give us a BCL that does not conform to what the linker
expect.
What's the impact ?
1. There is the expected reduction of metadata in mscorlib. Since both
methods don't call other API there's no indirect effect (removal).
--- before 2017-01-15 11:12:44.000000000 -0500
+++ after 2017-01-15 11:12:56.000000000 -0500
@@ -13166,9 +13166,6 @@
System.Void System.Security.SecurityException::.ctor(System.Runtime.Serialization.SerializationInfo,System.Runtime.Serialization.StreamingContext)
System.Void System.Security.SecurityException::.ctor(System.String)
System.Void System.Security.SecurityException::GetObjectData(System.Runtime.Serialization.SerializationInfo,System.Runtime.Serialization.StreamingContext)
-System.Boolean System.Security.SecurityManager::CheckElevatedPermissions()
-System.Security.SecurityManager
-System.Void System.Security.SecurityManager::EnsureElevatedPermissions()
System.Security.SecurityRulesAttribute
System.Security.SecurityRuleSet System.Security.SecurityRulesAttribute::m_ruleSet
System.Void System.Security.SecurityRulesAttribute::.ctor(System.Security.SecurityRuleSet)
2. There is no visible size change (even with #1) in mscorlib.dll due to
padding (compiler /filealign)
mscorlib.dll 793,600 793,600 0 0.00 %
3. there's a *very* small reduction of mscorlib.*.aotdata size
mscorlib.armv7.aotdata 717,264 717,216 -48 -0.01 %
mscorlib.arm64.aotdata 712,840 712,704 -136 -0.02 %
AOT data *.aotdata 6,460,064 6,459,880 -184 0.00 %
4. there's no change in executable size - normal as the AOT compiler has
_likely_ already doing the same optimization (before this commit)
Executable 29,270,272 29,270,272 0 0.00 %
Full comparison: https://gist.github.com/spouliot/0464c8fa3a92b6486dfd90595d9eb718
We're not storing selectors in fields anymore so this step does nothing
expect iterating over the code.
The step is still needed for `mmp` (for Xamarin.Mac.dll)
Simplify code a bit to avoid constant null checking and also take advantage of
the fact that HashSet.Add returns if the value was added or not (to avoid a
Contains check).
This allows mtouch to give better error message when something unexpected
occurs in the linker pipeline (at least for the sub-steps).
Practically it means fewer, contextless MT2001 errors. The replacements
error code are more precise, e.g.
* what was being done;
* what was being processed
and helps both diagnosing and, possibly, gives clues for workarounds
Build extensions and the container app in the same mtouch process, by storing
all the mtouch arguments when called to build extensions in a text file, and
then reloading those arguments when called to build the main app.
This is required if we want to share code between extensions and the
container.
This is a series of fixes to the dynamic libraries we build / create to remove
any unnecessary bloat (unused architectures, bitcode).
A brand new watchOS app with no changes goes from 35MB to 11MB with these
fixes (with incremental builds disabled, the app size is 10MB).
--------------------------------------------------------------------------------
* [runtime] Split list of architectures into simulator and device-specific lists.
* [runtime] Build separate dylibs for device and simulator.
Build separate dylibs for device and simulator, since we already install these
into different locations.
This makes both the simulator and device builds slightly faster (since the
respective dylibs are smaller, and less data to copy around).
For watchOS apps, this saves ~430kb.
* [runtime] Strip bitcode from dylibs. Fixes#51352.
We know that dylibs will never be shipped to the App Store, so we'll never
need them to have bitcode.
So just strip the bitcode from all our dylibs, since this makes apps with
fastdev significantly smaller (and thus much faster to upload to watch
devices).
For watchOS apps this is a very significant improvement: a branch new watchOS
app without any changes goes from 35MB to 17MB.
https://bugzilla.xamarin.com/show_bug.cgi?id=51352
* [mtouch] Fix dylib compilation to not embed full bitcode.
Facts
=====
a. The output from the AOT compiler is an assembly (.s) file.
b. Clang's assembler does not support -fembed-bitcode-marker (only -fembed-
bitcode), so when we ask clang to -fembed-bitcode-marker, the assembler
receives a -fembed-bitcode argument.
c. This means that the assembled object file does not contain the
__LLVM/__bitcode and __LLVM/__cmdline sections (it does however contain an
__LLVM/__asm section).
d. The native linker will create a bitcode assembly section if none of the
object files passed to the linker contain a __LLVM/__bitcode section and
there's an __LLVM/__asm section present.
e. The end result is that when we build to a dylib, we end up (unexpectedly,
because we ask Clang to -fembed-bitcode-marker) including both armv7k and
bitcode in the dylib, thus bloating the dylib size significantly.
Solution
========
We manually add the __LLVM/__bitcode and __LLVM/__cmdline sections to the .s
file Mono's AOT compiler generated. This way the .o file will have the magic
sections, and the linker will not include bitcode (only the bitcode marker) in
the final library.
An empty watchOS extension with incremental builds is now 6MB smaller (down
to 11MB from 17MB).
Don't create multiple writer streams for the same underlying file, instead
store the writer stream on the log instance, and return it whenever needed.
This fixes an issue where there would be multiple writer streams, which could
cause exceptions when creating more writer streams (access defined errors
because the file is already open).
This was changed in 1b1e345d72 since bmac.exe says --new-style is
deprecated.
However some users (MonoTouch.Hosting) manually call bmac (instead of going
through a binding project), and with 1b1e345d72 we changed the behavior when
--new-style was passed, so that it ended up trying to build Classic instead
(which is clearly not right for --new-style).
So make the bmac script understand --new-style again (to not break
compatibility), but don't forward the flag to bmac.exe (to avoid the
deprecation warning).
* [tests] Use the target directory from the loaded configuration.
* [xharness] Find the root directory based on xharness.exe's location (unless specified).
* [tests] Add makefile target to generate test config using the system XI.
Rolf Bjarne Kvinge
batmaninpink
Vlad Brezae
Vincent Dondain
Jeffrey Stedfast
Alan McGovern
Sebastien Pouliot
Alex Soto
Chris Hamons
Manuel de la Pena