* [static registrar] Optimize creation of delegates for blocks.
Optimize creation of delegates for blocks so that it doesn't require the
dynamic registrar.
This is done by getting the metadata token for the Create method that creates
the delegate, and embed that metadata token in the generated code from the
static registrar.
Also add tests, since this scenario was not covered by tests already.
* [mmptest] Fix test after recent changes.
* [test-libraries] Avoid duplicate symbols.
* [tests] Update according to changes.
Assemblies that satisfy all of these conditions:
* Are not in the container project.
* Are in multiple app extensions.
can (and should) be bundled in the container app.
Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=58873
* [tests] Don't run mmp regression tests in parallel. Fixes maccore #645.
For some reason vstool becomes confused and starts erroring out if multiple
vstools are running at the same time.
So work around this by executing the MMP regression tests serialized.
Fixes https://github.com/xamarin/maccore/issues/645.
* [xharness] Upload the logs from the mmp regression tests.
* [xharness] Improve html report by formatting newlines and tabs in test failure messages using corresponding html tags.
* [xharness] Improve nunit test reporting in html report by creating a list of failures.
- https://github.com/xamarin/xamarin-macios/issues/3367
- App Store will now fail builds if you add in a 32-bit dylib
- If you are a 32-bit app you don't need the 64-bit part of your fat
dylib anyway
- Add --optimize=-trim-architectures to allow customization of behavior, as not everyone
uses app store
In addition, while writing tests for this is was noticed that mmp tests did not "really" run Release configuration correctly in most cases. Fixing this turned out to be a bit of a pain, but necessary to correctly test this (and other things).
- Turns out that /p:configuration:debug is not sufficient to tell mmp to
do the right thing
- That, in most projects, sets the DebugSymbols property, which really
is what is checked.
- However, two of our projects did not have that, so we always did
release mmp work.
- Removed configuration property for tests and added real "Release"
configuration option
This fixes a startup crash in code shared app extensions due to having the
wrong value set in the runtime (the dynamic registrar was removed, but the
executable didn't know it).
This can be reproduced by running monotouch-test with all optimizations in
Debug mode (because some of the exception marshalling tests are only enabled
in debug mode), so add such a configuration to xharness. To avoid bloating PR
builds, this configuration is only enabled when running all tests (either
manually selecting all tests for a PR, or on Wrench, where everything is
always tested).
The ProtocolTest test is located in the bindings-test assembly, and it
requires certain conditional compilation defines to be set in order to behave
properly.
With this fix we correctly set these defines when cloning the bindings-test
project (in addition to when we set the defines for the main project, which is
either monotouch-test or linkall for this test).
Fixes https://github.com/xamarin/maccore/issues/655.
* [tests][macos] Fix AuthenticodeDeformatterTest.VerifySignedAssembly failure on modern. Fixes 3207
Modern does not, by default/design, ship a machine.config file. This
means it only knowns about the default crypto shipped with .NET.
That's a problem for MD2 for which some old certificates might still
exists in the computer store, like our wrench bots.
That's generally not a problem since XM apps delegate trust to macOS
except for one case: Authenticode.
So verifying an authenticode signature can end up, thru X509Chain,
loading a certificate using MD2 without knowing how to create the
digest algorithm - and fail.
The fix is to register the algorithm manually (if not found).
https://github.com/xamarin/xamarin-macios/issues/3207
* [tests] Only call CryptoConfig on XM modern builds and add a reference to Mono.Security.dll on such projects
This fixes an issue where multiple copies of the GuiUnit project would have
the same output directory, causing random problems when building those
projects in parallel.
The [Protocol] attributes the test checks for are not linked away in
monotouch-test (because monotouch-test only links SDK assemblies), but they
are linked away in link all, so adjust the test accordingly.
* [registrar] Fix resolving linked away generic types. Fixes#3523.
Fixes#3523.
* [tests] Use a link all test instead of mtouch test.
It's much faster, since we're already building link all everywhere.
* [tests] Use the static registrar in all linkall simulator configurations.
When code sharing is enabled, only the container app/target will have a
LinkContext.
So make sure the static registrar finds that link context when it needs
information from the linker.
https://github.com/xamarin/xamarin-macios/issues/3514
* [tests] Automatically rebuild 'test.config' if it can't be found.
This condition usually happens when I 'git clean -xfd' in tests/ and then want
to run tests from xharness.
The manual step is to run "make" in tests/, but it's better if we can do it automatically.
* [msbuild] Add file to fix build.
We want to avoid separate `mscorlib.dll` assemblies for 32/64 bits so
the architecture specific code for `n[u]int` and `nfloat` must be
preserved in both cases.
Because a single, slightly larger, assembly is much smaller than two
(slightly smaller) ones.
This is not a common situation since the extraneous preserved API are
often used in the application (or 3rd party code) so, in most cases,
a single `mscorlib.dll` was already used.
This merely close the gap for some cases, like our `link all` application
where this happened.
* [mtouch] Rename temporary directories so the order assemblies move is clearer.
I implemented this myself, but I can never remember in which order assemblies
go from one directory to another during the build.
So number these temporary directories, so that even the most forgetful minds
can understand without having to remember anything.
* [tests] Update test according to temporary directory name change.
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
Implement support for optimizing away the dynamic registrar.
* Add a new property (Runtime.DynamicRegistrationSupported) that indicates at
runtime whether dynamic registration is available.
* Check this new property whenever we need dynamic registration.
* Add an optimization to mtouch/mmp that detects whether dynamic registration
is required, and if not, changes Runtime.DynamicRegistrationSupported to
return false.
* Add an optimization to mtouch/mmp that inlines
Runtime.DynamicRegistrationSupported as a constant value.
The end result is that the linker will link away the dynamic registrar if
mtouch/mmp detects that it's not needed.
Benchmark
---------
I've compared the size of entire apps built for device:
|test | Before | After | Diff | % |
|:-----------------------------|-------:|-------:|-------:|------:|
|[monotouch-test/Release][1] | 95.7mb | 95.0mb | -680kb | -0.7% |
|[link sdk/Release][2] | 21.2mb | 20.9mb | -245kb | -1.2% |
|[minimalistic app/Release][3] | 4.58mb | 4.32mb | -259kb | -5.7% |
[1]: https://gist.github.com/rolfbjarne/3871e36d2de8db1a8eee1d9f9276d3d2#monotouch-test-release-wall-optimizations-enabled
[2]: https://gist.github.com/rolfbjarne/3871e36d2de8db1a8eee1d9f9276d3d2#link-sdk
[3]: https://gist.github.com/rolfbjarne/3871e36d2de8db1a8eee1d9f9276d3d2#minimalistic-app
Rolf Bjarne Kvinge
Chris Hamons
Sebastien Pouliot
Vincent Dondain
Manuel de la Pena
Alex Soto