* Change dotnet-linker to only care about whether we're actually trimming anything or not.
* Allow LinkMde/MtouchLink to not be set if TrimMode is set.
* Detect if any assemblies are linked or not by checking the global TrimMode
property + any TrimMode properties on assemblies.
Fixes https://github.com/xamarin/xamarin-macios/issues/13518.
Pass -dead_strip to the native linker like we do for legacy Xamarin:
* If there are no custom linker arguments.
* If all third-party bindings in the app has SmartLink = true (this doesn't
show up in the PR, but the logic exists for legacy Xamarin and is already
executed for .NET, the resulting Application.DeadStrip value just wasn't
taken into account).
This shrinks the app size a bot for a Hello World app:
* Before: 10.659.731 (https://gist.github.com/rolfbjarne/b5892a5c7fb8663d38e2b69f67bce90c)
* After: 9.940.240 (https://gist.github.com/rolfbjarne/8404394180fb9971bd2f1475b747c70a)
* Difference: -719.491 (-6.7 %)
We extract frameworks from third-party libraries when running the linker, and
we need to store this information on disk and the reload it after the linker
has executed, and add it to the existing MSBuild variables that keep track of
the user frameworks and dynamic libraries that have to be copied to the app
bundle.
Fixes the framework-test tests.
* Add support for Mono Components.
* Modify how we look up symbols from native libraries shipped with Mono: we keep
track of which native libraries we linked with, and depending on how we linked
to those assemblies, we look the symbols up at runtime in either the current executable
(if linking statically), or the actual library (where the P/Invoke says they're
supposed to be).
* This means that we have to propagate how libmono is linked from the MSBuild code
to the Application class so that our existing logic is able to correctly determine
which native mono lib to use.
* Modify how we list the P/Invokes we need to preserve by taking into account the
list of native libraries from Mono we have to link with (for .NET). For legacy
Xamarin, I've reverted the logic to how it was before we started adding .NET support.
Fixes https://github.com/xamarin/xamarin-macios/issues/10950.
Fixes https://github.com/xamarin/xamarin-macios/issues/11145.
Fixes https://github.com/xamarin/xamarin-macios/issues/12100.
* Add support for the interpreter everywhere.
* Add support for the AOT compiler everywhere we didn't support it before,
because the interpreter needs it (at least System.Private.CoreLib.dll must
be AOT-compiled when using the interpreter).
* Do FullAOT compilation on Mac Catalyst/ARM64 if we're not using the
interpreter, since we can't use the JIT.
* Fix monotouch-test to be green on Mac Catalyst/ARM64.
Fixes https://github.com/xamarin/xamarin-macios/issues/11724.
Fixes https://github.com/xamarin/xamarin-macios/issues/11421.
This also meant propagating how libmono is linked from the MSBuild code to the Application
class so that our existing logic is able to correctly determine which native mono
lib to use.
So that the ComputeAOTArguments can compute the llvm-path value to pass to the AOT
compiler (the llvm-path value states where the opt and llc command-line tools are,
and they're next to the AOT compiler).
* Bump to .NET 6.0.100-preview.3.21152.10
* Bump to 6.0.100-preview.3.21152.10.
* Bump to 6.0.100-preview.3.21161.7.
* Bump to .NET 6.0.100-preview.3.21161.23.
* [dotnet-linker] Bump ILink and use the supported method of getting an assembly's location.
* Bump to MSBuild.StructuredLogger 2.1.364 to get support for newer binlog versions.
* Fix build failure
Fixes:
TestHelpers/BuildEngine.cs(161,24): error CS0433: The type 'ProjectEvaluationFinishedEventArgs' exists in both 'Microsoft.Build.Framework, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' and 'StructuredLogger, Version=2.1.0.0, Culture=neutral, PublicKeyToken=d4c7181801cb6448' [/Users/builder/azdo/_work/1/s/xamarin-macios/tests/msbuild/Xamarin.MacDev.Tests/Xamarin.MacDev.Tests.csproj]
* Update parsing binlog properties.
* Be more defensive.
* [tests] Make sure the InternalsVisibleToAttribute constructor isn't linked away.
* [tests] Implement better printing of binlogs.
The latest MSBuild.StructuredLogger made some internal changes to the Message
property which confuses some of our test logic. So implement manual printing
of the log entries that we care about to make sure they conform to the
expected format by the tests (the output is mimicing what 'msbuild /v:diag
foo.binlog' would show).
Add support for parsing --setenv from bundler arguments and passing them to
dotnet-linker so that the environment variables can be baked into the app at
launch (like they would be for mtouch/mmp).
Use a private property (prefixed with underscore) for now, until we can decide
on a better/general name.
Also add a variation to xharness to build monotouch-test with CoreCLR
(building works fine, but it crashes at startup, which is expected at this
point).
This allows us to undo a workaround we made for a missing API in the linker
reference assembly (an AnnotationStore.AddPreservedMethod overload).
This also requires a change to use MessageContainer.CreateCustomErrorMessage
instead of MessageContainer.CreateErrorMessage, because apparently having a
reference assembly doesn't mean there can't be incompatible changes in it 😒.
* [dotnet-linker] Add support for writing to the same MSBuild output items multiple times.
* Split parts of LinkerConfiguration.WriteOutputForMSBuild into a FlushOutputForMSBuild
method (the part that does the actual writing).
* Make WriteOutputForMSBuild just store the items in a dictionary.
* Add a DoneStep that runs at the very end and that writes out the MSBuild output
items.
* [dotnet-linker] Link with GSS when building for iOS/Mac Catalyst.
Add a ComputeNativeBuildFlagsStep, which computes the flags to pass to the native
compiler + native linker. This is currently a very simple implementation, but it
will become more complex as support for missing features are added.
GSS is required because of libSystem.Net.Security.Native.a:
Undefined symbols for architecture arm64:
"___gss_c_nt_hostbased_service_oid_desc", referenced from:
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_c_nt_user_name_oid_desc", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_cred_no_ci_flags_x_oid_desc", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_ntlm_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_spnego_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
l___const.NetSecurityNative_AcquireCredSpNego.gss_mech_spnego_OID_set_desc in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_accept_sec_context", referenced from:
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcquireAcceptorCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred_with_password", referenced from:
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_delete_sec_context", referenced from:
_NetSecurityNative_DeleteSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_status", referenced from:
_NetSecurityNative_DisplayMinorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_DisplayMajorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_import_name", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_indicate_mechs", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_init_sec_context", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_inquire_context", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_oid_equal", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_buffer", referenced from:
_NetSecurityNative_ReleaseGssBuffer in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_cred", referenced from:
_NetSecurityNative_ReleaseCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ReleaseName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_oid_set", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_set_cred_option", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_unwrap", referenced from:
_NetSecurityNative_Unwrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_wrap", referenced from:
_NetSecurityNative_Wrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
ld: symbol(s) not found for architecture arm64
Improve error reporting in dotnet-linker by not requiring an instance of a LinkerConfiguration
to report errors. This is accomplished by making the LinkerConfiguration.Report method
a static method.
Otherwise reporting errors before we've successfully created a LinkerConfiguration
turns out to be troublesome (we end up throwing another exception, usually a NullReferenceException,
which is just confusing).
* Continue using our own error handling logic, and print our problems to stderr.
* Also use the linker's messaging facilities to report a more generic error,
in case stderr doesn't show up for some reason.
The .NET linker comes with a way to remove attributes (by passing '--link-attributes
some.xml' as a command-line argument), but this has a few problems:
* We'd need to figure out which attributes to remove before running the linker,
but the code to figure out which optimizations have been enabled (and which attributes
should be removed) is in our custom linker code. We'd need to refactor a big chunk
of code to move this logic out of our custom linker code.
* We need to keep the removed attributes around, because the static registrar needs
them. Our custom linker logic is not notified for removed attributes, which means
we'd need to store all attributes for the attribute types we're interested in (as
opposed to this solution, where we only store attributes that are actually removed).
* The attributes we want removed may contain references to types we don't want
linked away. If we ask the linker to remove those attributes, then the types may
be linked away as well, and there's no good way around this.
The end result is that a custom step is the best solution for now.
Fixes these monotouch-test tests when enabling all optimizations:
Xamarin.BindingTests.ProtocolTest
[FAIL] OnlyProtocol : [Protocol] IP1
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.OnlyProtocol() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 47
[FAIL] ProtocolWithBaseType : [Protocol] IP2
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.ProtocolWithBaseType() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 79
[FAIL] ProtocolWithBaseTypeAndModel : [Protocol] IP3
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.ProtocolWithBaseTypeAndModel() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 115
Fixes these linkall tests:
Linker.Shared.OptimizeGeneratedCodeTest
[FAIL] IsARM64CallingConvention : optimized: no ldsfld instruction
Expected: 0
But was: 1
at Linker.Shared.BaseOptimizeGeneratedCodeTest.IsARM64CallingConvention() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 527
[FAIL] SetupBlockPerfTest : At least 6x speedup
Expected: greater than 6
But was: 1.0876440665344851d
at Linker.Shared.BaseOptimizeGeneratedCodeTest.SetupBlockPerfTest() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 120
And linkall is now green for .NET/Debug.
* [dotnet] Pass the Optimize flags from the extra bundler arguments to the linker configuration.
Also call Application.InitializeCommon to initialize the application instance. The
important part here is that InitializeCommon calls Optimizations.Initialize to compute
the default optimizations. It also calls Set*ExceptionMode and sets the default EnableCoopGC
value (so we don't need to call/set those anymore), and it does a few other initialization
tasks which we don't need yet, but eventually will.
And finally remember to parse the bundler arguments before using them in the dotnet
build logic. How did this not cause problems before? 🤦
* [tests] Set the verbosity using the additional args instead of an internal variable.
The internal _BundlerVerbosity variable is overwritten now (with the verbosity
value from the additional args).
* [xharness] Disable tvOS generation for the introspection/.NET test, it incorrect and needs fixing.
The Assembly.IsFrameworkAssembly property is used in two places:
* In Driver.IsBoundAssembly to return early when determining if an assembly has any NSObject subclasses: c1c5b9aac6/tools/mtouch/mtouch.cs (L1155-L1168)
* In Assembly.ExtractNativeLinkInfo to return early when looking for assemblies with LinkWith attributes: c1c5b9aac6/tools/common/Assembly.cs (L150-L154)
In both cases this definition of framework assembly works today and seems likely to work in the future as well.
I also went through and looked at all the usages of Profile.IsSdkAssembly, and it's used to:
* Decide which assemblies are selected for "link sdk"
* Decide which assemblies are considered an 'sdk' assembly for creating a user framework of all the sdk assemblies
* Bail out early when deciding whether:
* An assembly references the product assembly (Xamarin.iOS.dll, etc.)
* An assembly can contain references to UIWebView
* An assembly can contain user resources
* An assembly is a binding project / has third-party native resources
* An assembly needs the dynamic registrar
* An assembly has FieldAttributes whose native fields must be preserved by the native linker
In all cases our .NET definition of 'SDK' seems to work both for now and in the future.
There are also a few usages which does not apply to .NET, so I've ignored them:
* When looking for a few BCL APIs that must be preserved (MobileApplyPreserveAttribute.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
* When linking away parameter names (MonoTouchMarkStep.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
This means:
* Move the parts of the ApplyPreserveAttribute step that we don't need for.NET
into a new MobileApplyPreserveAttribute step, and have mtouch and mmp use
that step instead of the ApplyPreserveAttribute step.
* Copy ApplyPreserveAttributeBase into dotnet-linker from the upstream tuner
source (with minor modifications) so that our ApplyPreserveAttribute step
compiles.
* Create a DotNetSubStepDispatcher class that we're going to use as our
substep dispatcher, create an instance of it and insert it into the list of
linker steps.
* Also a workaround for the lack of LinkContext.GetAssemblies (): add a step
that collects all the assemblies and stores them in a list, so that we can
have our own GetAssemblies implementation.
I filed a linker issue to see if we can get LinkContext.GetAssemblies ()
exposed to us: https://github.com/mono/linker/issues/1455.
Fixes this startup crash with the linkall test:
2020-08-26 19:47:10.219697+0200 link all[32709:6065783] Xamarin.iOS: Fatal error: failed to load the method 'ObjCRuntime.Runtime.Initialize'.
=================================================================
Native Crash Reporting
=================================================================
Got a abrt while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x104007b0e - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
0x103fb4437 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_handle_native_crash
0x104007365 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
0x7fff51c005fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_platform.dylib : _sigtramp
0x0 - Unknown
0x7fff51af0b7c - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
0x103daad98 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_assertion_message
0x103dade77 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_initialize
0x103dbf80b - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_main
0x103cd2f0d - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/link all : main
0x7fff51a231fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdyld.dylib : start
This is a partial/modified port of the initial linker support (bc88790201)
Co-authored-by: Sebastien Pouliot <sebastien@xamarin.com>
* [dotnet] Pass exception marshaling options to the linker configuration, and pass it along to the Application instance.
* [dotnet] Write the selected exception marshaling modes to the generated main file.
* [dotnet-linker] Set the default cooperative GC mode.
The code to select the default exception marshalling mode needs it.
Add a Registrar step that is responsible for dealing with the registrar during the
build. Currently only supports the dynamic and partial static registrar, the full
static registrar support will come later.
Refactor Application creation to happen earlier, and to split out the cache
creation. This way we can create the Application instance before processing
the configuration, and as we process any configuration we can set properties
on the Application instance.
* Make Driver.Verbosity the single place where we store the verbosity level.
* Respect any default verbosity by adding to the existing verbosity instead of
setting it directly.
* There's no need to set ErrorHelper.Verbosity, the Driver.Verbosity setter
already does it.
This means that for the .NET linker code we'll treat
~/.xamarin-bundler-verbosity like we treat ~/.mtouch-verbosity for mtouch, and
parse it to set the verbosity.
Rolf Bjarne Kvinge
Sebastien Pouliot