TL&DR: This PR
1. Removes the creation of the `.dSYM` based on `Debug Information` [1]
2. Adds dSYM support to XM msbuild (now shared with XI implementation)
3. Archive the `.dSYM` directories (plural) properly, e.g.
```
msbuild -p:Configuration=Release -p:ArchiveOnBuild=true
```
Why ? The long story...
Historically `.dSYM` for Xamarin.Mac have not been very useful, largely
because (most of) the code is JITed so not much is known before runtime.
So they were simply not generated during the builds...
However AOT options were added to Xamarin.Mac, making them potentially
more useful. Also symbols from `libmono` and other native libraries /
frameworks can prove useful when diagnosing application crashes.
Unsurprisingly developers looking to get symbols eventually found _a way_
[1] to get a `.dSYM` for their applications - but it was not quite
correct because:
* setting the debug information option meant that `mmp` would be supplied with `-debug`. This disables several optimizations that are, by default, enabled for release builds. IOW generating symbols should have no effect on the executing code (but it had);
* it was produced when compiling the native launcher, so the symbols coverage was incomplete. How much depends if mono was statically or dynamically linked. However this would not cover any AOTed code nor bundled libraries or user frameworks.
* the .dSYM was produced inside the `x.app/Contents/MacOS/`, side-by-side with the native executable, which makes it part of the **signed** `.app` and also part of the created (and signed) `.pkg`. This had a large impact on the application's, disk and download, size(s). Manually (re)moving the `.dSYM` means re-signing the app and re-creating (and signing) the `.pkg` is not a good solution.
[1] https://forums.xamarin.com/discussion/139705/how-to-symbolicate-a-xam-mac-crash-log
Additional fixes
* Use `Directory.Move` instead of running the `mv` command
While the result is identical there is a cost to spawn several `mv`
processes. Doing it in parallel (might have) helped but that setup
also comes at a cost.
`Directory.Move` the four `.dylib.dSYM` of an app takes 1 ms, while
the existing code took 17 ms to do the same.
* Fix building mmptest since the DeleteDebugSymbolCommand constant is not present (nor used) anymore
* Bump to .NET 6.0.100-alpha.1.21060.3.
* Fix dotnet command line arguments.
* dotnet build: the project file must be the first argument.
* dotnet build/publish: use the documented verbosity format.
* Update version number in tests.
* [tests/introspection] Adjust introspection to cope with different library names in the new .NET version.
* [tests/link sdk] Adjust the LinkSdkRegressionTest.SpecialFolder test according to the new version of .NET 6.
* [tests/link sdk] Preserve a required method in System.Private.CoreLib to work around a bug in .NET 6.
Ref: https://github.com/dotnet/runtime/issues/46908.
* Revert "[CI][VSTS] Add the donet 6 pkg as a dependency. (#10348)"
This reverts commit 6de4e717e7.
There's no need to provision .NET 6, it's done automatically.
* [dotnet-linker] Add support for writing to the same MSBuild output items multiple times.
* Split parts of LinkerConfiguration.WriteOutputForMSBuild into a FlushOutputForMSBuild
method (the part that does the actual writing).
* Make WriteOutputForMSBuild just store the items in a dictionary.
* Add a DoneStep that runs at the very end and that writes out the MSBuild output
items.
* [dotnet-linker] Link with GSS when building for iOS/Mac Catalyst.
Add a ComputeNativeBuildFlagsStep, which computes the flags to pass to the native
compiler + native linker. This is currently a very simple implementation, but it
will become more complex as support for missing features are added.
GSS is required because of libSystem.Net.Security.Native.a:
Undefined symbols for architecture arm64:
"___gss_c_nt_hostbased_service_oid_desc", referenced from:
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_c_nt_user_name_oid_desc", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_cred_no_ci_flags_x_oid_desc", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_ntlm_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_spnego_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
l___const.NetSecurityNative_AcquireCredSpNego.gss_mech_spnego_OID_set_desc in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_accept_sec_context", referenced from:
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcquireAcceptorCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred_with_password", referenced from:
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_delete_sec_context", referenced from:
_NetSecurityNative_DeleteSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_status", referenced from:
_NetSecurityNative_DisplayMinorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_DisplayMajorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_import_name", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_indicate_mechs", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_init_sec_context", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_inquire_context", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_oid_equal", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_buffer", referenced from:
_NetSecurityNative_ReleaseGssBuffer in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_cred", referenced from:
_NetSecurityNative_ReleaseCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ReleaseName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_oid_set", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_set_cred_option", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_unwrap", referenced from:
_NetSecurityNative_Unwrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_wrap", referenced from:
_NetSecurityNative_Wrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
ld: symbol(s) not found for architecture arm64
TD&LR: This PR simplifies how we refer to user frameworks and fixes both
warnings and non-optimal (app) output.
Much longer story:
Additional testing on macOS showed some build-time warnings and an
[extra (dupe) file](a20f8aba41 (diff-54fd7d9cd5deae57f30195be0a43133eace03c1132401741a317e0ae8d5e13fdR34)).
Logs shows that we referred to the xcframework several times, where once
should have been enough.
```
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework/Universal
```
The first `/native-reference` line produced a warning like:
```
MMP warning MM2006: Native library 'Universal.xcframework' was referenced but could not be found.
```
which makes sense as the tools (both `mmp` and `mtouch`) are not, by
design, aware of (unresolved) xcframeworks.
Removing `{NativeReference}` from `Xamarin.Mac.Common.targets` (and
`Xamarin.iOS.Common.targets`) as it has already been processed by
`_ExpandNativeReferences` solves this.
The other part of the issue (next two lines) is because `msbuild` does
not track changes to directories like it does for files - and the
workaround (in `_ExpandNativeReferences`) had to be copied in other
places (both XI and XM `_CompileToNative`) and that was not enough (and
would eventually need to be duplicated again and again).
This could lead to duplicate entries (i msbuild logs) like
```
NativeReferences
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework/Univeral
```
which maps to our extra entries.
In order to simplify things we make the `_ExpandNativeReferences` resolve
the full path to the library name (not the `.framework` directory) which
simplifies both `_CompileToNative` and ensure a single way (at least for
`msbuild`) to provide this data to the tools (`mmp` and `mtouch`).
Using a file, instead of a directory, is also more consistent for the
existing `-framework` option, e.g. we provide the names like:
```
--framework=CoreLocation
--framework=ModelIO
```
So adding a full path that include the name is more appropriate, e.g.
``` --framework=/Users/poupou/git/master/xamarin-macios/tests/xharness/tmp-test-dir/xcframework-test760/bin/AnyCPU/Debug/bindings-xcframework-test.resources/XTest.xcframework/ios-i386_x86_64-simulator/XTest.framework/XTest
```
Finally for macOS applications it turns out we were embedding yet another
copy of the framework's library inside the `MonoBundle`, which is clearly
wrong, because of the last entry.
```
$ l bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
-rwxr-xr-x 1 poupou staff 167152 2 Dec 16:16 bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
```
The tool now checks if a provided library is inside a framework (or not)
which is a good validation to have anyway when it gets called directly,
i.e. not thru `msbuild`.
Remove the misleading title 'Device tests..' for 'Tests..'. The message
already contains the context of the test execution, it can be:
* Build
* VSTS: device tests iOS32b
* VSTS: device tests tvOS
* VSTS: device tests iOS
Removing the 'Device' word is enough to do not confuse users.
fixes: https://github.com/xamarin/maccore/issues/2358
Since 'packages' is a common dir name that is ignored. Revert the change
in the .gitignore and rename the template path since it just means a one
liner change in the entry yaml file.
fixes: https://github.com/xamarin/maccore/issues/2359
User frameworks for macOS often uses symlinks (as Xcode creates them
this way).
This cause problem cause the symlink is on the binary and we expected
the `_CodeSignature` directory to by side-by-side with the binary. This
was missing and cause exceptions when codesigning such frameworks.
A second problem happened because `mmp` use `lipo -thin` to remove
non-required architectures. However when a framework has symlinks, like:
```
├── Frameworks
│ └── Universal.framework
│ ├── Resources -> Versions/Current/Resources
│ ├── Universal -> Versions/Current/Universal
│ └── Versions
│ ├── A
│ │ ├── Resources
│ │ │ └── Info.plist
│ │ ├── Universal
│ │ └── _CodeSignature
│ │ └── CodeResources
│ └── Current -> A
```
then this actually replaced the (very small) symlink with a thin version
of the framework. Which means the original one was still _fat_ and the
whole app was now larger than the original version.
Sample used: https://github.com/spouliot/xcframework/tree/main/xamarin/xcf-mac
The GitHup url to be used to create comments in PRs is diff to the one
for comments. Use the build reason AND the changeID to identify if we
are building due to a PR and use the correct url.
fixes: https://github.com/xamarin/maccore/issues/2356
Co-authored-by: Alex Soto <alex@alexsoto.me>
The prefix of the location of the logs depends on the platform where it
was executed (else we step on them) but that was not added in the env
var use by xharness.
fixes: https://github.com/xamarin/maccore/issues/2349
* [tests] Build test-libraries for Mac Catalyst.
* [msbuild] Add support for Mac Catalyst binding projects.
* [mtouch] Allow frameworks for Mac Catalyst apps.
* [mtouch] Put frameworks in the expected location for Mac Catalyst apps.
* [msbuild] Create the Resources directory before trying to put files in it.
* [src] Generate the string constants for each framework from our list of frameworks.
This makes it less error prone to add new frameworks (one less place to add
them), and easier to add new platforms (only one place to make sure the list
of frameworks is correct).
This also revelead a few issues in our list of frameworks (missing frameworks,
frameworks that have been moved, etc.), which have been fixed.
* [mmp] Generate the library path -> namespace map for the linker to optimize away dlopen statements.
* Fix indentation to be just as broken as the surrounding indentation.
* Remove unused Cecil reference.
* Fix c&p oversight.
* Adjust more Mac Catalyst API.
* CFNetwork is its own top-level framework.
Rework the code to generate an API comparison and a generator comparison: we
now clone xamarin-macios into a different directory, checks out the hash we
want to compare against, build the whole thing, and then compare the results.
This is somewhat slower than the previous implementation (which would checkout
the previous hash and build that on top of the existing build), but since we
don't have to build mono, it shouldn't make too much of a difference (the
biggest chunk of our build is building src/, and we had to do that anyway with
the previous implementation as well).
The upside is that it should be much more reliable.
Fixes https://github.com/xamarin/maccore/issues/2034.
Fixes https://github.com/xamarin/maccore/issues/1467.
Fixes https://github.com/xamarin/maccore/issues/1059.
Maybe https://github.com/xamarin/maccore/issues/1173 too.
* Install the Mac Catalyst versions of the mono libraries and BCL.
* The BCL is the same as the one for Xamarin.iOS, which means it has to be post-processed a bit to work with a Xamarin.MacCatalyst.dll
* Build our runtime for Mac Catalyst.
* Build a Xamarin.MacCatalyst.dll with the Mac Catalyst API (it compiles, but I haven't looked at the API surface at all). This PR assumes we're going to have a new TargetFrameworkIdentifier for Mac Catalyst, but a final decision has not been made (see https://github.com/dotnet/runtime/issues/44882), so this may change.
* Build a Xamarin.iOS.dll that contains type forwarders to Mac Catalyst for all the types that exist in both Mac Catalyst and Xamarin.iOS.
* Add support to xharness for running introspection on Mac Catalyst (there are a lot of failures because the API surface is wrong)
* Add support to our msbuild tasks and mtouch for building Mac Catalyst apps. This basically comes down to adding a new case in numerous places to either do things the iOS way or the macOS way, depending on each case.
* Add a __MACCATALYST__ define (which is in addition to the __IOS__ define).
In some places we have to provide the macOS version, and in other places the
iOS version. Add a map and the corresponding code to convert between the two,
and use them when needed.
Manuel de la Pena
Sebastien Pouliot
Rolf Bjarne Kvinge
Mike Bond
Alex Soto
monojenkins
Filip Navara