* [monotouch-test] Use a more descriptive label for items added to the keychain in KeyTest.
* [monotouch-test] Use a more descriptive label for some items in CertificateTest.
Also make generated keys non-permanent so that they're not added to the keychain.
* [monotouch-test] Clean up a few items added to the keychain by CertificateTest.
* [msbuild] Unify how CFBundleName is calculated.
Previously, CFBundleName would default to:
* iOS: CFBundleDisplayName (if set), otherwise the app bundle name.
* all other platforms: the app bundle name.
Now unify the logic so that we have the same behavior on all platforms.
This is a breaking change under the following conditions:
* Building for iOS
* CFBundleName is not set in the Info.plist
* CFBundleDisplayName is set in the Info.plist
* CFBundleDisplayName from the Info.plist is different from AssemblyName in
the csproj (which is the value used to calculate the app bundle name).
The fix would be to:
* Set CFBundleName in the Info.plist to the desired value.
This change works in previous versions of Xamarin.iOS as well.
* Update tests.
* We assigned the same value to DetectedBundleId in every branch, so instead
assign it once at the beginning.
* We assigned the same value to DetectedAppId to most branches, so instead
assign that value at the beginning as a default value.
This change perse does not do other thing than removing the use of
virtuanenv and remove the installation of python-magic.
We have taken this approach because the images that will be used to provision the CI
bots will have python-magic installed by default. We want to do this
because:
1. CI should not be modifying the machine, we do not install any pkg.
2. python-magic should be present, if not, we will get an import error
and we will know how to fix it (re-image).
This way the bin directory doesn't unnecessarily get an updated timestamp, so
that some build targets in the dotnet/ directory don't rebuild unnecessarily.
Using `Path.Combine` with a full qualified path for the 2nd argument will
return that 2nd argument (ignoring the first one).
That meant the `StampPath` was pointing to the actual files that were
just signed - overwriting them with a 0-length data (empty).
The solution is to make the 2nd argument relative, starting after
`.app/` so `Path.Combine` works as expected (being relative) while
allowing signing multiple files that have the same name (in different
directories).
ref: https://github.com/xamarin/xamarin-macios/issues/10445
Today both `mtouch` and `mmp` are copying the entire `.framework`
directories inside the `.app\[Contents\]Frameworks\` directory.
However not everything in a framework is required at runtime. The most
common unrequired files would be headers (`Headers/*.h`) and modules
(`Modules/*`).
Looking at Xcode build output we can see something like:
```
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -exclude Headers -exclude PrivateHeaders -exclude Modules -exclude \*.tbd -bitcode-strip replace-with-marker -bitcode-strip-tool
```
which excludes a few more, less common, files.
This _builtin_ command is not available externally (for us to re-use)
but it hints that Xcode is likely using `rsync` to avoid copying part of
the files.
Note: the builtin command also _likely_ calls `bitcode_strip` too (or has
similar code embedded) and `mtouch` already does so too
There's a cost to spawning an external process, like `rsync`, which we
avoid by having our own file copier, which clones files (almost zero
cost). That does not support excluding files, but deleting files is also
very cheap. Testing shows copying a framework to be less than 1 ms, even
with with extra deletion step.
* Tweak `GetRealPath` to optionally not to warn if the path does not exists
since, in this case, it's a check we want to do after resolving the path
This fixes several (5) MTouch tests looking for specific (and no extra)
warnings
```
Unable to canonicalize the path '/Users/builder/azdo/_work/2/s/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory195/testApp.app/Frameworks/Mono.framework/CVS': No such file or directory (2).
```
because:
> If the file to be deleted does not exist, no exception is thrown.
https://docs.microsoft.com/en-us/dotnet/api/system.io.file.delete?view=net-5.0
and yes this is different from `Directory.Delete` and PR https://github.com/xamarin/xamarin-macios/pull/10441
* Fix MT0015 test failure
The MT0015 test creates a directory where a file is expected.
That's fine except the code for handling this was a bit weird. It
worked because of a `TryDelete` on the path, which avoided the
`UnauthorizedAccessException` when `File.Delete` is used on a path.
Then later there's a `Directory.Exists` check that would throw...
The code now does the `Directory.Exists` first and only call
`File.Delete` if it returns false. The throwing of the exception
is kept since the code (and test) are already present (and this
minimize changes and chance of other surprises)
* [dotnet-linker] Add the RemoveUserResources sub step.
This comes with a few changes to the RemoveUserResources sub step as well:
* Bail out earlier if we're in the simulator (no need to do any processing at
all - we know at the very beginning if we're building for the simulator).
* Do a positive simulator check, instead of a negative device check (because
Mac[Catalyst] are neither devices - so they pass the negative device check).
* Remove all the conditional mtouch/mmp code, and figure out at runtime which
resource prefixes we need to check for.
It was also necessary to change where steps are added to the pipeline: we have
to remove resources before the OutputStep, but at the same time we have to do
it after the ExtractBindingLibraries step, otherwise the
ExtractBindingLibraries step won't find any binding libraries to extract. So
move the ExtractBindingLibraries, LoadNonSkippedAssemblies and
ListExportedSymbols to before the OutputStep (to keep their internal order),
and then add the RemoveUserResources after those.
This fixes the following link sdk/link all test when running on device:
BundledResources.ResourcesTest
[FAIL] Bundled : No resources
Expected: 0
But was: 2
* [tests] Update the BuildInterdependentBindingProjects test now that we're removing resources.
This allows us to undo a workaround we made for a missing API in the linker
reference assembly (an AnnotationStore.AddPreservedMethod overload).
This also requires a change to use MessageContainer.CreateCustomErrorMessage
instead of MessageContainer.CreateErrorMessage, because apparently having a
reference assembly doesn't mean there can't be incompatible changes in it 😒.
Fixes this warning when running xharness:
Unknown file: fsharplibrary.fsproj (extension: .fsproj). There might be a default inclusion behavior for this file.
* Fix system version checks to work properly on Mac Catalyst (which uses the macOS
version as its system version).
* Add the framework-specific defines to the build for monotouch-test.csproj (using
the generated response files), this way we can use them in the tests.
* Sprinkle conditionals in numerous places - I tried using either framework-specific
or XAMCORE_3_0 whenever that made since (instead of using Mac Catalyst as a condition).
* Updated a few tests to use non-deprecated API (because the deprecated API often
doesn't exist in Mac Catalyst).
* Also a few minor API fixes to make any corresponding tests compile.
TL&DR:
This effectively change nothing - but prevents (warn) if both options
contradict themselves.
Long Story....
So we have two ways to control the codesign's `--timetamp` option but
they both ignore each other so we can end up with the option being
set more than once at build time.
`DisableTimestamp` was the original one. It was meant for iOS (and
derivative OS) and disable the option (which requires the network)
for simulator or debug builds. IOW we _wanted_ timestamps when doing
release builds for devices.
```
DisableTimestamp="$(_CodesignDisableTimestamp)"
```
```
<_CodesignDisableTimestamp>False</_CodesignDisableTimestamp>
<_CodesignDisableTimestamp Condition="'$(_SdkIsSimulator)' == 'true' Or '$(_BundlerDebug)' == 'true'">True</_CodesignDisableTimestamp>
```
Now disabling the timestamp did not mean it was enabled. We did not ask
for a timestamp, leaving it to the default which from `man codesign`
means:
> If this option is not given at all, a system-specific default behavior is invoked.
> This may result in some but not all code signatures being timestamped.
Then `UseSecureTimestamp` was added for macOS builds. If hardening is
enabled then a secure timestamp is required.
`msbuild/Xamarin.Mac.Tasks/Xamarin.Mac.Common.targets` `_CodesignAppBundle`
```
UseSecureTimestamp="$(UseHardenedRuntime)"
```
However it's also exposed for iOS (shared target) in
`msbuild/Xamarin.Shared/Xamarin.Shared.targets` `__CodesignNativeLibraries`
but it would always be `false` in that case.
Adding this option means there's now always a `--timestamp` option given,
either to enable it (no URL so it means using Apple's server) or to
disable it (`=none`) but since it's controlled by `UseHardenedRuntime`,
which is macOS only, then iOS device builds are never timestamped.
An alternative would be to have `UseSecureTimestamp` as a macOS-only
option - but that would change how we currently sign the iOS applications
and I'd rather not change things that are known to work.
Code signing for Mac Catalyst is very much like code signing for macOS, except that:
* If a provisioning profile is required, then we must find a code signing certificate.
* Provide a code signing key even if we don't need a code signing certificate.
This makes the tests in monotouch-test that require a correctly signed app pass.
According to Apple's documentation, an app linked with iOS 12 or later must
enable a specific entitlement, otherwise calls to CNCopyCurrentNetworkInfo
will always return NULL.
All the other *NoProxy tests use the Microsoft uri, which seems to be the uri
that's not supposed to need a proxy (according to what I understand from the
code) - in other words, it looks like this was a c&p error.
Fixes this test failure when running on device:
[FAIL] TestPACParsingAsyncNoProxy : Expected: None
But was: HTTPS
at MonoTouchFixtures.CoreFoundation.ProxyTest.TestPACParsingAsyncNoProxy () [0x000fa] in /Users/rolf/work/maccore/onedotnet/xamarin-macios/tests/monotouch-test/CoreFoundation/ProxyTest.cs:238
* [monotouch-test] Adjust ColorTest.CreateByMatchingToColorSpace to be Xcode 11+ only.
This test uses CGColor.CreateSrgb, which is Xcode 11+ only.
* [monotouch-test] Restrict AvoidOccluderConstraintTest to iOS 11+.
TL&DR: This PR
1. Removes the creation of the `.dSYM` based on `Debug Information` [1]
2. Adds dSYM support to XM msbuild (now shared with XI implementation)
3. Archive the `.dSYM` directories (plural) properly, e.g.
```
msbuild -p:Configuration=Release -p:ArchiveOnBuild=true
```
Why ? The long story...
Historically `.dSYM` for Xamarin.Mac have not been very useful, largely
because (most of) the code is JITed so not much is known before runtime.
So they were simply not generated during the builds...
However AOT options were added to Xamarin.Mac, making them potentially
more useful. Also symbols from `libmono` and other native libraries /
frameworks can prove useful when diagnosing application crashes.
Unsurprisingly developers looking to get symbols eventually found _a way_
[1] to get a `.dSYM` for their applications - but it was not quite
correct because:
* setting the debug information option meant that `mmp` would be supplied with `-debug`. This disables several optimizations that are, by default, enabled for release builds. IOW generating symbols should have no effect on the executing code (but it had);
* it was produced when compiling the native launcher, so the symbols coverage was incomplete. How much depends if mono was statically or dynamically linked. However this would not cover any AOTed code nor bundled libraries or user frameworks.
* the .dSYM was produced inside the `x.app/Contents/MacOS/`, side-by-side with the native executable, which makes it part of the **signed** `.app` and also part of the created (and signed) `.pkg`. This had a large impact on the application's, disk and download, size(s). Manually (re)moving the `.dSYM` means re-signing the app and re-creating (and signing) the `.pkg` is not a good solution.
[1] https://forums.xamarin.com/discussion/139705/how-to-symbolicate-a-xam-mac-crash-log
Additional fixes
* Use `Directory.Move` instead of running the `mv` command
While the result is identical there is a cost to spawn several `mv`
processes. Doing it in parallel (might have) helped but that setup
also comes at a cost.
`Directory.Move` the four `.dylib.dSYM` of an app takes 1 ms, while
the existing code took 17 ms to do the same.
* Fix building mmptest since the DeleteDebugSymbolCommand constant is not present (nor used) anymore
Xamarin.iOS don't AOT compile resource assemblies, so we shouldn't do so in
.NET either (and in any case they end up causing native linking failures due
to duplicate native symbols).
* [monotouch-test] Fix conditional compilation constants when building for device.
* [tests/monotouch-test] Don't use entitlements when building monotouch-test for device.
* Bump to .NET 6.0.100-alpha.1.21060.3.
* Fix dotnet command line arguments.
* dotnet build: the project file must be the first argument.
* dotnet build/publish: use the documented verbosity format.
* Update version number in tests.
* [tests/introspection] Adjust introspection to cope with different library names in the new .NET version.
* [tests/link sdk] Adjust the LinkSdkRegressionTest.SpecialFolder test according to the new version of .NET 6.
* [tests/link sdk] Preserve a required method in System.Private.CoreLib to work around a bug in .NET 6.
Ref: https://github.com/dotnet/runtime/issues/46908.
* Revert "[CI][VSTS] Add the donet 6 pkg as a dependency. (#10348)"
This reverts commit 6de4e717e7.
There's no need to provision .NET 6, it's done automatically.
* [dotnet-linker] Add support for writing to the same MSBuild output items multiple times.
* Split parts of LinkerConfiguration.WriteOutputForMSBuild into a FlushOutputForMSBuild
method (the part that does the actual writing).
* Make WriteOutputForMSBuild just store the items in a dictionary.
* Add a DoneStep that runs at the very end and that writes out the MSBuild output
items.
* [dotnet-linker] Link with GSS when building for iOS/Mac Catalyst.
Add a ComputeNativeBuildFlagsStep, which computes the flags to pass to the native
compiler + native linker. This is currently a very simple implementation, but it
will become more complex as support for missing features are added.
GSS is required because of libSystem.Net.Security.Native.a:
Undefined symbols for architecture arm64:
"___gss_c_nt_hostbased_service_oid_desc", referenced from:
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_c_nt_user_name_oid_desc", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_cred_no_ci_flags_x_oid_desc", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_ntlm_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_spnego_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
l___const.NetSecurityNative_AcquireCredSpNego.gss_mech_spnego_OID_set_desc in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_accept_sec_context", referenced from:
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcquireAcceptorCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred_with_password", referenced from:
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_delete_sec_context", referenced from:
_NetSecurityNative_DeleteSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_status", referenced from:
_NetSecurityNative_DisplayMinorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_DisplayMajorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_import_name", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_indicate_mechs", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_init_sec_context", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_inquire_context", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_oid_equal", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_buffer", referenced from:
_NetSecurityNative_ReleaseGssBuffer in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_cred", referenced from:
_NetSecurityNative_ReleaseCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ReleaseName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_oid_set", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_set_cred_option", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_unwrap", referenced from:
_NetSecurityNative_Unwrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_wrap", referenced from:
_NetSecurityNative_Wrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
ld: symbol(s) not found for architecture arm64
Mac Catalyst needs much of the Mac signing logic, so merge the iOS and Mac
versions of DetectSigningIdentity so that the Xamarin.iOS.Tasks assembly
contains what it needs for Mac Catalyst.
Rolf Bjarne Kvinge
Manuel de la Pena
Sebastien Pouliot