Rename our product assemblies to:
* Microsoft.iOS.dll
* Microsoft.tvOS.dll
* Microsoft.macOS.dll
* Microsoft.MacCatalyst.dll
This makes it easy to distinguish between legacy Xamarin and .NET whenever the
product assembly is mentioned, and I've also chosen the platform part of the
name to match how the platforms are named elsewhere (this also makes it
possible to simplify our build logic, since we can remove a lot of special
casing).
Fixes https://github.com/xamarin/xamarin-macios/issues/13748.
* [tools] Unify Application.link_flags and Application.gcc_flags from mtouch and mmp into Application.CustomLinkFlags.
* [tests] Update mtouch tests according to mtouch changes.
* [tools] Extract the logic to parse OSPlatformAttribute platform names to a separate file/class.
* [introspection] Migrate .NET code to use the new .NET-style availability attributes.
This also means using the 'ApplePlatform' enum instead of the 'PlatformName'
enum, because the latter will be removed in .NET.
* [FileProvider] Exclude some deprecated API from .NET.
* [AVFoundation] Adjust availability attribute for AVCaptureStillImageOutput.HighResolutionStillImageOutputEnabled.
* Update tests.
This makes it easier to iterate over all the *_SDK_VERSION variables in
template code, because they're all named using the standard platform names we
use elsewhere.
On CI we'll collect all the binlogs in the repository and make them available
for post-build analysis if need be, so this will make it easier to diagnose
build problems.
This makes it possible to preserve embedded debug symbols for a release build:
by passing "--package-debug-symbols=true" to mmp. Previously we'd remove the
symbols unconditionally for release builds.
This also conserves the old behavior (strip symbols in release builds), unless
users have explicitly passed "--package-debug-symbols=true" to mmp (because
PackageManagedDebugSymbols defaults to the same value as EnableDebug).
Fixes https://github.com/xamarin/xamarin-macios/issues/12263.
* making sure new strings get added to designer and resources plus the test
* Next wave of changes to csproj to incorporate Rolf's changes
* fixing path
* Update tests/msbuild/Xamarin.MacDev.Tasks.Tests/TaskTests/LocalizationStringTest.cs
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/mtouch/LocalizationTests.cs
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* forgot the include
Co-authored-by: tj_devel709 <antlambe@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
We already define it in the generated code, so the native compiler shows a warning:
Microsoft.macOS.registrar.coreclr.x86_64.m:1:9: warning: 'CORECLR_RUNTIME' macro redefined [-Wmacro-redefined]
#define CORECLR_RUNTIME
^
<command line>:2:9: note: previous definition is here
#define CORECLR_RUNTIME 1
^
* Update to new linker custom steps API
* PR feedback
- Fix indentation
- Add Initialize(LinkContext) to ExceptionalMarkHandler
- Remove unnecessary ifdef
- Use IsSetter/IsGetter
- Use [0] instead of Single()
- Avoid allocating empty collections
* Note override issue
* Clean up comments
* Move `DynamicRegistrationSupported` change earlier, along with the
detection code.
This solve the issue that `ILLink` does a similar job _before_ we have
the chance to disable the dynamic registrar.
* ILLink does not support considering other attributes as `[Preserve]`
when it is itself preserved at the assembly-level.
This ignored test is checking that feature so it cannot be enabled
for `NET`
Added to known breaking changes https://github.com/xamarin/xamarin-macios/issues/8900
* Fix removal of the dynamic registrar on legacy
* Fix IntPtr size inlining
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@microsoft.com>
* The generated static registration code will eventually be different.
* The generated code has to be compiled with different compiler flags.
This also required adding a new overload of xamarin_mono_object_release for the generated
code to compile.
* Extract the code we use to configure the assembly resolver during a normal
mmp run to make it usable for --runregistrar.
* Configure the assembly resolver we use for --runregistrar.
* Pass the assembly resolver to the registrar so that it's actually used.
* Adjust the System.Void lookup to look everywhere even if we find a corlib,
since behavior changes a bit now that we're using an assembly resolver:
* Previous behavior:
1. In .NET mode, look for a corlib named System.Private.CoreLib, and fail to find it.
2. Look in all the loaded assemblies for System.Void (and find it in System.Runtime.dll).
* Broken behavior as a result of the resolver changes:
1. Find corlib as System.Private.CoreLib.dll
2. Fail to find System.Void in System.Private.CoreLib.dll, since we'd only look in corlib.
* New behavior
1. Find corlib as System.Private.CoreLib.dll
2. Fail to find System.Void in System.Private.CoreLib.dll, but find it in System.Runtime.dll,
since we're now looking in all the loaded assemblies.
This is required to make VSMac's usage of --runregistrar
In this case we can obsolete the attribute on both legacy and dotnet
since the replacement attribute is available on both. However this does
require a small update to the legacy linker (and is part of the PR).
Fix https://github.com/xamarin/xamarin-macios/issues/10674
* Add support for Xamarin.Mac arm64
* Add compile product definition task
Xamarin.Mac can be provided with a ProductDefinition file for the generated pkg. Normally, providing a product definition was optional. However, with Apple Silicon, we have an extra issue : `productbuild` needs to know what architectures your package target. If not provided with them, it will guess to the best of its abilities. However, on Catalina and lower, the guess is x86_64, even if you have an arm64 slice. To fix this, we add a new task to compile the product definition and use this file to create the pkg. If you provide your own Product Definition, we can check and warn if the architectures don't match what we expect. If the file doesn't exist or there is no architecture, we set it ourselves based on our target architectures.
* Don't reference dynamic objC_send on arm64
When building in debug, we currently try to link dynamic objC_send symbols when targeting a 64-bit architecture. However, this is actually only defined on Intel architectures, not on arm64, so we end up failing because we're referring symbols that don't exist. Rework the `GetRequiredSymbols` to take an abi, and tag those symbols to only be valid on i386/x86_64, so they don't get referred at all when building on arm64, but still get referred in x86_64.
* Fix improper delete/move with already existing directories
* Fix stret requirement for Xamarin.Mac in arm64.
The generator supposes that we're running in x64 mode, refactor to take into account the possibility of running in arm64.
* Implement OS version generation in Product.plist, based on MinimumSystemVersion of the app
* Re-generalize some mmp registrar rules
`Microsoft.macOS.registrar` was missed by the current rule set
* Fix mmp tests
* Set E7072 as not translated
Tests were failing otherwise
* Rename Xamarin.Mac lib/x86_64 folder to 64bits (currently all targeted archs are the same)
* Fix style issues
* Fix `ToLower` usage for invariant usage
* Fix xtro-sharpie test
Use a private property (prefixed with underscore) for now, until we can decide
on a better/general name.
Also add a variation to xharness to build monotouch-test with CoreCLR
(building works fine, but it crashes at startup, which is expected at this
point).
* Add configure option to disable building for legacy Xamarin.
This can greatly speed up the debug-edit-build cycle when doing .NET
development, since it cuts down the build time in half more or less.
* Bump maccore.
New commits in xamarin/maccore:
* xamarin/maccore@548fa45432 [mlaunch] Disable building mlaunch when not including the legacy Xamarin build. (#2403)
Diff: 0562e08b12..548fa45432
* Bump Xamarin.MacDev.
New commits in xamarin/Xamarin.MacDev:
* xamarin/Xamarin.MacDev@1e738e9 [Xamarin.MacDev] Extract the code to convert between Mac Catalyst versions to a separate file. (#89)
* xamarin/Xamarin.MacDev@a3bb12c [Xamarin.MacDev] Add methods to map between iOS and macOS versions for Mac Catalyst. (#88)
Diff: 02d6d05be3..1e738e9f7f
* [msbuild] Use the macOS SDK to build Mac Catalyst apps instead of the iOS SDK
From a native compilation perspective, compilating a Mac Catalyst is the macOS SDK
+ a dash of iOS, so use the native macOS SDK to compile, and then do the corresponding
adjustments elsewhere.
At the same time document which version we want for the sdk version and the deployment
target in mtouch, and adjust the code accordingly (sdk version: macOS version; deployment
target: iOS version).
* Update resource files
* Add new entry to canary test for string localization.
Today both `mtouch` and `mmp` are copying the entire `.framework`
directories inside the `.app\[Contents\]Frameworks\` directory.
However not everything in a framework is required at runtime. The most
common unrequired files would be headers (`Headers/*.h`) and modules
(`Modules/*`).
Looking at Xcode build output we can see something like:
```
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -exclude Headers -exclude PrivateHeaders -exclude Modules -exclude \*.tbd -bitcode-strip replace-with-marker -bitcode-strip-tool
```
which excludes a few more, less common, files.
This _builtin_ command is not available externally (for us to re-use)
but it hints that Xcode is likely using `rsync` to avoid copying part of
the files.
Note: the builtin command also _likely_ calls `bitcode_strip` too (or has
similar code embedded) and `mtouch` already does so too
There's a cost to spawning an external process, like `rsync`, which we
avoid by having our own file copier, which clones files (almost zero
cost). That does not support excluding files, but deleting files is also
very cheap. Testing shows copying a framework to be less than 1 ms, even
with with extra deletion step.
* Tweak `GetRealPath` to optionally not to warn if the path does not exists
since, in this case, it's a check we want to do after resolving the path
This fixes several (5) MTouch tests looking for specific (and no extra)
warnings
```
Unable to canonicalize the path '/Users/builder/azdo/_work/2/s/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory195/testApp.app/Frameworks/Mono.framework/CVS': No such file or directory (2).
```
TL&DR: This PR
1. Removes the creation of the `.dSYM` based on `Debug Information` [1]
2. Adds dSYM support to XM msbuild (now shared with XI implementation)
3. Archive the `.dSYM` directories (plural) properly, e.g.
```
msbuild -p:Configuration=Release -p:ArchiveOnBuild=true
```
Why ? The long story...
Historically `.dSYM` for Xamarin.Mac have not been very useful, largely
because (most of) the code is JITed so not much is known before runtime.
So they were simply not generated during the builds...
However AOT options were added to Xamarin.Mac, making them potentially
more useful. Also symbols from `libmono` and other native libraries /
frameworks can prove useful when diagnosing application crashes.
Unsurprisingly developers looking to get symbols eventually found _a way_
[1] to get a `.dSYM` for their applications - but it was not quite
correct because:
* setting the debug information option meant that `mmp` would be supplied with `-debug`. This disables several optimizations that are, by default, enabled for release builds. IOW generating symbols should have no effect on the executing code (but it had);
* it was produced when compiling the native launcher, so the symbols coverage was incomplete. How much depends if mono was statically or dynamically linked. However this would not cover any AOTed code nor bundled libraries or user frameworks.
* the .dSYM was produced inside the `x.app/Contents/MacOS/`, side-by-side with the native executable, which makes it part of the **signed** `.app` and also part of the created (and signed) `.pkg`. This had a large impact on the application's, disk and download, size(s). Manually (re)moving the `.dSYM` means re-signing the app and re-creating (and signing) the `.pkg` is not a good solution.
[1] https://forums.xamarin.com/discussion/139705/how-to-symbolicate-a-xam-mac-crash-log
Additional fixes
* Use `Directory.Move` instead of running the `mv` command
While the result is identical there is a cost to spawn several `mv`
processes. Doing it in parallel (might have) helped but that setup
also comes at a cost.
`Directory.Move` the four `.dylib.dSYM` of an app takes 1 ms, while
the existing code took 17 ms to do the same.
* Fix building mmptest since the DeleteDebugSymbolCommand constant is not present (nor used) anymore
TD&LR: This PR simplifies how we refer to user frameworks and fixes both
warnings and non-optimal (app) output.
Much longer story:
Additional testing on macOS showed some build-time warnings and an
[extra (dupe) file](a20f8aba41 (diff-54fd7d9cd5deae57f30195be0a43133eace03c1132401741a317e0ae8d5e13fdR34)).
Logs shows that we referred to the xcframework several times, where once
should have been enough.
```
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework/Universal
```
The first `/native-reference` line produced a warning like:
```
MMP warning MM2006: Native library 'Universal.xcframework' was referenced but could not be found.
```
which makes sense as the tools (both `mmp` and `mtouch`) are not, by
design, aware of (unresolved) xcframeworks.
Removing `{NativeReference}` from `Xamarin.Mac.Common.targets` (and
`Xamarin.iOS.Common.targets`) as it has already been processed by
`_ExpandNativeReferences` solves this.
The other part of the issue (next two lines) is because `msbuild` does
not track changes to directories like it does for files - and the
workaround (in `_ExpandNativeReferences`) had to be copied in other
places (both XI and XM `_CompileToNative`) and that was not enough (and
would eventually need to be duplicated again and again).
This could lead to duplicate entries (i msbuild logs) like
```
NativeReferences
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework/Univeral
```
which maps to our extra entries.
In order to simplify things we make the `_ExpandNativeReferences` resolve
the full path to the library name (not the `.framework` directory) which
simplifies both `_CompileToNative` and ensure a single way (at least for
`msbuild`) to provide this data to the tools (`mmp` and `mtouch`).
Using a file, instead of a directory, is also more consistent for the
existing `-framework` option, e.g. we provide the names like:
```
--framework=CoreLocation
--framework=ModelIO
```
So adding a full path that include the name is more appropriate, e.g.
``` --framework=/Users/poupou/git/master/xamarin-macios/tests/xharness/tmp-test-dir/xcframework-test760/bin/AnyCPU/Debug/bindings-xcframework-test.resources/XTest.xcframework/ios-i386_x86_64-simulator/XTest.framework/XTest
```
Finally for macOS applications it turns out we were embedding yet another
copy of the framework's library inside the `MonoBundle`, which is clearly
wrong, because of the last entry.
```
$ l bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
-rwxr-xr-x 1 poupou staff 167152 2 Dec 16:16 bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
```
The tool now checks if a provided library is inside a framework (or not)
which is a good validation to have anyway when it gets called directly,
i.e. not thru `msbuild`.
User frameworks for macOS often uses symlinks (as Xcode creates them
this way).
This cause problem cause the symlink is on the binary and we expected
the `_CodeSignature` directory to by side-by-side with the binary. This
was missing and cause exceptions when codesigning such frameworks.
A second problem happened because `mmp` use `lipo -thin` to remove
non-required architectures. However when a framework has symlinks, like:
```
├── Frameworks
│ └── Universal.framework
│ ├── Resources -> Versions/Current/Resources
│ ├── Universal -> Versions/Current/Universal
│ └── Versions
│ ├── A
│ │ ├── Resources
│ │ │ └── Info.plist
│ │ ├── Universal
│ │ └── _CodeSignature
│ │ └── CodeResources
│ └── Current -> A
```
then this actually replaced the (very small) symlink with a thin version
of the framework. Which means the original one was still _fat_ and the
whole app was now larger than the original version.
Sample used: https://github.com/spouliot/xcframework/tree/main/xamarin/xcf-mac
* [src] Generate the string constants for each framework from our list of frameworks.
This makes it less error prone to add new frameworks (one less place to add
them), and easier to add new platforms (only one place to make sure the list
of frameworks is correct).
This also revelead a few issues in our list of frameworks (missing frameworks,
frameworks that have been moved, etc.), which have been fixed.
* [mmp] Generate the library path -> namespace map for the linker to optimize away dlopen statements.
* Fix indentation to be just as broken as the surrounding indentation.
* Remove unused Cecil reference.
* Fix c&p oversight.
* Adjust more Mac Catalyst API.
* CFNetwork is its own top-level framework.
* [src] Build the .NET version of our product assemblies using a .NET 5 BCL.
We're not shipping the .NET product assemblies in any stable release, so we
can use a preview version of the .NET 5 BCL.
Also:
* Add all the nuget feeds we need to the top-level NuGet.config, even for .NET
5/6, there shouldn't be any conflicts with stable feeds since we use exact
version numbers.
* Generate a top-level global5.json which is copied to every directory that
needs a .NET 5 global.json (overriding the .NET 3.1 global.json in the root
directory).
* Use the expected dotnet binary during our local build.
* [tests] Fix the bgen tests to use .NET 5.
* [xharness] Set the current directory to the project directory when running .NET tests.
This way we end up using the dotnet version that's configured in global.json for the tests.
This means:
* Move the parts of the ApplyPreserveAttribute step that we don't need for.NET
into a new MobileApplyPreserveAttribute step, and have mtouch and mmp use
that step instead of the ApplyPreserveAttribute step.
* Copy ApplyPreserveAttributeBase into dotnet-linker from the upstream tuner
source (with minor modifications) so that our ApplyPreserveAttribute step
compiles.
* Create a DotNetSubStepDispatcher class that we're going to use as our
substep dispatcher, create an instance of it and insert it into the list of
linker steps.
* Also a workaround for the lack of LinkContext.GetAssemblies (): add a step
that collects all the assemblies and stores them in a list, so that we can
have our own GetAssemblies implementation.
I filed a linker issue to see if we can get LinkContext.GetAssemblies ()
exposed to us: https://github.com/mono/linker/issues/1455.
Fixes this startup crash with the linkall test:
2020-08-26 19:47:10.219697+0200 link all[32709:6065783] Xamarin.iOS: Fatal error: failed to load the method 'ObjCRuntime.Runtime.Initialize'.
=================================================================
Native Crash Reporting
=================================================================
Got a abrt while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x104007b0e - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
0x103fb4437 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_handle_native_crash
0x104007365 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
0x7fff51c005fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_platform.dylib : _sigtramp
0x0 - Unknown
0x7fff51af0b7c - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
0x103daad98 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_assertion_message
0x103dade77 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_initialize
0x103dbf80b - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_main
0x103cd2f0d - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/link all : main
0x7fff51a231fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdyld.dylib : start
This is a partial/modified port of the initial linker support (bc88790201)
Co-authored-by: Sebastien Pouliot <sebastien@xamarin.com>
* [mmp] Rename LinkMode.All to LinkMode.Full.
So that we can continue to use Enum.Parse<LinkMode> to parse 'Full' as the link mode.
* [dotnet] Implement support for our different link modes.
Tell the managed linker what to do with each input assembly depending the selected
link mode (link all, link sdk, don't link).
Rolf Bjarne Kvinge
Manuel de la Pena
Filip Navara
Sebastien Pouliot
TJ Lambert
Sven Boemer
mathieubourgeois