Change dSYM generation and native stripping to occur immediately before code signing,
in a newly minted post processing target.
Challenges:
* Both calling 'strip' and 'codesign' on an executable modifies that executable,
which means that we must make sure to not call 'dsymutil' on the same binary at
a later point unless it's been rebuilt.
* Thus we must make sure to update 'dsymutil's stamp file whenever we call 'strip'
and/or 'codesign' on an executable.
* Just like for code signing, we must store the libraries (either static or dynamic)
we post process in extension/watch/rid-specific projects, so that these libraries
can be loaded in containing projects and processed there.
* In universal .NET builds, debug symbols are created for the universal app bundle,
not for each rid-specific version of the app bundle. So I had to add logic to create
the native symbol lists (MtouchSymbolsList) for each rid-specific build, but then
collect them and merge those lists for the universal app bundle.
The existing SymbolStrip call we did right after linking the native executable has
been removed, because we have to do that after creating the dSYM (which the GenerateDebugSymbols
target does).
Also add tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/14067.
Also establish when we should do post processing: only in the outermost build. This
is a slight change from previous behavior, where we'd run strip/dsymutil separately
for app extensions and watch apps.
The main theme here is that code signing will be done in the outermost
executable project, not in any app extension projects or watch projects, nor
during the RID-specific build of a .NET universal app. This makes codesigning
easier to reason about and other affected logic (such as strip/dsymutil)
easier to handle, in particular for .NET universal apps. Another benefit is
that the differences between the iOS and macOS code bases have been
eliminated.
The first step is to collect all the information we need from the targets
files. Every app bundle (be it app extension, watch app or main app) will add
its own output app bundle (.app/.appex) to the _CodesignBundle item group.
Then every app bundle will load this informarion from referenced app bundles,
and finally store this information on disk (in the 'codesign-bundle.items'
file). This means that in the end the main app bundle will have a list of all
contained app bundles in the app (recursively), in the _CodesignBundle item
group.
Separately we keep a list of other items that need signing, in the
_CodesignItems item group, and we do the same store/load logic for every
contained/contained app bundle (in the 'codesign.items' file, so a the end the
main app bundle will have a list of all the _CodesignItems for all contained
app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and
_StoreCodesigningData targets.
The next step is to use the new ComputeCodesignItems task to compute
everything we need to know for code signing. This task takes over the
responsibility for listing all the *.dylib and *.metallib files, and the
*.framework directories in the app bundles, that need signing (which was
previously done in the targets file). This logic is significantly easier to
write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use
to determine if something needs (re-)signing. Previously .framework
directories did not have a stamp location, so they'd always end up resigned in
a rebuild, while now we'll automatically skip signing *.framework directories
unless something changed in them.
I've also tried to comment everything thorougly, for the next poor soul having
to deal with any bugs.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same
thing for all platforms.
* We're now always signing *.framework directories for all platforms (like we
do for *.dylib files), since frameworks are pretty much like dylibs anyways.
I've verified that this works both by running the submission tests and running
and launching a sample project on device from Windows.
Mac Catalyst and macOS projects support an 'EnableCodeSigning' property to determine
whether an app is signed or not. In order to bring parity on mobile platforms, add
support for this property for iOS, tvOS and watchOS projects as well - if not set,
we'll still have the old behavior of signing device builds and not signing simulator builds.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The main theme here is that code signing will be done in the outermost executable
project, not in any app extension projects or watch projects, nor during the RID-specific
build of a .NET universal app. This makes codesigning easier to reason about and
other affected logic (such as strip/dsymutil) easier to handle, in particular for
.NET universal apps. Another benefit is that the differences between the iOS and
macOS code bases have been eliminated.
The first step is to collect all the information we need from the targets files.
Every app bundle (be it app extension, watch app or main app) will add its own output
app bundle (.app/.appex) to the _CodesignBundle item group. Then every app bundle
will load this informarion from referenced app bundles, and finally store this information
on disk (in the 'codesign-bundle.items' file). This means that in the end the main
app bundle will have a list of all contained app bundles in the app (recursively),
in the _CodesignBundle item group.
Separately we keep a list of other items that need signing, in the _CodesignItems
item group, and we do the same store/load logic for every contained/contained app
bundle (in the 'codesign.items' file, so a the end the main app bundle will have
a list of all the _CodesignItems for all contained app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and _StoreCodesigningData
targets.
The next step is to use the new ComputeCodesignItems task to compute everything we
need to know for code signing. This task takes over the responsibility for listing
all the *.dylib and *.metallib files, and the *.framework directories in the app
bundles, that need signing (which was previously done in the targets file). This
logic is significantly easier to write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use to
determine if something needs (re-)signing. Previously .framework directories did
not have a stamp location, so they'd always end up resigned in a rebuild, while now
we'll automatically skip signing *.framework directories unless something changed
in them.
I've also tried to comment everything thorougly, for the next poor soul having to
deal with any bugs, as well has adding a comprehensive test for the new task.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same thing
for all platforms.
* We're now always signing *.framework directories for all platforms (like we do
for *.dylib files), since frameworks are pretty much like dylibs anyways.
I'm removing the old logic first, because the new logic is so different that a diff
just complicates understanding what's happening (there's not much value in comparing
textually what's changed when it's pretty much a complete rewrite).
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to run dsymutil on at the same time.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to dsymutil, to make command lines
easier to copy-paste.
Make the CollectBundleResourcesDependsOn property public, so that custom
targets can inject themselves into the build early enough to add additional
BundleResource or Content items (by adding their custom target's name to the
CollectBundleResourcesDependsOn property).
Fixes https://github.com/xamarin/xamarin-macios/issues/11984.
These properties aren't used.
This also allows us to remove the CompiledArchitectures output property from
the MTouch task, because it's not longer used anymore either.
* Remove ObjCRuntime.nfloat (in favor of System.Runtime.InteropServices.NFloat).
* Automatically add a reference to the System.Runtime.InteropServices.Internal
package, so that developers get the new NFloat API (with operators) we've
added post .NET 6 (but don't do this for .NET 7).
* Automatically add a global using alias for
System.Runtime.InteropServices.NFloat -> nfloat. This is not behind the
usual `ImplicitUsings` condition our other implicit usings are, because
they're off by default for existing projects, and the main target for the
global using alias for nfloat is upgraded projects.
* Automatically generate a global using alias (like above) in the generator
for all code the generator compiles.
* Update xtro entries to reference System.Runtime.InteropServices.NFloat
instead of ObjCRuntime.nfloat.
* Add a workaround for a hopefully temporary issue with .NET/CoreCLR where the
wrong runtime pack is selected otherwise (without the new NFloat API, so
nothing works at runtime).
Ref: https://github.com/xamarin/xamarin-macios/issues/13087
Rename our product assemblies to:
* Microsoft.iOS.dll
* Microsoft.tvOS.dll
* Microsoft.macOS.dll
* Microsoft.MacCatalyst.dll
This makes it easy to distinguish between legacy Xamarin and .NET whenever the
product assembly is mentioned, and I've also chosen the platform part of the
name to match how the platforms are named elsewhere (this also makes it
possible to simplify our build logic, since we can remove a lot of special
casing).
Fixes https://github.com/xamarin/xamarin-macios/issues/13748.
Add support for the PublishFolderType metadata on Content and BundleResource
items, which makes it possible to change the location in the app bundle for
these items (this was possible to do before with the Link metadata), but most
importantly it also makes it possible to choose to *not* bundle these items in
the app bundle (which was not possible before with the Link metadata, nor any
other means).
At first I thought setting CopyToPublishDirectory /
CopyToOutputDirectory=Never would accomplish that, but it turns out we don't
honor those, and since we already have this behavior of ignoring
CopyToPublishDirectory / CopyToOutputDirectory in legacy Xamarin, I didn't
want to change it in .NET.
So I added support for honoring the PublishFolderType metadata instead, which
is new and we already support for other item groups. This is accomplished by
adding all Content and BundleResource items with PublishFolderType set to the
ResolvedFileToPublish item group (where we already handle any
PublishFolderType values), and then we ignore such Content and BundleResource
items in our CollectBundleResources task.
Also update the documentation and add tests.
Not embedding third-party libraries in the binding assembly is the future, and
let's try to enable it by default starting with .NET.
Fixes https://github.com/xamarin/xamarin-macios/issues/12530.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
* Add support for specifying metadata on items that are to be codesigned to
override any general codesign setting.
* Make the Codesign task able to sign files and directories that may depend on
eachother.
Implement support for ordering signing so that directories containing files
that also must be signed are signed after those files.
This is implemented by:
1. Normalize all input (resolve symlinks, create full path, etc.)
2. Sort by path length (longest to shortest paths). This way we're certain
that if we find a directory, we'll know that we won't find any files
later in the list from inside that directory.
3. Group into an ordered list of buckets, where each bucket contains files
and directories that don't depend on eachother (i.e. they can all be
signed in parallel).
This makes it possible to call Codesign once, listing both the app bundle
itself, and all the individual files or directories inside that need
signing, and the Codesign task will sign the items in an order that ensures
parent directories are always signed after any files or directories inside.
* Finally rework code signing to sign everything with a single call to the
Codesign task in the _CodesignAppBundle target for the executable project,
instead of having multiple calls to the Codesign task from multiple targets
(and projects). This makes it easier to reason about what's being signed,
and it also makes it easier to add files to the signing process.
* Also make the Codesign task able to figure out if something needs to be
signed, and change the _CodesignAppBundle target to not keep track of
inputs/outputs, because it becomes quite complex (for directories, it needs
to keep track of all the files inside that directory, and also if there's
anything in the directory that's also being signed).
The targets were somewhat different, where the Mac version seemed much less
evolved than the iOS version, so this meant mostly update the iOS version to
be Mac-compatible as well (and use it for macOS).
In .NET, all files that should be published (put into the final .app bundle) are put into the @(ResolvedFileToPublish) item group, and at the end of the build process, .NET will publish all the files in that item group. Which files are in this item group, and how they're put in there, is out of our control (it's just how the build process works in .NET), so we'll have to cope.
Additionally, publishing an app for Apple platforms is different than publishing other .NET apps, because we can't just put all the files in the a directory like .NET usually does, we have a fairly strict directory structure we need to follow, and it also differs between platforms (the structure is different between macOS and iOS for instance).
This means that for every file in the `ResolvedFileToPublish` item group, we have to figure out:
* Should it be put into the app bundle in the first place?
* If so, in which subdirectory (if any)?
This PR implements these changes. The exact details are explained in a document in the PR, but the general logic is:
* We make an educated guess for some types of files we know about (assemblies, unmanaged libraries, images, etc).
* We provide a way to set metadata on each item specifying which type of item it is (assembly, unmanaged library, image, etc), and we'll treat the item as such. This method can also be used to override the guess we made (for files that shouldn't be published for instance).
* We warn if we run into files we're not educated enough to be able to guess about, and for which there's no custom metadata set.
Fixes https://github.com/xamarin/xamarin-macios/issues/12572.
Touch all files when copying an app extension into the app.
This fixes an issue where we'd in some cases run into this scenario:
* Build once (successfully), and sign the app (and app extension), including all dylibs.
* We build again, copy the (unsigned) app extension into the app again, but
not sign dylibs because the timestamp went back into the past.
named 'Info.plist', and assume that's the app manifest.
That doesn't quite work when we end up with multiple 'Info.plist' entries in any
of those item groups (one example being a framework as a BundleResource - all frameworks
have an Info.plist, and there's no good way to distinguish what the developer's intention
was).
So:
1. Implement a 'AppManifestDetectionEnabled' property to disable automatic app manifest
detection.
2. Add a public 'AppBundleManifest' property that specifies the app manifest
(this is just a renamed version of our previously private '_AppManifest' property).
This makes it possible for app developers to:
* Disable automatic app manifest detection.
* Still have an app manifest by specifying it manually.
* Disable automatic app manifest detection, but also not specify an app manifest
manually (so no custom app manifest at all).
Also:
* Rename '_AppBundleManifest' to '_AppBundleManifestPath' to make it less confusing
with the new 'AppBundleManifest' property.
The GetNuGetShortFolderName task only exists in .NET builds, which means we
have to find another way to compute the path to resources in nugets for legacy
builds. Do it the simple way by hardcoding the possible values.
Fixes https://github.com/xamarin/xamarin-macios/issues/13469.
* [tests] Create a libtest.xcframework and libtest2.xcframework
* [tests] Make bindings-test and bindings-test2 use an xcframework instead of plain static library
* [msbuild] Add support for xcframeworks with static libraries in them.
* List the frameworks libtest needs.
* [tests] Update .NET unit tests according to test project changes.
* [tests] Add new test to verify that packing an old-style binding project doesn't work.
It's not needed, and may in some cases do the wrong thing: for instance, this
logic will automatically reference any *.dll files in the project directory,
which breaks the build if those *.dll files aren't actually assemblies.
The '_AppExtensionRoot' contains the correct parent directory of the 'PlugIns'
directory for all platforms, so use that instead of appending 'PlugIns' to
'_AppBundlePath' - which is incorrect on macOS and Mac Catalyst, because the
'PlugIns' parent directory is '$(_AppBundlePath)/Contents' on those platforms.
Fixes https://github.com/xamarin/xamarin-macios/issues/13415.
Rolf Bjarne Kvinge