Sometimes we want to copy the entire input directory from Windows to the Mac
when executing the Ditto task remotely, and sometimes we don't.
In particular we do not want to copy the input directory when the directory on
Windows is an incomplete mirror of what's on the Mac - one scenario being when
copying the app bundle to prepare for IPA creation. The .app directory on
Windows is not complete - all the files are there (maybe? not quite sure, but
that's beside the point here), but some may be empty, because when we only
care about the timestamp for a file, we'll create an empty file on Windows to
mirror the actual file on Mac. Copying this incomplete directory to the Mac,
overwriting the correct files there, will break things badly.
However, sometimes we're not mirroring a directory on Windows, but instead we
have directories as actual build input (for instances frameworks from NuGets),
and in that case we want to copy everything to the Mac.
So this PR adds a parameter to the Ditto task to optionally copy the directory
from Windows for remote builds, and we enable this behavior when we want it -
specifically when copying frameworks.
Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009 while not
regressing https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635.
Ref: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009
Ref: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635
Ref: https://github.com/xamarin/xamarin-macios/pull/14375
Updated to 1.5.8, including fixes for a hang in build cancellation during SayHello and also an improvement in the port forwarding logic when establishing the SSH connection
Change dSYM generation and native stripping to occur immediately before code signing,
in a newly minted post processing target.
Challenges:
* Both calling 'strip' and 'codesign' on an executable modifies that executable,
which means that we must make sure to not call 'dsymutil' on the same binary at
a later point unless it's been rebuilt.
* Thus we must make sure to update 'dsymutil's stamp file whenever we call 'strip'
and/or 'codesign' on an executable.
* Just like for code signing, we must store the libraries (either static or dynamic)
we post process in extension/watch/rid-specific projects, so that these libraries
can be loaded in containing projects and processed there.
* In universal .NET builds, debug symbols are created for the universal app bundle,
not for each rid-specific version of the app bundle. So I had to add logic to create
the native symbol lists (MtouchSymbolsList) for each rid-specific build, but then
collect them and merge those lists for the universal app bundle.
The existing SymbolStrip call we did right after linking the native executable has
been removed, because we have to do that after creating the dSYM (which the GenerateDebugSymbols
target does).
Also add tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/14067.
Also establish when we should do post processing: only in the outermost build. This
is a slight change from previous behavior, where we'd run strip/dsymutil separately
for app extensions and watch apps.
* Save all the NativeReference metadata in binding resource packages.
* Copy all the NativeReference metadata to new items when resolving native references.
This makes it possible to set custom metadata on NativeReferences, and have that
metadata show up when it's needed, which might not be in the same project (for instance
if the native reference is in a binding project, we might want the custom metadata
when we load the native references from the binding project's resource package -
another case is when app extensions have native references, we might want any custom
metadata in the main executable project to know how to handle certain types of native
references).
Also sort the metadata we write to binding resource packages, so that the output
is stable. This required updating the corresponding tests.
The main theme here is that code signing will be done in the outermost
executable project, not in any app extension projects or watch projects, nor
during the RID-specific build of a .NET universal app. This makes codesigning
easier to reason about and other affected logic (such as strip/dsymutil)
easier to handle, in particular for .NET universal apps. Another benefit is
that the differences between the iOS and macOS code bases have been
eliminated.
The first step is to collect all the information we need from the targets
files. Every app bundle (be it app extension, watch app or main app) will add
its own output app bundle (.app/.appex) to the _CodesignBundle item group.
Then every app bundle will load this informarion from referenced app bundles,
and finally store this information on disk (in the 'codesign-bundle.items'
file). This means that in the end the main app bundle will have a list of all
contained app bundles in the app (recursively), in the _CodesignBundle item
group.
Separately we keep a list of other items that need signing, in the
_CodesignItems item group, and we do the same store/load logic for every
contained/contained app bundle (in the 'codesign.items' file, so a the end the
main app bundle will have a list of all the _CodesignItems for all contained
app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and
_StoreCodesigningData targets.
The next step is to use the new ComputeCodesignItems task to compute
everything we need to know for code signing. This task takes over the
responsibility for listing all the *.dylib and *.metallib files, and the
*.framework directories in the app bundles, that need signing (which was
previously done in the targets file). This logic is significantly easier to
write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use
to determine if something needs (re-)signing. Previously .framework
directories did not have a stamp location, so they'd always end up resigned in
a rebuild, while now we'll automatically skip signing *.framework directories
unless something changed in them.
I've also tried to comment everything thorougly, for the next poor soul having
to deal with any bugs.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same
thing for all platforms.
* We're now always signing *.framework directories for all platforms (like we
do for *.dylib files), since frameworks are pretty much like dylibs anyways.
I've verified that this works both by running the submission tests and running
and launching a sample project on device from Windows.
The ComputeCodesignItems does not touch any files, and all the input files
should already exist on the mac, so there's no need to copy files back and
forth.
Mac Catalyst and macOS projects support an 'EnableCodeSigning' property to determine
whether an app is signed or not. In order to bring parity on mobile platforms, add
support for this property for iOS, tvOS and watchOS projects as well - if not set,
we'll still have the old behavior of signing device builds and not signing simulator builds.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The CodesignEntitlements and CodesignResourceRules properties can be relative paths,
and they might be coming from a referenced project. This means that if they're relative
paths, we must resolve them to a full path using the project that defined them (which
is specified using the 'SourceProjectPath' metadata).
The main theme here is that code signing will be done in the outermost executable
project, not in any app extension projects or watch projects, nor during the RID-specific
build of a .NET universal app. This makes codesigning easier to reason about and
other affected logic (such as strip/dsymutil) easier to handle, in particular for
.NET universal apps. Another benefit is that the differences between the iOS and
macOS code bases have been eliminated.
The first step is to collect all the information we need from the targets files.
Every app bundle (be it app extension, watch app or main app) will add its own output
app bundle (.app/.appex) to the _CodesignBundle item group. Then every app bundle
will load this informarion from referenced app bundles, and finally store this information
on disk (in the 'codesign-bundle.items' file). This means that in the end the main
app bundle will have a list of all contained app bundles in the app (recursively),
in the _CodesignBundle item group.
Separately we keep a list of other items that need signing, in the _CodesignItems
item group, and we do the same store/load logic for every contained/contained app
bundle (in the 'codesign.items' file, so a the end the main app bundle will have
a list of all the _CodesignItems for all contained app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and _StoreCodesigningData
targets.
The next step is to use the new ComputeCodesignItems task to compute everything we
need to know for code signing. This task takes over the responsibility for listing
all the *.dylib and *.metallib files, and the *.framework directories in the app
bundles, that need signing (which was previously done in the targets file). This
logic is significantly easier to write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use to
determine if something needs (re-)signing. Previously .framework directories did
not have a stamp location, so they'd always end up resigned in a rebuild, while now
we'll automatically skip signing *.framework directories unless something changed
in them.
I've also tried to comment everything thorougly, for the next poor soul having to
deal with any bugs, as well has adding a comprehensive test for the new task.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same thing
for all platforms.
* We're now always signing *.framework directories for all platforms (like we do
for *.dylib files), since frameworks are pretty much like dylibs anyways.
I'm removing the old logic first, because the new logic is so different that a diff
just complicates understanding what's happening (there's not much value in comparing
textually what's changed when it's pretty much a complete rewrite).
* [DittoTask] If Source is a folder, add all its content as ITaskItem
* [DittoTask] Undoing undo...
* Accidentally hit Cmd+Z and removed an using statement
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Alex Soto <alex@alexsoto.me>
For reasons I don't quite understand, ditto might fail when executed by XMA:
Target Name=_CopyDirectoriesToBundle Project=C:\Users\rolf\source\iOSApp4\iOSApp4.csproj
Building target "_CopyDirectoriesToBundle" completely.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework/ArcGIS-arm64" does not exist.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\Runtimecore.framework/Runtimecore" does not exist.
Ditto
Assembly = C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\tools\msbuild\iOS\..\iOS\Xamarin.iOS.Tasks.dll
Parameters
Destination = bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework
TouchDestinationFiles = True
SessionId = <SessionId>
Source = C:\Users\rolf\.nuget\packages\esri.arcgisruntime.runtimes.ios\100.13.0\framework\ios-arm64\native\ArcGIS-arm64.framework\
Ditto: <timestamp> - Started
Ditto: <timestamp> - Initializing
[xma]: Trying to get a Build Connection for Session '<SessionId>': Xamarin.Messaging.Build.Client.BuildConnection.<SessionId>, Lifetime: Build
Ditto: <timestamp> - Initialized
Ditto: <timestamp> - There's no available inputs to copy to the Mac
Ditto: <timestamp> - Serializing intputs
Ditto: <timestamp> - Executing
[xma]: Starting remote task execution for 'iOSApp4': Xamarin.MacDev.Tasks.Ditto
[xma]: Sending Request Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic xvs/build/execute-task/iOSApp4/15f3833002fDitto
[xma]: Received Response of Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic build<SessionId>4396rolf/+/xvs/build/execute-task/iOSApp4/15f3833002fDitto
Ditto: <timestamp> - Logging messages
/usr/bin/ditto C:/Users/rolf/.nuget/packages/esri.arcgisruntime.runtimes.ios/100.13.0/framework/ios-arm64/native/ArcGIS-arm64.framework/ bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework
ditto: bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework: File exists
Errors
C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\targets\Xamarin.Shared.Sdk.targets(668,3): error MSB6006: "ditto" exited with code 1. [C:\Users\rolf\source\iOSApp4\iOSApp4.csproj]
Ditto: <timestamp> - Finished
This doesn't happen when building on macOS, nor if I copy the offending ditto
command and execute it manually on macOS.
Since I don't know why the problem occurs in the first place, I don't know why
passing full paths to 'ditto' works either. It shouldn't cause problems
elsewhere though.
Ref: https://github.com/xamarin/xamarin-macios/issues/13665
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to run dsymutil on at the same time.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to dsymutil, to make command lines
easier to copy-paste.
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to strip at the same time.
* Strip in parallel.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to strip, to make command lines
easier to copy-paste.
* Remove test that is now outdated. We have other tests that run strip
anyways, so this shouldn't be a problem.
This fixes an issue where we'd do logic with Windows-style paths on macOS, and that's
never the right thing to do.
For the LinkNativeCode task, this would manifest as this error when building from windows:
> ld: file too small (length=0) file 'obj/Debug/net6.0-ios/iossimulator-x64/nativelibraries/libSystem.Native.dylib' for architecture x86_64
because the 'ShouldCopyToBuildServer' method would return incorrect results.
For the Codesign task, it would manifest as an exception trying to create a
directory with an empty string (because the directory name of a windows-style
path is an empty string on macOS).
Since this exception was quite useless (just getting the exception message
didn't tell me much about what caused the exception, because it had no stack
trace information), I've also improved error reporting in both of these tasks.
Make the CollectBundleResourcesDependsOn property public, so that custom
targets can inject themselves into the build early enough to add additional
BundleResource or Content items (by adding their custom target's name to the
CollectBundleResourcesDependsOn property).
Fixes https://github.com/xamarin/xamarin-macios/issues/11984.
These properties aren't used.
This also allows us to remove the CompiledArchitectures output property from
the MTouch task, because it's not longer used anymore either.
Also change the key for our Info.plist entry with the version number in .NET, and document the change.
We now use "com.microsoft.<platform in lower case>" instead of "com.xamarin.ios" (for all platforms).
Fixes https://github.com/xamarin/xamarin-macios/issues/14108.
Co-authored-by: TJ Lambert <50846373+tj-devel709@users.noreply.github.com>
* Remove ObjCRuntime.nfloat (in favor of System.Runtime.InteropServices.NFloat).
* Automatically add a reference to the System.Runtime.InteropServices.Internal
package, so that developers get the new NFloat API (with operators) we've
added post .NET 6 (but don't do this for .NET 7).
* Automatically add a global using alias for
System.Runtime.InteropServices.NFloat -> nfloat. This is not behind the
usual `ImplicitUsings` condition our other implicit usings are, because
they're off by default for existing projects, and the main target for the
global using alias for nfloat is upgraded projects.
* Automatically generate a global using alias (like above) in the generator
for all code the generator compiles.
* Update xtro entries to reference System.Runtime.InteropServices.NFloat
instead of ObjCRuntime.nfloat.
* Add a workaround for a hopefully temporary issue with .NET/CoreCLR where the
wrong runtime pack is selected otherwise (without the new NFloat API, so
nothing works at runtime).
Ref: https://github.com/xamarin/xamarin-macios/issues/13087
This way it's easier to copy-paste the path to the these files from terminal output
and open/run it (with a relative/partial path you'll need to know the current directory,
which is just an annoying thing to figure out sometimes).
Rename our product assemblies to:
* Microsoft.iOS.dll
* Microsoft.tvOS.dll
* Microsoft.macOS.dll
* Microsoft.MacCatalyst.dll
This makes it easy to distinguish between legacy Xamarin and .NET whenever the
product assembly is mentioned, and I've also chosen the platform part of the
name to match how the platforms are named elsewhere (this also makes it
possible to simplify our build logic, since we can remove a lot of special
casing).
Fixes https://github.com/xamarin/xamarin-macios/issues/13748.
Add support for the PublishFolderType metadata on Content and BundleResource
items, which makes it possible to change the location in the app bundle for
these items (this was possible to do before with the Link metadata), but most
importantly it also makes it possible to choose to *not* bundle these items in
the app bundle (which was not possible before with the Link metadata, nor any
other means).
At first I thought setting CopyToPublishDirectory /
CopyToOutputDirectory=Never would accomplish that, but it turns out we don't
honor those, and since we already have this behavior of ignoring
CopyToPublishDirectory / CopyToOutputDirectory in legacy Xamarin, I didn't
want to change it in .NET.
So I added support for honoring the PublishFolderType metadata instead, which
is new and we already support for other item groups. This is accomplished by
adding all Content and BundleResource items with PublishFolderType set to the
ResolvedFileToPublish item group (where we already handle any
PublishFolderType values), and then we ignore such Content and BundleResource
items in our CollectBundleResources task.
Also update the documentation and add tests.
Hopefully fixes random build failures like this:
> xamarin-macios/msbuild/Versions.ios.g.cs(3,1): error CS1022: Type or namespace definition, or end-of-file expected
Not embedding third-party libraries in the binding assembly is the future, and
let's try to enable it by default starting with .NET.
Fixes https://github.com/xamarin/xamarin-macios/issues/12530.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Mauro Agnoletti
Rolf Bjarne Kvinge
Israel Soto