It includes a change revert because of threading freeze issues.
An issue has been detected in Xamarin.Messaging 1.5.26 and 1.6.1 that makes the code unstable and sensitive to thread freeze issues, so we are reverting back the changes until we can stabilize it.
This delays the fix of a bug in MSBuild command line builds for iOS remote builds.
This also meant:
* Using 'latest' as the C# language version for all msbuild/ project files.
* Enabling warnaserror for nullability warnings.
* Fix any nullability warnings in the CompileAppManifest files.
* Fix a nullability warning in the Ditto task.
* Fix any '== null' or '!= null' to use 'is null' and 'is not null'.
Use 'Microsoft.<platform>' as the product name instead of 'Xamarin.[iOS|Mac]' for .NET builds. That makes error messages like this:
> [...]\Microsoft.iOS.Sdk\15.4.200-ci.windows2.62\tools\msbuild\iOS\Xamarin.Shared.targets(1676,3): Could not find Xamarin.iOS in /usr/local/share/dotnet/packs/Microsoft.iOS.Sdk/15.4.200-ci.windows2.62/.
look a bit better in .NET:
> [...]\Microsoft.iOS.Sdk\15.4.200-ci.windows2.62\tools\msbuild\iOS\Xamarin.Shared.targets(1676,3): Could not find Microsoft.iOS in /usr/local/share/dotnet/packs/Microsoft.iOS.Sdk/15.4.200-ci.windows2.62/.
* Move logic to validate the UIDeviceFamily value to shared code.
* Remove logic related to watchOS 1 apps, it's been dead for a while.
* Change logic to not overwriting any existing UIDeviceFamily values in the customer's
Info.plist.
This makes the code more platform-agnostic and easier to work with across all platforms
(such as adding new validations).
Fixes this warning from the Codesign task:
C:\Users\rolf\...\Microsoft.iOS.Sdk\15.4.200-...\tools\msbuild\iOS\Xamarin.Shared.targets(2045,3): Cannot create 'C:\Users\rolf\source\iOSApp4\bin\Debug\net6.0-ios\ios-arm64\device-builds\iphone14.2-15.3.1\iOSApp4.app\Frameworks\ArcGIS-arm64.framework' because a file or directory with the same name already exists.
C:\Users\rolf\...\Microsoft.iOS.Sdk\15.4.200-...\tools\msbuild\iOS\Xamarin.Shared.targets(2045,3): Cannot create 'C:\Users\rolf\source\iOSApp4\bin\Debug\net6.0-ios\ios-arm64\device-builds\iphone14.2-15.3.1\iOSApp4.app\Frameworks\Runtimecore.framework' because a file or directory with the same name already exists.
which occurs when the Codesign task asks XVS to create output files for files from
inside ditto'ed directories, and if XVS created output files for those directories
in the Ditto task, then XVS would be trying to create files inside these output files
as if they were directories. That doesn't work (thus the warning).
I've fixed this by:
* Removing the 'ShouldCreateOutputFile' implementation. The ShouldCreateOutputFile
method is called on Windows, and we can't determine from Windows whether the destination
is a directory or a file.
* Remove the [Output] attribute for the Destination property, this way XVS doesn't
automatically try to create an output file for whatever the destination is.
* Add another CopiedFiles output property, which contains all the copied files
(and only files), so that XVS mirrors this with output files on Windows.
Fixes part of https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1505990/.
When XVS creates output/stamp files on Windows, the paths must be relative,
because otherwise XVS will append the full macOS path to the current Windows
directory and get garbage.
XVS also expects only files as output, so don't return any directories.
Fixes part of https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1505990/.
* Fix resolving paths to required test files (test files can be found relative to the root path of the repository, not relative to where Xamarin.Mac is installed)
* Don't try to sign symlinks - we can end up trying to sign the target of the symlink twice simultaneously.
* Fix finding libxammac.dylib and Xamarin.Mac.dll when testing a system installation (when MAC_DESTDIR or TESTS_USE_SYSTEM are set).
* Remove a few .NET tests we don't need anymore.
Fixes https://github.com/xamarin/maccore/issues/2560.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Sometimes we want to copy the entire input directory from Windows to the Mac
when executing the Ditto task remotely, and sometimes we don't.
In particular we do not want to copy the input directory when the directory on
Windows is an incomplete mirror of what's on the Mac - one scenario being when
copying the app bundle to prepare for IPA creation. The .app directory on
Windows is not complete - all the files are there (maybe? not quite sure, but
that's beside the point here), but some may be empty, because when we only
care about the timestamp for a file, we'll create an empty file on Windows to
mirror the actual file on Mac. Copying this incomplete directory to the Mac,
overwriting the correct files there, will break things badly.
However, sometimes we're not mirroring a directory on Windows, but instead we
have directories as actual build input (for instances frameworks from NuGets),
and in that case we want to copy everything to the Mac.
So this PR adds a parameter to the Ditto task to optionally copy the directory
from Windows for remote builds, and we enable this behavior when we want it -
specifically when copying frameworks.
Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009 while not
regressing https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635.
Ref: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009
Ref: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635
Ref: https://github.com/xamarin/xamarin-macios/pull/14375
Updated to 1.5.8, including fixes for a hang in build cancellation during SayHello and also an improvement in the port forwarding logic when establishing the SSH connection
Change dSYM generation and native stripping to occur immediately before code signing,
in a newly minted post processing target.
Challenges:
* Both calling 'strip' and 'codesign' on an executable modifies that executable,
which means that we must make sure to not call 'dsymutil' on the same binary at
a later point unless it's been rebuilt.
* Thus we must make sure to update 'dsymutil's stamp file whenever we call 'strip'
and/or 'codesign' on an executable.
* Just like for code signing, we must store the libraries (either static or dynamic)
we post process in extension/watch/rid-specific projects, so that these libraries
can be loaded in containing projects and processed there.
* In universal .NET builds, debug symbols are created for the universal app bundle,
not for each rid-specific version of the app bundle. So I had to add logic to create
the native symbol lists (MtouchSymbolsList) for each rid-specific build, but then
collect them and merge those lists for the universal app bundle.
The existing SymbolStrip call we did right after linking the native executable has
been removed, because we have to do that after creating the dSYM (which the GenerateDebugSymbols
target does).
Also add tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/14067.
Also establish when we should do post processing: only in the outermost build. This
is a slight change from previous behavior, where we'd run strip/dsymutil separately
for app extensions and watch apps.
* Save all the NativeReference metadata in binding resource packages.
* Copy all the NativeReference metadata to new items when resolving native references.
This makes it possible to set custom metadata on NativeReferences, and have that
metadata show up when it's needed, which might not be in the same project (for instance
if the native reference is in a binding project, we might want the custom metadata
when we load the native references from the binding project's resource package -
another case is when app extensions have native references, we might want any custom
metadata in the main executable project to know how to handle certain types of native
references).
Also sort the metadata we write to binding resource packages, so that the output
is stable. This required updating the corresponding tests.
The main theme here is that code signing will be done in the outermost
executable project, not in any app extension projects or watch projects, nor
during the RID-specific build of a .NET universal app. This makes codesigning
easier to reason about and other affected logic (such as strip/dsymutil)
easier to handle, in particular for .NET universal apps. Another benefit is
that the differences between the iOS and macOS code bases have been
eliminated.
The first step is to collect all the information we need from the targets
files. Every app bundle (be it app extension, watch app or main app) will add
its own output app bundle (.app/.appex) to the _CodesignBundle item group.
Then every app bundle will load this informarion from referenced app bundles,
and finally store this information on disk (in the 'codesign-bundle.items'
file). This means that in the end the main app bundle will have a list of all
contained app bundles in the app (recursively), in the _CodesignBundle item
group.
Separately we keep a list of other items that need signing, in the
_CodesignItems item group, and we do the same store/load logic for every
contained/contained app bundle (in the 'codesign.items' file, so a the end the
main app bundle will have a list of all the _CodesignItems for all contained
app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and
_StoreCodesigningData targets.
The next step is to use the new ComputeCodesignItems task to compute
everything we need to know for code signing. This task takes over the
responsibility for listing all the *.dylib and *.metallib files, and the
*.framework directories in the app bundles, that need signing (which was
previously done in the targets file). This logic is significantly easier to
write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use
to determine if something needs (re-)signing. Previously .framework
directories did not have a stamp location, so they'd always end up resigned in
a rebuild, while now we'll automatically skip signing *.framework directories
unless something changed in them.
I've also tried to comment everything thorougly, for the next poor soul having
to deal with any bugs.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same
thing for all platforms.
* We're now always signing *.framework directories for all platforms (like we
do for *.dylib files), since frameworks are pretty much like dylibs anyways.
I've verified that this works both by running the submission tests and running
and launching a sample project on device from Windows.
The ComputeCodesignItems does not touch any files, and all the input files
should already exist on the mac, so there's no need to copy files back and
forth.
Mac Catalyst and macOS projects support an 'EnableCodeSigning' property to determine
whether an app is signed or not. In order to bring parity on mobile platforms, add
support for this property for iOS, tvOS and watchOS projects as well - if not set,
we'll still have the old behavior of signing device builds and not signing simulator builds.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The CodesignEntitlements and CodesignResourceRules properties can be relative paths,
and they might be coming from a referenced project. This means that if they're relative
paths, we must resolve them to a full path using the project that defined them (which
is specified using the 'SourceProjectPath' metadata).
The main theme here is that code signing will be done in the outermost executable
project, not in any app extension projects or watch projects, nor during the RID-specific
build of a .NET universal app. This makes codesigning easier to reason about and
other affected logic (such as strip/dsymutil) easier to handle, in particular for
.NET universal apps. Another benefit is that the differences between the iOS and
macOS code bases have been eliminated.
The first step is to collect all the information we need from the targets files.
Every app bundle (be it app extension, watch app or main app) will add its own output
app bundle (.app/.appex) to the _CodesignBundle item group. Then every app bundle
will load this informarion from referenced app bundles, and finally store this information
on disk (in the 'codesign-bundle.items' file). This means that in the end the main
app bundle will have a list of all contained app bundles in the app (recursively),
in the _CodesignBundle item group.
Separately we keep a list of other items that need signing, in the _CodesignItems
item group, and we do the same store/load logic for every contained/contained app
bundle (in the 'codesign.items' file, so a the end the main app bundle will have
a list of all the _CodesignItems for all contained app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and _StoreCodesigningData
targets.
The next step is to use the new ComputeCodesignItems task to compute everything we
need to know for code signing. This task takes over the responsibility for listing
all the *.dylib and *.metallib files, and the *.framework directories in the app
bundles, that need signing (which was previously done in the targets file). This
logic is significantly easier to write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use to
determine if something needs (re-)signing. Previously .framework directories did
not have a stamp location, so they'd always end up resigned in a rebuild, while now
we'll automatically skip signing *.framework directories unless something changed
in them.
I've also tried to comment everything thorougly, for the next poor soul having to
deal with any bugs, as well has adding a comprehensive test for the new task.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same thing
for all platforms.
* We're now always signing *.framework directories for all platforms (like we do
for *.dylib files), since frameworks are pretty much like dylibs anyways.
I'm removing the old logic first, because the new logic is so different that a diff
just complicates understanding what's happening (there's not much value in comparing
textually what's changed when it's pretty much a complete rewrite).
* [DittoTask] If Source is a folder, add all its content as ITaskItem
* [DittoTask] Undoing undo...
* Accidentally hit Cmd+Z and removed an using statement
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Alex Soto <alex@alexsoto.me>
For reasons I don't quite understand, ditto might fail when executed by XMA:
Target Name=_CopyDirectoriesToBundle Project=C:\Users\rolf\source\iOSApp4\iOSApp4.csproj
Building target "_CopyDirectoriesToBundle" completely.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework/ArcGIS-arm64" does not exist.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\Runtimecore.framework/Runtimecore" does not exist.
Ditto
Assembly = C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\tools\msbuild\iOS\..\iOS\Xamarin.iOS.Tasks.dll
Parameters
Destination = bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework
TouchDestinationFiles = True
SessionId = <SessionId>
Source = C:\Users\rolf\.nuget\packages\esri.arcgisruntime.runtimes.ios\100.13.0\framework\ios-arm64\native\ArcGIS-arm64.framework\
Ditto: <timestamp> - Started
Ditto: <timestamp> - Initializing
[xma]: Trying to get a Build Connection for Session '<SessionId>': Xamarin.Messaging.Build.Client.BuildConnection.<SessionId>, Lifetime: Build
Ditto: <timestamp> - Initialized
Ditto: <timestamp> - There's no available inputs to copy to the Mac
Ditto: <timestamp> - Serializing intputs
Ditto: <timestamp> - Executing
[xma]: Starting remote task execution for 'iOSApp4': Xamarin.MacDev.Tasks.Ditto
[xma]: Sending Request Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic xvs/build/execute-task/iOSApp4/15f3833002fDitto
[xma]: Received Response of Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic build<SessionId>4396rolf/+/xvs/build/execute-task/iOSApp4/15f3833002fDitto
Ditto: <timestamp> - Logging messages
/usr/bin/ditto C:/Users/rolf/.nuget/packages/esri.arcgisruntime.runtimes.ios/100.13.0/framework/ios-arm64/native/ArcGIS-arm64.framework/ bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework
ditto: bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework: File exists
Errors
C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\targets\Xamarin.Shared.Sdk.targets(668,3): error MSB6006: "ditto" exited with code 1. [C:\Users\rolf\source\iOSApp4\iOSApp4.csproj]
Ditto: <timestamp> - Finished
This doesn't happen when building on macOS, nor if I copy the offending ditto
command and execute it manually on macOS.
Since I don't know why the problem occurs in the first place, I don't know why
passing full paths to 'ditto' works either. It shouldn't cause problems
elsewhere though.
Ref: https://github.com/xamarin/xamarin-macios/issues/13665
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to run dsymutil on at the same time.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to dsymutil, to make command lines
easier to copy-paste.
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to strip at the same time.
* Strip in parallel.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to strip, to make command lines
easier to copy-paste.
* Remove test that is now outdated. We have other tests that run strip
anyways, so this shouldn't be a problem.
This fixes an issue where we'd do logic with Windows-style paths on macOS, and that's
never the right thing to do.
For the LinkNativeCode task, this would manifest as this error when building from windows:
> ld: file too small (length=0) file 'obj/Debug/net6.0-ios/iossimulator-x64/nativelibraries/libSystem.Native.dylib' for architecture x86_64
because the 'ShouldCopyToBuildServer' method would return incorrect results.
For the Codesign task, it would manifest as an exception trying to create a
directory with an empty string (because the directory name of a windows-style
path is an empty string on macOS).
Since this exception was quite useless (just getting the exception message
didn't tell me much about what caused the exception, because it had no stack
trace information), I've also improved error reporting in both of these tasks.
Make the CollectBundleResourcesDependsOn property public, so that custom
targets can inject themselves into the build early enough to add additional
BundleResource or Content items (by adding their custom target's name to the
CollectBundleResourcesDependsOn property).
Fixes https://github.com/xamarin/xamarin-macios/issues/11984.
These properties aren't used.
This also allows us to remove the CompiledArchitectures output property from
the MTouch task, because it's not longer used anymore either.
Also change the key for our Info.plist entry with the version number in .NET, and document the change.
We now use "com.microsoft.<platform in lower case>" instead of "com.xamarin.ios" (for all platforms).
Fixes https://github.com/xamarin/xamarin-macios/issues/14108.
Co-authored-by: TJ Lambert <50846373+tj-devel709@users.noreply.github.com>