Improve error reporting when an external tool fails to print some of stderr (up to 1024 characters).
Before:
error : clang exited with code 1
After:
error : clang exited with code 1:
error : [...]/xamarin-macios/tests/dotnet/MyInterpretedApp/iOS/obj/Debug/net6.0-ios/iossimulator-x64/linker-cache/main.x86_64.mm:58:25: error: use of undeclared identifier 'MONO_AOT_MODE_INTERP_ONLY'; did you mean 'MONO_AOT_MODE_INTERP'?
error : mono_jit_set_aot_mode (MONO_AOT_MODE_INTERP_ONLY);
error : ^~~~~~~~~~~~~~~~~~~~~~~~~
error : MONO_AOT_MODE_INTERP
error : [...]/xamarin-macios/builds/downloads/dotnet-sdk-6.0.301-rtm.22254.17-osx-x64/packs/Microsoft.iOS.Runtime.iossimulator-x64/15.4.16-ci.x64-interpreter-only/runtimes/iossimulator-x64/native/xamarin/mono-runtime.h:452:2: note: 'MONO_AOT_MODE_INTERP' declared here
error : MONO_AOT_MODE_INTERP,
error : ^
* This is a potential mitigation for slower transition to native code when
exception marshalling is enabled (#14812).
* A minor modification was required in the linker, to make sure any modified
assemblies are saved.
Fixes https://github.com/xamarin/xamarin-macios/issues/4940.
Pick up --aot arguments in MtouchExtraArgs and pass them to the AOT compiler
when building a .NET project. This makes it possible to work around #14887 by
manually increasing the number of trampolines.
Ref: https://github.com/xamarin/xamarin-macios/issues/14887
When building a binding project, we need to execute bgen (and csc) on the mac. Figuring
out where these files are on the Mac is rather complicated from a remotely executed
task, so instead we execute a sub-build that computes these properties.
In legacy Xamarin this was accomplished by building the 'Xamarin.iOS.ObjCBinding.Common.props'
file using msbuild, and invoking a custom target that prints the property we're looking
for (the 'targetGetPropertyValue_*' targets).
For multiple reasons this approach doesn't work in .NET anymore (in particular it
seems that the 'Xamarin.iOS.ObjCBinding.Common.After.targets' file with the custom
'targetGetPropertyValue_*' targets is nowhere to be found, but logic has also moved
around in the .targets/.props files which makes just building the 'Xamarin.iOS.ObjCBinding.Common.props'
not work correctly since the properties we need wouldn't be set).
So I'm adding a new task that does a sub-build, using either msbuild or dotnet as
appropriate, to compute the properties we need. Instead of building the 'Xamarin.iOS.ObjCBinding.Common.props'
file, the task creates an actual binding project (an empty one), and executes the
new '_WriteRemoteGeneratorProperties' target in this binding project.
An additional advantage in this new task is that it will only execute one sub-build
where all the properties are computed (the previous approach executed one sub-msbuild
per property).
In order to keep code as similar as possible between legacy Xamarin and .NET, the
new task is being used for legacy Xamarin as well (and the old approach deleted).
This fixes building binding projects on Windows in .NET.
The output directory might or might not be where the app bundle is: by default
it is, but if someone sets the PkgPackageDir variable to provide an alternate
directory for the pkg, then that won't be where we'll find the app bundle.
The good news is that we already have a property that tells us where the app
bundle is (the 'AppBundleDir' property), so just use that instead.
Also:
* Make sure that the output directory exists before we try to write to it.
* Only pass full paths to productbuild, which for some reason doesn't seem to
like relative paths.
Fixes https://github.com/xamarin/xamarin-macios/issues/14751.
This also meant:
* Using 'latest' as the C# language version for all msbuild/ project files.
* Enabling warnaserror for nullability warnings.
* Fix any nullability warnings in the CompileAppManifest files.
* Fix a nullability warning in the Ditto task.
* Fix any '== null' or '!= null' to use 'is null' and 'is not null'.
Use 'Microsoft.<platform>' as the product name instead of 'Xamarin.[iOS|Mac]' for .NET builds. That makes error messages like this:
> [...]\Microsoft.iOS.Sdk\15.4.200-ci.windows2.62\tools\msbuild\iOS\Xamarin.Shared.targets(1676,3): Could not find Xamarin.iOS in /usr/local/share/dotnet/packs/Microsoft.iOS.Sdk/15.4.200-ci.windows2.62/.
look a bit better in .NET:
> [...]\Microsoft.iOS.Sdk\15.4.200-ci.windows2.62\tools\msbuild\iOS\Xamarin.Shared.targets(1676,3): Could not find Microsoft.iOS in /usr/local/share/dotnet/packs/Microsoft.iOS.Sdk/15.4.200-ci.windows2.62/.
* Move logic to validate the UIDeviceFamily value to shared code.
* Remove logic related to watchOS 1 apps, it's been dead for a while.
* Change logic to not overwriting any existing UIDeviceFamily values in the customer's
Info.plist.
This makes the code more platform-agnostic and easier to work with across all platforms
(such as adding new validations).
Fixes this warning from the Codesign task:
C:\Users\rolf\...\Microsoft.iOS.Sdk\15.4.200-...\tools\msbuild\iOS\Xamarin.Shared.targets(2045,3): Cannot create 'C:\Users\rolf\source\iOSApp4\bin\Debug\net6.0-ios\ios-arm64\device-builds\iphone14.2-15.3.1\iOSApp4.app\Frameworks\ArcGIS-arm64.framework' because a file or directory with the same name already exists.
C:\Users\rolf\...\Microsoft.iOS.Sdk\15.4.200-...\tools\msbuild\iOS\Xamarin.Shared.targets(2045,3): Cannot create 'C:\Users\rolf\source\iOSApp4\bin\Debug\net6.0-ios\ios-arm64\device-builds\iphone14.2-15.3.1\iOSApp4.app\Frameworks\Runtimecore.framework' because a file or directory with the same name already exists.
which occurs when the Codesign task asks XVS to create output files for files from
inside ditto'ed directories, and if XVS created output files for those directories
in the Ditto task, then XVS would be trying to create files inside these output files
as if they were directories. That doesn't work (thus the warning).
I've fixed this by:
* Removing the 'ShouldCreateOutputFile' implementation. The ShouldCreateOutputFile
method is called on Windows, and we can't determine from Windows whether the destination
is a directory or a file.
* Remove the [Output] attribute for the Destination property, this way XVS doesn't
automatically try to create an output file for whatever the destination is.
* Add another CopiedFiles output property, which contains all the copied files
(and only files), so that XVS mirrors this with output files on Windows.
Fixes part of https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1505990/.
When XVS creates output/stamp files on Windows, the paths must be relative,
because otherwise XVS will append the full macOS path to the current Windows
directory and get garbage.
XVS also expects only files as output, so don't return any directories.
Fixes part of https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1505990/.
* Fix resolving paths to required test files (test files can be found relative to the root path of the repository, not relative to where Xamarin.Mac is installed)
* Don't try to sign symlinks - we can end up trying to sign the target of the symlink twice simultaneously.
* Fix finding libxammac.dylib and Xamarin.Mac.dll when testing a system installation (when MAC_DESTDIR or TESTS_USE_SYSTEM are set).
* Remove a few .NET tests we don't need anymore.
Fixes https://github.com/xamarin/maccore/issues/2560.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Sometimes we want to copy the entire input directory from Windows to the Mac
when executing the Ditto task remotely, and sometimes we don't.
In particular we do not want to copy the input directory when the directory on
Windows is an incomplete mirror of what's on the Mac - one scenario being when
copying the app bundle to prepare for IPA creation. The .app directory on
Windows is not complete - all the files are there (maybe? not quite sure, but
that's beside the point here), but some may be empty, because when we only
care about the timestamp for a file, we'll create an empty file on Windows to
mirror the actual file on Mac. Copying this incomplete directory to the Mac,
overwriting the correct files there, will break things badly.
However, sometimes we're not mirroring a directory on Windows, but instead we
have directories as actual build input (for instances frameworks from NuGets),
and in that case we want to copy everything to the Mac.
So this PR adds a parameter to the Ditto task to optionally copy the directory
from Windows for remote builds, and we enable this behavior when we want it -
specifically when copying frameworks.
Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009 while not
regressing https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635.
Ref: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1506009
Ref: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1492635
Ref: https://github.com/xamarin/xamarin-macios/pull/14375
Change dSYM generation and native stripping to occur immediately before code signing,
in a newly minted post processing target.
Challenges:
* Both calling 'strip' and 'codesign' on an executable modifies that executable,
which means that we must make sure to not call 'dsymutil' on the same binary at
a later point unless it's been rebuilt.
* Thus we must make sure to update 'dsymutil's stamp file whenever we call 'strip'
and/or 'codesign' on an executable.
* Just like for code signing, we must store the libraries (either static or dynamic)
we post process in extension/watch/rid-specific projects, so that these libraries
can be loaded in containing projects and processed there.
* In universal .NET builds, debug symbols are created for the universal app bundle,
not for each rid-specific version of the app bundle. So I had to add logic to create
the native symbol lists (MtouchSymbolsList) for each rid-specific build, but then
collect them and merge those lists for the universal app bundle.
The existing SymbolStrip call we did right after linking the native executable has
been removed, because we have to do that after creating the dSYM (which the GenerateDebugSymbols
target does).
Also add tests.
Fixes https://github.com/xamarin/xamarin-macios/issues/14067.
* Save all the NativeReference metadata in binding resource packages.
* Copy all the NativeReference metadata to new items when resolving native references.
This makes it possible to set custom metadata on NativeReferences, and have that
metadata show up when it's needed, which might not be in the same project (for instance
if the native reference is in a binding project, we might want the custom metadata
when we load the native references from the binding project's resource package -
another case is when app extensions have native references, we might want any custom
metadata in the main executable project to know how to handle certain types of native
references).
Also sort the metadata we write to binding resource packages, so that the output
is stable. This required updating the corresponding tests.
The CodesignEntitlements and CodesignResourceRules properties can be relative paths,
and they might be coming from a referenced project. This means that if they're relative
paths, we must resolve them to a full path using the project that defined them (which
is specified using the 'SourceProjectPath' metadata).
The main theme here is that code signing will be done in the outermost executable
project, not in any app extension projects or watch projects, nor during the RID-specific
build of a .NET universal app. This makes codesigning easier to reason about and
other affected logic (such as strip/dsymutil) easier to handle, in particular for
.NET universal apps. Another benefit is that the differences between the iOS and
macOS code bases have been eliminated.
The first step is to collect all the information we need from the targets files.
Every app bundle (be it app extension, watch app or main app) will add its own output
app bundle (.app/.appex) to the _CodesignBundle item group. Then every app bundle
will load this informarion from referenced app bundles, and finally store this information
on disk (in the 'codesign-bundle.items' file). This means that in the end the main
app bundle will have a list of all contained app bundles in the app (recursively),
in the _CodesignBundle item group.
Separately we keep a list of other items that need signing, in the _CodesignItems
item group, and we do the same store/load logic for every contained/contained app
bundle (in the 'codesign.items' file, so a the end the main app bundle will have
a list of all the _CodesignItems for all contained app bundles (recursively).
The previous steps occur in the _CollectCodesigningData and _StoreCodesigningData
targets.
The next step is to use the new ComputeCodesignItems task to compute everything we
need to know for code signing. This task takes over the responsibility for listing
all the *.dylib and *.metallib files, and the *.framework directories in the app
bundles, that need signing (which was previously done in the targets file). This
logic is significantly easier to write, debug and test in C# than MSBuild.
In addition the ComputeCodesignItems also figures out a stamp file path we use to
determine if something needs (re-)signing. Previously .framework directories did
not have a stamp location, so they'd always end up resigned in a rebuild, while now
we'll automatically skip signing *.framework directories unless something changed
in them.
I've also tried to comment everything thorougly, for the next poor soul having to
deal with any bugs, as well has adding a comprehensive test for the new task.
Behavioral differences:
* We were always signing *.dylib files for macOS. We're now doing the same thing
for all platforms.
* We're now always signing *.framework directories for all platforms (like we do
for *.dylib files), since frameworks are pretty much like dylibs anyways.
For reasons I don't quite understand, ditto might fail when executed by XMA:
Target Name=_CopyDirectoriesToBundle Project=C:\Users\rolf\source\iOSApp4\iOSApp4.csproj
Building target "_CopyDirectoriesToBundle" completely.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework/ArcGIS-arm64" does not exist.
Output file "bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\Runtimecore.framework/Runtimecore" does not exist.
Ditto
Assembly = C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\tools\msbuild\iOS\..\iOS\Xamarin.iOS.Tasks.dll
Parameters
Destination = bin\Debug\net6.0-ios\iossimulator-x64\publish\..\device-builds\iphone11.6-14.8.1\iOSApp4.app\\Frameworks\\ArcGIS-arm64.framework
TouchDestinationFiles = True
SessionId = <SessionId>
Source = C:\Users\rolf\.nuget\packages\esri.arcgisruntime.runtimes.ios\100.13.0\framework\ios-arm64\native\ArcGIS-arm64.framework\
Ditto: <timestamp> - Started
Ditto: <timestamp> - Initializing
[xma]: Trying to get a Build Connection for Session '<SessionId>': Xamarin.Messaging.Build.Client.BuildConnection.<SessionId>, Lifetime: Build
Ditto: <timestamp> - Initialized
Ditto: <timestamp> - There's no available inputs to copy to the Mac
Ditto: <timestamp> - Serializing intputs
Ditto: <timestamp> - Executing
[xma]: Starting remote task execution for 'iOSApp4': Xamarin.MacDev.Tasks.Ditto
[xma]: Sending Request Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic xvs/build/execute-task/iOSApp4/15f3833002fDitto
[xma]: Received Response of Xamarin.Messaging.Build.Contracts.ExecuteTaskMessage to topic build<SessionId>4396rolf/+/xvs/build/execute-task/iOSApp4/15f3833002fDitto
Ditto: <timestamp> - Logging messages
/usr/bin/ditto C:/Users/rolf/.nuget/packages/esri.arcgisruntime.runtimes.ios/100.13.0/framework/ios-arm64/native/ArcGIS-arm64.framework/ bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework
ditto: bin/Debug/net6.0-ios/iossimulator-x64/publish/../device-builds/iphone11.6-14.8.1/iOSApp4.app//Frameworks//ArcGIS-arm64.framework: File exists
Errors
C:\Users\rolf\source\maui\bin\dotnet\packs\Microsoft.iOS.Sdk\15.2.303-ci.ditto-windows.56\targets\Xamarin.Shared.Sdk.targets(668,3): error MSB6006: "ditto" exited with code 1. [C:\Users\rolf\source\iOSApp4\iOSApp4.csproj]
Ditto: <timestamp> - Finished
This doesn't happen when building on macOS, nor if I copy the offending ditto
command and execute it manually on macOS.
Since I don't know why the problem occurs in the first place, I don't know why
passing full paths to 'ditto' works either. It shouldn't cause problems
elsewhere though.
Ref: https://github.com/xamarin/xamarin-macios/issues/13665
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to run dsymutil on at the same time.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to dsymutil, to make command lines
easier to copy-paste.
* Enable nullability and fix code accordingly.
* Augment it to be able to take multiple files to strip at the same time.
* Strip in parallel.
* Execute using xcrun (ref: #3931)
* Pass the full path to the executable file to strip, to make command lines
easier to copy-paste.
* Remove test that is now outdated. We have other tests that run strip
anyways, so this shouldn't be a problem.
This fixes an issue where we'd do logic with Windows-style paths on macOS, and that's
never the right thing to do.
For the LinkNativeCode task, this would manifest as this error when building from windows:
> ld: file too small (length=0) file 'obj/Debug/net6.0-ios/iossimulator-x64/nativelibraries/libSystem.Native.dylib' for architecture x86_64
because the 'ShouldCopyToBuildServer' method would return incorrect results.
For the Codesign task, it would manifest as an exception trying to create a
directory with an empty string (because the directory name of a windows-style
path is an empty string on macOS).
Since this exception was quite useless (just getting the exception message
didn't tell me much about what caused the exception, because it had no stack
trace information), I've also improved error reporting in both of these tasks.
These properties aren't used.
This also allows us to remove the CompiledArchitectures output property from
the MTouch task, because it's not longer used anymore either.
Also change the key for our Info.plist entry with the version number in .NET, and document the change.
We now use "com.microsoft.<platform in lower case>" instead of "com.xamarin.ios" (for all platforms).
Fixes https://github.com/xamarin/xamarin-macios/issues/14108.
Co-authored-by: TJ Lambert <50846373+tj-devel709@users.noreply.github.com>
* Remove ObjCRuntime.nfloat (in favor of System.Runtime.InteropServices.NFloat).
* Automatically add a reference to the System.Runtime.InteropServices.Internal
package, so that developers get the new NFloat API (with operators) we've
added post .NET 6 (but don't do this for .NET 7).
* Automatically add a global using alias for
System.Runtime.InteropServices.NFloat -> nfloat. This is not behind the
usual `ImplicitUsings` condition our other implicit usings are, because
they're off by default for existing projects, and the main target for the
global using alias for nfloat is upgraded projects.
* Automatically generate a global using alias (like above) in the generator
for all code the generator compiles.
* Update xtro entries to reference System.Runtime.InteropServices.NFloat
instead of ObjCRuntime.nfloat.
* Add a workaround for a hopefully temporary issue with .NET/CoreCLR where the
wrong runtime pack is selected otherwise (without the new NFloat API, so
nothing works at runtime).
Ref: https://github.com/xamarin/xamarin-macios/issues/13087
Add support for the PublishFolderType metadata on Content and BundleResource
items, which makes it possible to change the location in the app bundle for
these items (this was possible to do before with the Link metadata), but most
importantly it also makes it possible to choose to *not* bundle these items in
the app bundle (which was not possible before with the Link metadata, nor any
other means).
At first I thought setting CopyToPublishDirectory /
CopyToOutputDirectory=Never would accomplish that, but it turns out we don't
honor those, and since we already have this behavior of ignoring
CopyToPublishDirectory / CopyToOutputDirectory in legacy Xamarin, I didn't
want to change it in .NET.
So I added support for honoring the PublishFolderType metadata instead, which
is new and we already support for other item groups. This is accomplished by
adding all Content and BundleResource items with PublishFolderType set to the
ResolvedFileToPublish item group (where we already handle any
PublishFolderType values), and then we ignore such Content and BundleResource
items in our CollectBundleResources task.
Also update the documentation and add tests.
* Add support for specifying metadata on items that are to be codesigned to
override any general codesign setting.
* Make the Codesign task able to sign files and directories that may depend on
eachother.
Implement support for ordering signing so that directories containing files
that also must be signed are signed after those files.
This is implemented by:
1. Normalize all input (resolve symlinks, create full path, etc.)
2. Sort by path length (longest to shortest paths). This way we're certain
that if we find a directory, we'll know that we won't find any files
later in the list from inside that directory.
3. Group into an ordered list of buckets, where each bucket contains files
and directories that don't depend on eachother (i.e. they can all be
signed in parallel).
This makes it possible to call Codesign once, listing both the app bundle
itself, and all the individual files or directories inside that need
signing, and the Codesign task will sign the items in an order that ensures
parent directories are always signed after any files or directories inside.
* Finally rework code signing to sign everything with a single call to the
Codesign task in the _CodesignAppBundle target for the executable project,
instead of having multiple calls to the Codesign task from multiple targets
(and projects). This makes it easier to reason about what's being signed,
and it also makes it easier to add files to the signing process.
* Also make the Codesign task able to figure out if something needs to be
signed, and change the _CodesignAppBundle target to not keep track of
inputs/outputs, because it becomes quite complex (for directories, it needs
to keep track of all the files inside that directory, and also if there's
anything in the directory that's also being signed).
This fixes the following problem:
* App with framework is built and signed.
* App is rebuilt, and the framework is copied in again.
* This time, the framework's executable's timestamp will be earlier than the
timestamp when it was last signed, and as such it won't be signed again.
Fix this by touching all the copied files when copying a directory to the app bundle.
The symlink might point to a file that has been updated, and if that file
needs more processing and the logic checks the symlink to check if the file is
up-to-date, then we need to reflect that in the symlink (for instance if a
framework binary is updated (which is a symlink on some platforms), it needs
to be resigned, even if the symlink itself didn't change).
Rolf Bjarne Kvinge
Manuel de la Pena
Israel Soto