We currently have an issue with codesigning duplicated files in parallel
(#20193). The proper fix is somewhat complex, so this implements a simple
workaround for customers, where they can just disable parallelism if they run
into this problem (and since it's simple, it's easier to backport too).
Additionally, it's not a bad idea to be able to configure the level of parallelism.
Ref: #20193.
* Enumerate files and directories instead of fetching them all in one go. This
is more efficient when there are a lot of files and directories to process,
but most importantly it'll also make enumeration cancellable.
* Stop enumerating if cancellation was requested.
* Add a few logging statements.
This might at least help to get better diagnostic information if for some
reason this task takes too long to execute. Ref: #20183.
#### Background
While cyclic assembly references are unusual they are supported by the runtime and encountered in practice. In our project we use a version of Xamarin.Forms retargeted for a modern .NET (as a stopgap solution before an update to full MAUI stack). Xamarin.Forms historically come with such an assembly cycle: `Xamarin.Forms.Core.dll` -> `Xamarin.Forms.Platform.dll` -> `Xamarin.Forms.Platform.iOS.dll` -> `Xamarin.Forms.Core.dll`. This is produced by first compiling `Xamarin.Forms.Core.dll` against a dummy `Xamarin.Forms.Platform.dll` (`netstandard2.0` assembly). Then the Platform assemblies are built, and finally forwarder versions of `Xamarin.Forms.Platform.dll` are created for each platform. This is all packaged into NuGets in such a way that each TFM gets the same `Xamarin.Forms.Core.dll` but a different set of `Xamarin.Forms.Platform.dll` and `Xamarin.Forms.Platform.<platform>.dll` assemblies.
#### Problem
With current workloads each incremental rebuild fails with:
```
/usr/local/share/dotnet/packs/Microsoft.iOS.Sdk/16.4.7125/targets/Xamarin.Shared.Sdk.targets(1047,3): error : Encountered an assembly reference cycle related to the assembly obj/Debug/net7.0-ios/iossimulator-arm64/linked/Xamarin.Forms.Core.dll
```
#### Solution
The PR updates the algorithm in `AOTCompile` to detect cycles using [Tarjan's algorithm](https://en.wikipedia.org/wiki/Tarjan%27s_strongly_connected_components_algorithm) for finding [strongly connected components](https://en.wikipedia.org/wiki/Strongly_connected_component) in a graph. When a cycle is detect any assembly in the cycle is considered up-to-date only if all the assemblies in the cycle are up-to-date.
With this change the project does incremental rebuilds correctly. I verified in the build output that the assembly cycle is considered up-to-date, and that any changes to the application assemblies still return `false` from the up-to-date check and get rebuilt.
---------
Co-authored-by: Filip Navara <filip.navara@gmail.com>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
This fixes makes this rather unhelpful error:
error MSB4018: The "CreateBindingResourcePackage" task failed unexpectedly.
error MSB4018: System.Exception: Could not lstat 'mylib.a': 2
error MSB4018: at Xamarin.Utils.PathUtils.IsSymlink(String file) in /Users/rolf/work/maccore/main/xamarin-macios/tools/common/PathUtils.cs:line 222
error MSB4018: at Xamarin.Utils.PathUtils.IsSymlinkOrContainsSymlinks(String directoryOrFile) in /Users/rolf/work/maccore/main/xamarin-macios/tools/common/PathUtils.cs:line 229
error MSB4018: at Xamarin.MacDev.Tasks.CreateBindingResourcePackageBase.ContainsSymlinks(ITaskItem[] items) in /Users/rolf/work/maccore/main/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/CreateBindingResourcePackageBase.cs:line 110
error MSB4018: at Xamarin.MacDev.Tasks.CreateBindingResourcePackageBase.Execute() in /Users/rolf/work/maccore/main/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/CreateBindingResourcePackageBase.cs:line 46
error MSB4018: at Xamarin.MacDev.Tasks.CreateBindingResourcePackage.Execute() in /Users/rolf/work/maccore/main/xamarin-macios/msbuild/Xamarin.MacDev.Tasks/Tasks/CreateBindingResourcePackage.cs:line 15
error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)
turn into this instead:
error : The NativeResource 'mylib.a' does not exist.
---------
Co-authored-by: Haritha Mohan <harithamohan@microsoft.com>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
If an assembly changes, then we must AOT compile that assembly again (which we already
did), in addition to any assembly that references the modified assembly (which we
didn't do).
So rework the AOTCompile target: remove the Inputs and Outputs (because the dependency
tracking is too complicated for MSBuild to resolve), and instead move the logic to
detect if an assembly must be AOT-compiled again into the AOTCompile task.
Note that this PR has a custom port to .NET 8: #18518.
Fixes https://github.com/xamarin/xamarin-macios/issues/17708.
---------
Co-authored-by: Alex Soto <alex@alexsoto.me>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
When building from Windows, we need to pass the path to the illink
assembly located on the Mac to the linker task. The educated guess we've
been using is a bit fragile and has been getting us problems almost on
each new .NET major release. On top of that, from .NET 8 the linker is
in a separate NuGet package, so the assembly is no longer located in the
SDK directory on the Mac.
The fix is to follow the same approach we use to find the AOT compiler
on the Mac by running a target that finds that information on the remote
Mac, and brings it back to Windows, where it is cached and use across
build.
Created a new XamarinBuildTask class to share most of the code needed
for this approach.
---------
Co-authored-by: GitHub Actions Autoformatter <github-actions-autoformatter@xamarin.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
---------
Co-authored-by: GitHub Actions Autoformatter <github-actions-autoformatter@xamarin.com>
Co-authored-by: tj-devel709 <tjlambert@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
---------
Co-authored-by: tj-devel709 <tjlambert@microsoft.com>
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/icxLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
Make the CompileAppManifest task work on Windows by:
* Not requiring the default SDK version (impossible to calculate without Xcode)
as an input property (we still require it when it's needed).
* Not requiring the SDK version (also impossible to calculate without Xcode).
* Using the minimum OS version we support as the as minimum OS version the app
supports if building on Windows and an SDK version has not been provided.
* Not adding any of the app manifest values we get from Xcode.
Previously, we'd do this:
* Collect all possible native references.
* Extract any compressed native references (*.framework.zip, *.xcframework.zip,
*.resources.zip) to disk.
* Resolve the resulting native references.
This doesn't work very well on Windows (in non-connected/Hot Restart mode),
because some compressed files may contain symlinks (in particular compressed
xcframeworks). If those symlinks are for any other platform than the one we're
building for, they shouldn't matter, but if we extract the entire compressed
xcframework before figuring out what we need from it, we'd run into symlinks
and not knowing whether they should be ignored or not.
So rework the process to:
* Collect all possible native references.
* Resolve the resulting native references, peeking into zip files if need be.
* Extract any compressed native references, but only the parts of the zip we need.
This way we won't run into any symlinks unless we really need them, and it
should also improve build performance slightly, even on macOS, since we're not
extracting files we won't need (which can be significant for xcframeworks).
Additionally:
* Add support for unzipping on Windows by using System.IO.Compression.
* Show an error if attempting to extract a symlink in the last step in the
reworked process on Windows.
* Some tests had to be updated (since they poked into internals of the
ResolveNativeReferences task, and those internals have changed).
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/ceLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
This is the pull request automatically created by the OneLocBuild task
in the build process to check-in localized files generated based upon
translation source files (.lcl files) handed-back from the downstream
localization pipeline. If there are issues in translations, visit
https://aka.ms/ceLocBug and log bugs for fixes. The OneLocBuild wiki is
https://aka.ms/onelocbuild and the localization process in general is
documented at https://aka.ms/AllAboutLoc.
This fixes an issue where this file would show up as modified after the
build, which breaks our API comparison.
Backport of #16476
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Hopefully this fixes some of the recurrent build problems where we'd fail to
restore nuget packages.
Ref: https://github.com/xamarin/maccore/issues/2620
I wasn't able to execute ilrepack.exe with .NET, so that continues to be
executed with mono.
Fixes an issue where we'd end up trying to link with the managed assembly instead:
> ld: warning: ignoring file .../bin/Debug/net6.0-ios/MyBinding.dll, building for iOS Simulator-x86_64 but attempting to link with file built for unknown-unsupported file format ( 0x4D 0x5A 0x90 0x00 0x03 0x00 0x00 0x00 0x04 0x00 0x00 0x00 0xFF 0xFF 0x00 0x00 )
Also add a warning if we run into other types of libraries in the future to
make it easier to diagnose.
* Add support for specifying custom entitlements with an MSBuild item group.
* Use this new support to automatically add the 'com.apple.security.cs.allow-jit'
entitlement to .NET desktop apps when building for release, since all apps that
go through notarization will need it in order to be able to use the JIT.
It's possible to override the default behavior by adding something like this to the project file:
<ItemGroup>
<CustomEntitlements Include="com.apple.security.cs.allow-jit" Type="Remove" />
</ItemGroup>
Fixes https://github.com/xamarin/xamarin-macios/issues/15745.
Fixes this compiler warning:
/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/Microsoft.Common.CurrentVersion.targets(1733,5): warning : ProjectReference 'xamarin-macios/msbuild/Messaging/Xamarin.Messaging.Build/Xamarin.Messaging.Build.csproj' was resolved using '.NETFramework,Version=v4.7.2' instead of the project target framework '.NETStandard,Version=v2.0'. This project may not be fully compatible with your project. [xamarin-macios/msbuild/Xamarin.iOS.Tasks.Windows/Xamarin.iOS.Tasks.Windows.csproj]
Improve error reporting when an external tool fails to print some of stderr (up to 1024 characters).
Before:
error : clang exited with code 1
After:
error : clang exited with code 1:
error : [...]/xamarin-macios/tests/dotnet/MyInterpretedApp/iOS/obj/Debug/net6.0-ios/iossimulator-x64/linker-cache/main.x86_64.mm:58:25: error: use of undeclared identifier 'MONO_AOT_MODE_INTERP_ONLY'; did you mean 'MONO_AOT_MODE_INTERP'?
error : mono_jit_set_aot_mode (MONO_AOT_MODE_INTERP_ONLY);
error : ^~~~~~~~~~~~~~~~~~~~~~~~~
error : MONO_AOT_MODE_INTERP
error : [...]/xamarin-macios/builds/downloads/dotnet-sdk-6.0.301-rtm.22254.17-osx-x64/packs/Microsoft.iOS.Runtime.iossimulator-x64/15.4.16-ci.x64-interpreter-only/runtimes/iossimulator-x64/native/xamarin/mono-runtime.h:452:2: note: 'MONO_AOT_MODE_INTERP' declared here
error : MONO_AOT_MODE_INTERP,
error : ^
When building a binding project, we need to execute bgen (and csc) on the mac. Figuring
out where these files are on the Mac is rather complicated from a remotely executed
task, so instead we execute a sub-build that computes these properties.
In legacy Xamarin this was accomplished by building the 'Xamarin.iOS.ObjCBinding.Common.props'
file using msbuild, and invoking a custom target that prints the property we're looking
for (the 'targetGetPropertyValue_*' targets).
For multiple reasons this approach doesn't work in .NET anymore (in particular it
seems that the 'Xamarin.iOS.ObjCBinding.Common.After.targets' file with the custom
'targetGetPropertyValue_*' targets is nowhere to be found, but logic has also moved
around in the .targets/.props files which makes just building the 'Xamarin.iOS.ObjCBinding.Common.props'
not work correctly since the properties we need wouldn't be set).
So I'm adding a new task that does a sub-build, using either msbuild or dotnet as
appropriate, to compute the properties we need. Instead of building the 'Xamarin.iOS.ObjCBinding.Common.props'
file, the task creates an actual binding project (an empty one), and executes the
new '_WriteRemoteGeneratorProperties' target in this binding project.
An additional advantage in this new task is that it will only execute one sub-build
where all the properties are computed (the previous approach executed one sub-msbuild
per property).
In order to keep code as similar as possible between legacy Xamarin and .NET, the
new task is being used for legacy Xamarin as well (and the old approach deleted).
This fixes building binding projects on Windows in .NET.
* Add support for specifying metadata on items that are to be codesigned to
override any general codesign setting.
* Make the Codesign task able to sign files and directories that may depend on
eachother.
Implement support for ordering signing so that directories containing files
that also must be signed are signed after those files.
This is implemented by:
1. Normalize all input (resolve symlinks, create full path, etc.)
2. Sort by path length (longest to shortest paths). This way we're certain
that if we find a directory, we'll know that we won't find any files
later in the list from inside that directory.
3. Group into an ordered list of buckets, where each bucket contains files
and directories that don't depend on eachother (i.e. they can all be
signed in parallel).
This makes it possible to call Codesign once, listing both the app bundle
itself, and all the individual files or directories inside that need
signing, and the Codesign task will sign the items in an order that ensures
parent directories are always signed after any files or directories inside.
* Finally rework code signing to sign everything with a single call to the
Codesign task in the _CodesignAppBundle target for the executable project,
instead of having multiple calls to the Codesign task from multiple targets
(and projects). This makes it easier to reason about what's being signed,
and it also makes it easier to add files to the signing process.
* Also make the Codesign task able to figure out if something needs to be
signed, and change the _CodesignAppBundle target to not keep track of
inputs/outputs, because it becomes quite complex (for directories, it needs
to keep track of all the files inside that directory, and also if there's
anything in the directory that's also being signed).
Augment the CreateBindingResourcePackage to support creating a zipped binding
resource package (which is just a zipped version of the binding resource
package). This can either be manually chosen by the new 'Compressed' property,
or automatically detected (create a zipped version when there's a symlink in
the binding resource package).
The default is to not create a zipped version in legacy Xamarin, and
automatically detect for .NET.
The problem this is trying to solve is when creating a NuGet package - NuGet
doesn't handle symlinks correctly and it's not possible to create a NuGet with
symlinks. Instead we need to create a zipfile with all the binding resources.
The default has been chosen so that we automatically create a zip file when
it's required for .NET, while still maintaining old behavior with legacy
Xamarin.
Rolf Bjarne Kvinge
Filip Navara
VS MobileTools Engineering Service 2
Emanuel Fernandez Dell'Oca