[net9.0] Reenable dedup optimization for all AOT modes (#20941)

Backport of #20936

---

## Description

As part of the fix for: https://github.com/dotnet/runtime/issues/99248
we disabled dedup optimization in partial/hybrid AOT mode (when both
interpreter and AOT compiler are enabled). This change got backported to
.NET 8 and with the latest servicing release regressed build times and
app sizes significantly as reported in:
https://github.com/xamarin/xamarin-macios/issues/20848

However, it turns out that disabling dedup optimization is not required
to fix https://github.com/dotnet/runtime/issues/99248 but instead we
should correct the Xamarin SDK integration with this optimization which
this PR is doing. The following section describes the initial problem in
more details.

## Overview of AOT modes and dedup optimization

When the repro project from
https://github.com/dotnet/runtime/issues/99248 is built with dedup
enabled in hybrid AOT+interpreter mode, the app crashes with:
```
024-07-23 14:32:37.524110+0200 IvansApp[12711:20244208] debug: AOT NOT FOUND: (wrapper other) object:gsharedvt_out_sig (intptr).
2024-07-23 14:32:37.524120+0200 IvansApp[12711:20244208] error: * Assertion at /Users/ivan/repos/runtime-mono-iOS/src/mono/mono/mini/interp/interp.c:2667, condition `is_ok (error)' not met, function:init_jit_call_info, Attempting to JIT compile method '(wrapper other) void object:gsharedvt_out_sig (intptr)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information.
```

To track down why these wrappers which are used to transition from
interpreter to AOT code, are not generated we need to understand when
they are compiled in different AOT modes with and without dedup
optimization enabled:

- In full AOT setup - all assemblies AOT compiled
    - `gsharedvt_out_sig` methods are never generated

- In hybrid AOT + interpreter setup - all assemblies AOT compiled:
`MtouchInterpreter=-all`
    - Dedup OFF:
- `gsharedvt_out_sig` methods are generated in AOT images of every
assembly (to enable interpreter calling into each specific assembly -
here wrappers with same signatures are duplicated)
    - Dedup ON:
- `gsharedvt_out_sig` methods are generated only in `aot-instances` AOT
image
- during AOT compilation of individual assemblies generation of
`gsharedvt_out_sig` is skipped
- during AOT compilation of `aot-instances` assembly we collect all
`gsharedvt_out_sig` variants from the full program scope and generate
code for them in `aot-instances` AOT image

- In hybrid AOT + interpreter setup - all assemblies interpreted except
a given assembly: `MtouchInterpreter=all,-MyAssembly`
    - Dedup OFF:
- `gsharedvt_out_sig` methods are generated in AOT image of `MyAssembly`
(to enable interpreter calling into it)
    - Dedup ON: <- $${\color{red} ISSUE }$$ 
- `gsharedvt_out_sig` methods *should* be generated only in
`aot-instances` AOT image, but the `aot-instances` image is missing
    - explanation:
- what happens is that generation of `gsharedvt_out_sig` is skipped
during AOT compilation of `MyAssembly` (as expected).
- But, the build does not mark `aot-instances` assembly as the one that
should be AOT compiled.
- The reason for this is that we have a global `_IsDedupEnabled` flag,
but when custom linker step analysis `aot-instances.dll` it does not see
it as an assembly which should not be interpreted.
- To explain that better: we mark *all* assemblies as to be interpreted
(via: `MtouchInterpreter=all`), but exclude only `MyAssembly` (via:
`MtouchInterpreter=all,-MyAssembly`).
- So when custom linker step processes `aot-instaces.dll` it treats it
as an assembly to be interpreted, so it does not mark it for AOT
compilation.
- This further results with `aot-instances` AOT image missing, and all
the methods which we skipped during AOT compilation never get generated.

## The fix

To fix this and address regressions reported in:
https://github.com/xamarin/xamarin-macios/issues/20848 we are reenabling
dedup optimization whenever AOT compilation is requested and fixing the
issue where the custom linker step for generating AOT parameters always
treates the dedup assembly as the one to be AOTed.

Once approved this should be backported to .NET 8 as servicing releases
are also affected with it.

---------

Co-authored-by: Ivan Povazan <ivan.povazan@gmail.com>
Co-authored-by: GitHub Actions Autoformatter <github-actions-autoformatter@xamarin.com>

dotnet/targets/Xamarin.Shared.Sdk.targets

@@ -1091,8 +1091,14 @@
 
 			<_AOTInputDirectory>$(_IntermediateNativeLibraryDir)aot-input/</_AOTInputDirectory>
 			<_AOTOutputDirectory>$(_IntermediateNativeLibraryDir)aot-output/</_AOTOutputDirectory>
-			<!-- Enable dedup optimization only in FullAOT mode -->
-			<_IsDedupEnabled Condition="'$(_IsDedupEnabled)' == '' And '$(_RunAotCompiler)' == 'true' And '$(MtouchInterpreter)' == '' And '$(IsMacEnabled)' == 'true'">true</_IsDedupEnabled>
+
+			<!-- 
+				Enable dedup optimization whenever we run AOT compiler
+				The only exception is maccatalyst-x64 which uses AOT when the interpreter is also enabled.
+				In such setup, during runtime, AOT images are loaded but marked as unusuable as they are compiled with `full` compiler switch and the code fallsback to interpreter.
+				Dedup AOT image is specially handled by the AOT runtime and it is not allowed to have it marked as unusuable.
+			-->
+			<_IsDedupEnabled Condition="'$(_IsDedupEnabled)' == '' And '$(_RunAotCompiler)' == 'true' And '$(IsMacEnabled)' == 'true' And '$(RuntimeIdentifier)' != 'maccatalyst-x64'">true</_IsDedupEnabled>
 			<_DedupAssembly Condition="'$(_IsDedupEnabled)' == 'true'">$(IntermediateOutputPath)aot-instances.dll</_DedupAssembly>
 
 			<!-- default to 'static' for Mac Catalyst to work around https://github.com/xamarin/xamarin-macios/issues/14686 -->

tests/dotnet/UnitTests/ProjectTest.cs

@@ -2006,10 +2006,11 @@ namespace Xamarin.Tests {
 			}
 		}
 
-		bool FindAssembly (string path, string dllName)
+		bool FindAOTedAssemblyFile (string path, string dllName)
 		{
-			foreach (string file in Directory.GetFiles (path, "*.dll", SearchOption.AllDirectories)) {
-				if (Path.GetFileName (file).Equals (dllName, StringComparison.OrdinalIgnoreCase)) {
+			var aotedAssemblyFileName = $"{dllName}.o";
+			foreach (string file in Directory.GetFiles (path, "*.o", SearchOption.AllDirectories)) {
+				if (Path.GetFileName (file).Equals (aotedAssemblyFileName, StringComparison.OrdinalIgnoreCase)) {
 					return true;
 				}
 			}
@@ -2020,11 +2021,17 @@ namespace Xamarin.Tests {
 		[Test]
 		[TestCase (ApplePlatform.iOS, "ios-arm64;", "-all,System.Private.CoreLib")]
 		[TestCase (ApplePlatform.iOS, "ios-arm64;", "all,-System.Private.CoreLib")]
+		[TestCase (ApplePlatform.iOS, "ios-arm64;", "-all")]
+		[TestCase (ApplePlatform.iOS, "ios-arm64;", "")]
 		[TestCase (ApplePlatform.MacCatalyst, "maccatalyst-arm64;maccatalyst-x64", "-all,System.Private.CoreLib")]
 		[TestCase (ApplePlatform.MacCatalyst, "maccatalyst-arm64;maccatalyst-x64", "all,-System.Private.CoreLib")]
+		[TestCase (ApplePlatform.MacCatalyst, "maccatalyst-arm64;maccatalyst-x64", "-all")]
+		[TestCase (ApplePlatform.MacCatalyst, "maccatalyst-arm64;maccatalyst-x64", "")]
 		[TestCase (ApplePlatform.TVOS, "tvos-arm64;", "-all,System.Private.CoreLib")]
 		[TestCase (ApplePlatform.TVOS, "tvos-arm64;", "all,-System.Private.CoreLib")]
-		public void PartialAOTTest (ApplePlatform platform, string runtimeIdentifiers, string mtouchInterpreter)
+		[TestCase (ApplePlatform.TVOS, "tvos-arm64;", "-all")]
+		[TestCase (ApplePlatform.TVOS, "tvos-arm64;", "")]
+		public void DedupEnabledTest (ApplePlatform platform, string runtimeIdentifiers, string mtouchInterpreter)
 		{
 			var project = "MySimpleApp";
 			Configuration.IgnoreIfIgnoredPlatform (platform);
@@ -2037,7 +2044,16 @@ namespace Xamarin.Tests {
 
 			DotNet.AssertBuild (project_path, properties);
 
-			Assert.True (!FindAssembly (appPath, "aot-instances.dll"), "Dedup optimization shouldn't been enabled for partial AOT compilation");
+			var objDir = GetObjDir (project_path, platform, runtimeIdentifiers);
+			if (platform == ApplePlatform.MacCatalyst) {
+				var objDirMacCatalystArm64 = Path.Combine (objDir, "maccatalyst-arm64");
+				Assert.True (FindAOTedAssemblyFile (objDirMacCatalystArm64, "aot-instances.dll"), $"Dedup optimization should be enabled for AOT compilation on: {platform} with RID: maccatalyst-arm64");
+
+				var objDirMacCatalystx64 = Path.Combine (objDir, "maccatalyst-x64");
+				Assert.False (FindAOTedAssemblyFile (objDirMacCatalystx64, "aot-instances.dll"), $"Dedup optimization should not be enabled for AOT compilation on: {platform} with RID: maccatalyst-x64");
+			} else {
+				Assert.True (FindAOTedAssemblyFile (objDir, "aot-instances.dll"), $"Dedup optimization should be enabled for AOT compilation on: {platform} with RID: {runtimeIdentifiers}");
+			}
 
 			var appExecutable = GetNativeExecutable (platform, appPath);
 

tools/common/Application.cs

@@ -1546,7 +1546,7 @@ namespace Xamarin.Bundler {
 			bool enable_debug = app.EnableDebug;
 			bool enable_debug_symbols = app.PackageManagedDebugSymbols;
 			bool llvm_only = app.EnableLLVMOnlyBitCode;
-			bool interp = app.IsInterpreted (Assembly.GetIdentity (filename));
+			bool interp = app.IsInterpreted (Assembly.GetIdentity (filename)) && !(isDedupAssembly.HasValue && isDedupAssembly.Value);
 			bool interp_full = !interp && app.UseInterpreter;
 			bool is32bit = (abi & Abi.Arch32Mask) > 0;
 			string arch = abi.AsArchString ();

tools/common/Assembly.cs

@@ -774,15 +774,17 @@ namespace Xamarin.Bundler {
 			}
 		}
 
+		public bool IsDedupAssembly { get; set; } = false;
+
 		public bool IsInterpreted {
 			get {
-				return App.IsInterpreted (Identity);
+				return IsDedupAssembly ? false : App.IsInterpreted (Identity);
 			}
 		}
 
 		public bool IsAOTCompiled {
 			get {
-				return App.IsAOTCompiled (Identity);
+				return IsDedupAssembly ? true : App.IsAOTCompiled (Identity);
 			}
 		}
 	}

tools/dotnet-linker/Steps/LoadNonSkippedAssembliesStep.cs

@@ -1,4 +1,5 @@
 using System;
+using System.IO;
 
 using Mono.Cecil;
 using Mono.Linker;
@@ -28,6 +29,7 @@ namespace Xamarin.Linker {
 				case AssemblyAction.Save:
 					var ad = Configuration.Target.AddAssembly (assembly);
 					var assemblyFileName = Configuration.GetAssemblyFileName (assembly);
+					ad.IsDedupAssembly = Path.GetFileName (Configuration.DedupAssembly).Equals (Path.GetFileName (assemblyFileName), StringComparison.OrdinalIgnoreCase);
 					ad.FullPath = assemblyFileName;
 					break;
 				case AssemblyAction.AddBypassNGen: // This should be turned into Save or Delete