Change reflection defaults (#6820)

Documentation/using-corert/optimizing-corert.md

@@ -0,0 +1,11 @@
+# Optimizing programs targeting CoreRT
+
+The CoreRT compiler provides multiple switches to influence the compilation process. These switches control the code and metadata that the compiler generates and affect the runtime behavior of the compiled program.
+
+## Options related to reflection
+
+By default, the compiler tries to maximize compatibility with existing .NET code at the expense of compilation speed and size of the output executable. This allows people to use their existing code that worked well in a fully dynamic mode without hitting issues caused by full AOT compilation. To read more about reflection, see the [Reflection in AOT mode](reflection-in-aot-mode.md) document. The compatibility behaviors can be turned off by adding/editing following properties in your project file:
+
+* `<RootAllApplicationAssemblies>false</RootAllApplicationAssemblies>`: this disables the compiler behavior where all code in the application assemblies is considered dynamically reachable. Leaving this option at the default value (`true`) has a significant effect on the size of the resulting executable because it prevents removal of unused code that would otherwise happen. The default value ensures compatibility with reflection-heavy applications.
+* `<IlcGenerateCompleteTypeMetadata>false</IlcGenerateCompleteTypeMetadata>`: this disables generation of complete type metadata. This is a compilation mode that prevents a situation where some members of a type are visible to reflection at runtime, but others aren't, because they weren't compiled.
+* `<IlcGenerateStackTraceData>false</IlcGenerateStackTraceData>`: this disables generation of stack trace metadata that provides textual names in stack traces. This is for example the text string one gets by calling `Exception.ToString()` on a caught exception. With this option disabled, stack traces will still be generated, but will be based on reflection metadata alone (they might be less complete).

Documentation/using-corert/reflection-in-aot-mode.md

@@ -0,0 +1,66 @@
+# Reflection in AOT mode #
+
+When .NET code is compiled ahead of time, a typical problem the ahead of time compiler faces is deciding what code to compile and what data structures to generate.
+
+For static languages such as C or C++, the problem of deciding what to include in the final executable is quite simple: one starts with including `main()` and establishing what other methods and data structures `main()` references. One then includes those references, the references of the references and so on, until there's no reference left to include. This concept is easy to understand and works great for languages like C or C++. Nice side effect of this approach is that the generated program is small. If the code doesn't call into e.g. the `printf` function, the `printf` function is not generated in the final executable.
+
+Problems with such approach start to show up on platforms that allow unconstrained reflection. Reflection is a mechanism .NET provides that allows developers to inspect the structure of the program at runtime and access/invoke types and their members. With unconstrained reflection, the definition of "program" includes "everything that one would have access to at the time of compiling the program".
+
+As a motivating example, consider this program:
+
+```csharp
+class Program
+{
+    public static void Main()
+    {
+        Console.Write("Name of type: ");
+        string typeName = Console.ReadLine();
+
+        // Allow to exit the program peacefully
+        if (String.IsNullOrEmpty(typeName))
+            return;
+
+        Console.Write("Name of method: ");
+        string methodName = Console.ReadLine();
+
+        Type.GetType(typeName).GetMethod(methodName).Invoke(null, null);
+    }
+
+    public static void SayHello()
+    {
+        Console.WriteLine("Hello!");
+    }
+}
+```
+
+The above program lets the user invoke any parameterless public static method on any type. For the naive compilation algorithm above, this program would work great for input strings `Program` and `Main` because the algorithm included method `Main` in the final executable. The program wouldn't work so great for inputs `Program` and `SayHello`†, because method `SayHello` wasn't called from anywhere. For the naive algorithm, the only way to fix the program for inputs `Program` and `SayHello` is to add a call to `SayHello` in `Main`.
+
+> † The behavior for `Type.GetMethod` on type `Program` and method `SayHello` would be to return `null` if `SayHello` wasn't compiled. The reason for this is that `Type.GetMethod` is documented to return `null` if there's no method with a given name, and for the purposes of the program, `SayHello` doesn't exist. The compiler could remember there used to be such method and write the information in the executable, but that would raise additional questions about whether the uncompiled method should be included in the list of methods returned from a `Type.GetMethods` call.
+
+While the example program above is not practical in reality, similar patterns exist in e.g. reflection based serialization and deserialization libraries that access members based on their names that could be literally downloaded from the internet.
+
+The dynamic nature of reflection doesn't pose a problem just for fully AOT .NET Runtimes. It's also a problem when tools such as [IL linker](https://github.com/dotnet/core/blob/master/samples/linker-instructions.md) are used to remove unnecessary code. The desire to remove unused code is stronger in fully AOT mode, since native code comes with a greater multiplicative factor (IL instructions are more compact than native instructions).
+
+## Solving reflection in full AOT mode ##
+
+The solution to reflection is about establishing what parts of the program can be reached dynamically and making sure their metadata and code is available at runtime.
+
+### Assume everything is accessed dynamically ###
+
+The compiler can simply assume that everything can be accessed dynamically. This means that everything will be compiled and available at runtime. This is the safest possible option, but results in big executables and long compilation times. "Everything" includes all of .NET Core framework code, including things like support for FTP or WCF. An app is unlikely to be relying on all of that.
+
+### Assume non-framework code is accessed dynamically ###
+
+The compiler can make an assumption that everything that is not part of .NET framework can be accessed dynamically. Unused parts of the framework will not be available for reflection, but real world programs rarely reflect on them. This option still produces executables that are pretty big (especially with many NuGet packages referenced), but their size is more practical.
+
+### Assume statically reachable code is accessed dynamically ###
+
+This is the algorithm we discussed above - only things that are reachable through the static callgraph will be available for reflection.
+
+### Assume code computed by static analysis is accessed dynamically ###
+
+The compiler can build insights into how reflection is used by analyzing the use of reflection APIs within the compiled program and using data flow analysis to see what elements are reflected on. This is effective for a lot of patterns (such as `typeof(Foo).GetMethod("Bar")`), but can also miss a lot of reflection use in practice. Currently, the .NET Native compiler used to build Universal Windows Apps can do this analysis. The CoreRT compiler is structured so that it can consume these inputs, but there's no component to provide them right now.
+
+### Assume nothing is accessed dynamically ###
+
+In CoreRT, reflection metadata (names of types, list of their methods, fields, signatures, etc.) is _optional_. The CoreRT runtime has its own minimal version of the metadata that represents the minimum required to execute managed code (think: base type and list of interfaces, offsets to GC pointers within an instance of the type, pointer to the finalizer, etc.). The metadata used by the reflection subsystem within the base class libraries is only used by the reflection stack and is not necessary to execute non-reflection code. For a .NET app that doesn't use reflection, the compiler can skip generating the reflection metadata completely. This option is currently [not publicly exposed](https://github.com/dotnet/corert/issues/6897), but it exists. People who would like to totally minimize the size of their applications or obfuscate their code could be interested in this option.

samples/MonoGame/rd.xml

@@ -1,10 +1,6 @@
 <Directives>
     <Application>
         <Assembly Name="MonoGame.Framework">
-            <Type Name="Microsoft.Xna.Framework.Content.EffectReader" Dynamic="Required All" />
-            <Type Name="Microsoft.Xna.Framework.Content.SpriteFontReader" Dynamic="Required All" />
-            <Type Name="Microsoft.Xna.Framework.Content.Texture2DReader" Dynamic="Required All" />
-            <Type Name="Microsoft.Xna.Framework.Content.RectangleReader" Dynamic="Required All" />
             <Type Name="Microsoft.Xna.Framework.Content.ListReader`1[[System.Char,mscorlib]]" Dynamic="Required All" />
         </Assembly>
         <Assembly Name="mscorlib" />

samples/WebApi/rd.xml

@@ -1,79 +1,24 @@
 <Directives>
     <Application>
-        <Assembly Name="SampleWebApi" Dynamic="Required All" />
-        <Assembly Name="Microsoft.AspNetCore.Server.Kestrel.Core">
-            <Type Name="Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Server.Kestrel.Core.Internal.KestrelServerOptionsSetup" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.Server.Kestrel" Dynamic="Required All"/>
-        <Assembly Name="Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets">
-            <Type Name="Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.Extensions.DependencyInjection" Dynamic="Required All">
-            <Type Name="Microsoft.Extensions.DependencyInjection.DefaultServiceProviderFactory" Dynamic="Required All" />
-            <Type Name="Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver" Dynamic="Required All" />
-            <Type Name="Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator" Dynamic="Required All" />
-        </Assembly>
         <Assembly Name="Microsoft.Extensions.Options">
             <Type Name="Microsoft.Extensions.Options.OptionsManager`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions,Microsoft.AspNetCore.Server.Kestrel.Core]]" Dynamic="Required All" />
             <Type Name="Microsoft.Extensions.Options.OptionsFactory`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions,Microsoft.AspNetCore.Server.Kestrel.Core]]" Dynamic="Required All" />
             <Type Name="Microsoft.Extensions.Options.OptionsMonitor`1[[Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions,Microsoft.Extensions.Logging.Console]]" Dynamic="Required All" />
         </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.Mvc.Core" Dynamic="Required All" />
-        <Assembly Name="Microsoft.AspNetCore.Routing">
-            <Type Name="Microsoft.AspNetCore.Routing.Internal.RoutingMarkerService" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Builder.RouterMiddleware" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Routing.Tree.TreeRouteBuilder" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Routing.DefaultInlineConstraintResolver" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Routing.RouteOptions" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.Mvc.Formatters.Json">
-            <Type Name="Microsoft.AspNetCore.Mvc.Formatters.Json.Internal.MvcJsonMvcOptionsSetup" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Mvc.MvcJsonOptions" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Mvc.MvcJsonOptionsConfigureCompatibilityOptions" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.Authorization">
-            <Type Name="Microsoft.AspNetCore.Authorization.DefaultAuthorizationPolicyProvider" Dynamic="Required All" />
-            <Type Name="Microsoft.AspNetCore.Authorization.AuthorizationOptions" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.Http">
-            <Type Name="Microsoft.AspNetCore.Http.HttpContextFactory" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.AspNetCore.HostFiltering">
-            <Type Name="Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware" Dynamic="Required All" />
-        </Assembly>
         <Assembly Name="Microsoft.AspNetCore.Hosting" Dynamic="Required All">
-            <Type Name="Microsoft.AspNetCore.Hosting.Internal.ApplicationLifetime" Dynamic="Required All" />
             <Type Name="Microsoft.AspNetCore.Hosting.Internal.StartupLoader+ConfigureServicesDelegateBuilder`1[[System.Object,System.Private.CoreLib]]" Dynamic="Required All" />
         </Assembly>
         <Assembly Name="Microsoft.Extensions.Logging.Abstractions">
             <Type Name="Microsoft.Extensions.Logging.Logger`1[[Microsoft.AspNetCore.Hosting.Internal.WebHost,Microsoft.AspNetCore.Hosting]]" Dynamic="Required All" />
         </Assembly>
-        <Assembly Name="Microsoft.Extensions.Logging">
-            <Type Name="Microsoft.Extensions.Logging.LoggerFactory" Dynamic="Required All" />
-        </Assembly>
         <Assembly Name="Microsoft.Extensions.Logging.Configuration">
-            <Type Name="Microsoft.Extensions.Logging.Configuration.LoggerProviderConfigurationFactory" Dynamic="Required All" />
             <Type Name="Microsoft.Extensions.Logging.Configuration.LoggerProviderConfiguration`1[[Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider,Microsoft.Extensions.Logging.Console]]" Dynamic="Required All" />
             <Type Name="Microsoft.Extensions.Logging.Configuration.LoggerProviderOptionsChangeTokenSource`2[[Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions,Microsoft.Extensions.Logging.Console],[Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider,Microsoft.Extensions.Logging.Console]]"  Dynamic="Required All" />
         </Assembly>
-        <Assembly Name="Microsoft.Extensions.Logging.Console">
-            <Type Name="Microsoft.Extensions.Logging.Console.ConsoleLoggerOptions" Dynamic="Required All" />
-            <Type Name="Microsoft.Extensions.Logging.Console.ConsoleLoggerProvider" Dynamic="Required All" />
-            <Type Name="Microsoft.Extensions.Logging.Console.ConsoleLoggerOptionsSetup" Dynamic="Required All" />
-        </Assembly>
-        <Assembly Name="Microsoft.Extensions.Logging.Debug">
-            <Type Name="Microsoft.Extensions.Logging.Debug.DebugLogger" Dynamic="Required All" />
-            <Type Name="Microsoft.Extensions.Logging.Debug.DebugLoggerProvider" Dynamic="Required All" />
-        </Assembly>
         <Assembly Name="System.Linq.Expressions">
             <Type Name="System.Linq.Expressions.ExpressionCreator`1[[Newtonsoft.Json.Serialization.ObjectConstructor`1[[System.Object,System.Private.CoreLib]],Newtonsoft.Json]]" Dynamic="Required All" />
             <Type Name="System.Linq.Expressions.ExpressionCreator`1[[System.Func`2[[System.Object,System.Private.CoreLib],[System.Object,System.Private.CoreLib]],System.Private.CoreLib]]" Dynamic="Required All" />
         </Assembly>
-        <Assembly Name="Microsoft.Extensions.ObjectPool">
-            <Type Name="Microsoft.Extensions.ObjectPool.DefaultObjectPoolProvider" Dynamic="Required All" />
-        </Assembly>
         <Assembly Name="Newtonsoft.Json">
             <Type Name="Newtonsoft.Json.Serialization.ObjectConstructor`1[[System.Object,System.Private.CoreLib]]" Dynamic="Required All" />
         </Assembly>
@@ -82,8 +27,5 @@
             <Type Name="System.ComponentModel.StringConverter" Dynamic="Required All" />
             <Type Name="System.ComponentModel.Int32Converter" Dynamic="Required All" />
         </Assembly>
-        <Assembly Name="Microsoft.Extensions.Configuration.Json">
-            <Type Name="Microsoft.Extensions.Configuration.Json.JsonConfigurationSource" Dynamic="Required All" />
-        </Assembly>
     </Application>
 </Directives>

src/BuildIntegration/Microsoft.NETCore.Native.targets

@@ -27,6 +27,15 @@ See the LICENSE file in the project root for more information.
     <TargetOS Condition="'$(TargetOS)' == ''">$(OS)</TargetOS>
   </PropertyGroup>
 
+  <!-- Set up the defaults for the compatibility mode -->
+  <PropertyGroup Condition="'$(NativeCodeGen)' != 'readytorun'">
+    <_BuildingInCompatibleMode Condition="$(RootAllApplicationAssemblies) == '' and $(IlcGenerateCompleteTypeMetadata) == '' and $(IlcGenerateStackTraceData) == ''">true</_BuildingInCompatibleMode>
+
+    <RootAllApplicationAssemblies Condition="$(RootAllApplicationAssemblies) == ''">true</RootAllApplicationAssemblies>
+    <IlcGenerateCompleteTypeMetadata Condition="$(IlcGenerateCompleteTypeMetadata) == ''">true</IlcGenerateCompleteTypeMetadata>
+    <IlcGenerateStackTraceData Condition="$(IlcGenerateStackTraceData) == ''">true</IlcGenerateStackTraceData>
+  </PropertyGroup>
+
   <PropertyGroup>
     <NativeObjectExt Condition="'$(TargetOS)' == 'Windows_NT'">.obj</NativeObjectExt>
     <NativeObjectExt Condition="'$(TargetOS)' != 'Windows_NT'">.o</NativeObjectExt>
@@ -223,7 +232,8 @@ See the LICENSE file in the project root for more information.
 
     <MakeDir Directories="$([System.IO.Path]::GetDirectoryName($(NativeObject)))" />
 
-    <Message Text="Generating native code" Importance="high" />
+    <Message Text="Generating native code" Condition="$(_BuildingInCompatibleMode) != 'true'" Importance="high" />
+    <Message Text="Generating compatible native code. To optimize for size or speed, visit https://aka.ms/OptimizeCoreRT" Condition="$(_BuildingInCompatibleMode) == 'true'" Importance="high" />
 
     <Exec Command="&quot;$(IlcPath)\tools\ilc&quot; @&quot;$(NativeIntermediateOutputPath)%(IlcCmd.ResponseFileName)&quot;" />
   </Target>

src/ILCompiler/src/ApplicationAssemblyRootProvider.cs

@@ -23,14 +23,14 @@ namespace ILCompiler
 
         public void AddCompilationRoots(IRootingServiceProvider rootProvider)
         {
-            foreach (var inputFile in _context.ReferenceFilePaths.Keys)
+            foreach (var inputFile in _context.ReferenceFilePaths)
             {
-                ProcessAssembly(inputFile, rootProvider);
+                ProcessAssembly(inputFile.Value, rootProvider);
             }
 
-            foreach (var inputFile in _context.InputFilePaths.Keys)
+            foreach (var inputFile in _context.InputFilePaths)
             {
-                ProcessAssembly(inputFile, rootProvider);
+                ProcessAssembly(inputFile.Value, rootProvider);
             }
         }
 
@@ -39,7 +39,9 @@ namespace ILCompiler
             EcmaModule assembly;
             try
             {
-                assembly = (EcmaModule)_context.ResolveAssembly(new AssemblyName(inputFile), false);
+                // We use GetModuleFromPath because it's more resilient to slightly wrong inputs
+                // (e.g. name of the file not matching the assembly name)
+                assembly = _context.GetModuleFromPath(inputFile);
             }
             catch (TypeSystemException.BadImageFormatException)
             {

src/ILCompiler/src/Program.cs

@@ -414,6 +414,9 @@ namespace ILCompiler
                     if (!systemModuleIsInputModule)
                         compilationRoots.Add(new ExportedMethodsRootProvider((EcmaModule)typeSystemContext.SystemModule));
                     compilationGroup = new SingleFileCompilationModuleGroup();
+
+                    if (_rootAllApplicationAssemblies)
+                        compilationRoots.Add(new ApplicationAssemblyRootProvider(typeSystemContext));
                 }
 
                 if (_nativeLib)
@@ -430,11 +433,6 @@ namespace ILCompiler
                 {
                     compilationRoots.Add(new RdXmlRootProvider(typeSystemContext, rdXmlFilePath));
                 }
-
-                if (_rootAllApplicationAssemblies)
-                {
-                    compilationRoots.Add(new ApplicationAssemblyRootProvider(typeSystemContext));
-                }
             }
 
             //

tests/CoreFX/runtest/CoreFXTestHarness/Test.csproj

@@ -15,6 +15,9 @@
         <TargetFramework>netcoreapp2.1</TargetFramework>
         <!-- Don't warn if some dependencies were rolled forward -->
         <NoWarn>$(NoWarn);NU1603</NoWarn>
+
+        <!-- Generating complete type metadata makes us hit various light up paths that then don't work -->
+        <IlcGenerateCompleteTypeMetadata>false</IlcGenerateCompleteTypeMetadata>
     </PropertyGroup>
 
     <ItemGroup>

tests/src/Simple/SimpleTest.targets

@@ -7,6 +7,7 @@
     <IntermediateOutputPath>$(MSBuildProjectDirectory)\obj\$(Configuration)\$(Platform)\</IntermediateOutputPath>
     <DebugType Condition="'$(DebugType)' == ''">portable</DebugType>
     <DefineConstants Condition="$(NativeCodeGen) == 'cpp'">$(DefineConstants);CODEGEN_CPP</DefineConstants>
+    <RootAllApplicationAssemblies>false</RootAllApplicationAssemblies>
   </PropertyGroup>
 
   <PropertyGroup>