We were not allowing these because we couldn't properly fill out the vtable (can't really make unboxing thunks to put in the vtable). We actually need the vtable, but only because the generic dictionary lives there and reflection might touch it.
[tfs-changeset: 1705047]
#2695 got closed without any effort to validate things work. While this test has changed since the issue got filed, maybe we can still luck out and hit the issue in the CI (in which case I'll reopen the issue). If not, at least we'll get rid of the comment that references the closed issue.
* Separate Issues on a per OS basis
* Add Linux exclusions
* Fix -p flag
* Add OSX exclusions for all-green
* Final Test Definitions
* Enable tests in runtest
* Unify OSX and Linux issues
* Update docs to reflect changes
* Remove Threading.Tasks.Tests and intermittent test failures
* Prune test list to include only System.Collections.Tests
* Add RSP file support
Incremental CI Improvements
Add JSON validation
* Define broad test exclusions
Remove duplicate test names
* Add OSX and Linux URLs
* Revert exit commands
* Change test log output directory and disable test execution printing
* Remove exclusions to test CI failure
* Disable simple and multimodule tests during CoreFX
* Revert "Remove exclusions to test CI failure"
This reverts commit 162a354365.
* Change exit code to test test result reporting
* Add exclusions
* Add removed exclusions
* Minor shell script fixes
- Enable struct Marshal APIs for reference types
- Fix mismatched argument order for native to managed marshaling
- Fix sizes reported by Marshal.SizeOf APIs
- Delete some unnecessary wrappers and fixed a few typos
Fixes#5674 and #5217
Code to call InitializeModules on startup and associated bug fixes and workarounds. Handles mismatched struct calling conventions between LLVM and Clang and works around the dup opcode incorrectly reloading values. Enables StartupCodeMain and fix associated bugs in pinvokes. Implements main return codes
Gets rid of `LINK : warning LNK4098: defaultlib 'LIBCMTD' conflicts with use of other libs; use /NODEFAULTLIB:library [d:\git\rt\tests\src\Simple\StaticLibrary\StaticLibrary.csproj]`.
Since pointer types cannot be boxed, reflection invented a `System.Reflection.Pointer` type that can be used in places where an object reference is required. This implements support for the `Pointer` type in:
* Delegate.DynamicInvoke for Project N and CoreRT
* MethodBase.Invoke for Project N and CoreRT
The MethodBase change is most annoying because it requires us to do extra work due to not being able to instantiate generics over pointer types (the invoke thunks are otherwise generic). We need to unwrap the pointed to types and compensate for that in the thunk. This also means we now create instantiations over void (for void pointers), but meh.
Also fixes two bugs found on the way (so that I get the whole trifecta of IL2IL, NUTC, and binder work):
* Pointer type hashcodes were generated wrong by the binder for pointer types deeper than level 1
* The null to default(pointer) conversion didn't work (this also affected pointer typed field setters)
Fixes bug 457960.
[tfs-changeset: 1692475]
Make the Server GC an optional component by building two flavors of the runtime and selecting a specific one in the build via the `ServerGarbageCollection` MSBuild property. Don't build the Server GC flavor for Web Assembly.
Fixes#5182, Fixes#5306.
This got broken when sorting got introduced to stabilize the executable image. Instead of relying on implicit ordering of nodes, use order specified in the map.
Since we currently allow unnormalized things in the dependency graph (see #5264), we need to ensure scanner produces a dependency graph that can be queried for unnormalized vtable slices.
This fixes a regression in a ASP.NET scenario. Also fixes#5508.
* add preliminary Library test project (#4985)
* update library test and make it pass on Windows (#4985)
* rename Library test project to SharedLibrary (#4985)
* update sharedlib test, make it pass on darwin (#4985)
* update msbuild tag names (#4985)
* move native runner rsp file to native intermediate output path (#4985)
* add test for static library, make it pass on unix (#4985)
* update static library test, make it pass on Windows
* skip shared library test for linux (#4985)
* fix StaticLibrary linux build (#4985)
* update unix linker args when building StaticLibrary native binary (#4985)
* add copyright headers to all newly added source files (#4985)
* fix multimodule build errors for native library builds (#4985)
* add comment explaining owning module of native library startup method (#4985)
* set owning module of native library startup method to gen'd module (#4985)
* remove unneeded System.Linq import (#4985)
* Implement static constructor triggering. Also includes several bug fixes found in the process:
* Don't make the ClassConstructorRunner depend on module intialization
* Moves non-GC statics and thread statics from globals to the type's data regions. GC statics can't be moved until we can call InitializeModules on startup.
* Devirtualizing interface calls to structs in order to be able to compile the class constructor runner.
* Add a prolog block before Block0 to allow branching to Block0, which happens in retail builds
* Correct use of unreachable in traps and at the end of finally blocks to fix more retail build problems
* Stop reusing spill slots when a spilled value is spilled again. This avoids cases where the spills feeding into a block don't use the same slot
* Enable thunks for NativeCallable methods and call RhpReversePInvoke for them
* Fix alignment for cases where a small type is followed by a larger one
* Workaround for Emscripten atomics bug
* Automatically execute WebAssembly tests on Windows by using emrun to launch a headless instance of Firefox (which doesn't interfere with any other Firefox usage). This should cut down on manual testing and help us move toward CI integration.
* Implement intrinsic call to InitializeArray.
RuntimeHelpers.InitializeArray is implemented as a call to LLVM
intrinsic llvm.memcpy.p0i8.p0i8.i32, which copies from generated global
constant to provided target array.
Currently the base size of the array object is hardcoded to 8, since no
EEType is provided in the intrinsic call.
Fix#5345.
Add testing code for InitializeArray in HelloWasm.
Roslyn generates RuntimeHelpers.InitializeArray for arrays of integral
types with initializer of length greater or equal than 3. Here we use
this feature to generate a call to RuntimeHelpers.InitializeArray for
test.
Catagorize array argument types in InitializeArray
Differently handle vectors, multidimensional arrays and object typed
arrays in InitializeArray intrinsic.
Picks up the latest version of Pri0 tests (as of yesterday).
There was a massive cleanup of Pri0 tests on the CoreCLR side so we now run only about 2000 tests. Hopefully, they're more representative now. Most of what we picked as "Top200" was no longer part of the Pri0 suite, so I made a new one.
We might want to consider switching to Pri1 for the rolling builds.
* Implement break opcode as llvm.debugtrap.
In Emscripten, `llvm.debugtrap` is implemented as a round-trip call to
`debugger;` statement JavaScript which will invoke debugger in browsers;
and in LLVM it's implemented as `unreachable` instruction of WASM. This
should be a better match than `llvm.trap` on `break` opcode semantics.
Fix#4511
Set IsDebuggerPresent to TRUE in case of WASM. Although
`Debugger._isDebuggerAttached` is not set, this modify itself will allow
`Debugger.Break` to work.
Add test in HelloWasm for `Debugger.Break`.
* Enable wasm building on OSX.
Upgrade libLLVM to 4.0.0 and LLVMSharp to 5.0.0
LLVM upgrade: Fix 'Use still stuck around after Def is destroyed' of the deleted basic blocks
Add wasm support in runtest.sh
Remove HelloWasm.csproj reference to .ilproj on non-windows OS because of dependency on ilasm
Fix LinkNative target to execute correct commands for Unix wasm builds.
Added support for building on Ubuntu 16.04.3
Update documentation on how to build WebAssembly.
* Implement castclass & isinst for wasm (#4510)
Implement castclass and isinst opcodes in ILToWebAssemblyImporter by
doing related function calls to methods in System.Runtime.TypeCast.
Fix#4510
Add castclass & isinst test in HelloWasm.
This includes three types of casting:
* castclass/isinst to classes.
* castclass/isinst to interfaces.
* castclass/isinst to array types.
For now, the second and third part of test is failing due to runtime
implementation problems which should be further digged into.
* Implements localloc and fixes other issues required to make Int32.ToString work (which relies on stack allocation, Spans and various value type special cases). Includes:
* Allocating localloc buffers on the C++ stack since they're guaranteed not to have GC references and LLVM might be able to optimize them a bit better
* Handling newobj for value types by allocating them in a spill slot
* Implementing the ByReference.get_Value intrinsic
* Fixing various shadow stack management bugs around calls. In particular, return values are now spilled to avoid the next call overwriting them.
* A few new tests
implemented array instructions, implemented internal calling convention, switched to managed runtime calls, Refactored call to not require stack operations, finished work to map pinvokes into the same generated methods/signatures
* Implement cpobj opcode for wasm
Implements the ILToWebAssembly.ImportCpObj method using an LLVM load and
store.
Adds a test for for wasm cpobj to the HelloWasm test using an IL
project.
* Ensure reflection has access to instance methods on byref-like types
Adding the test uncovered a subtle issue. When rooting a virtual method for reflection, we should root the virtual method use of the slot defining method (not e.g. the `ToString` method on the byref-like type - we don't have virtual method use information for that).
* Disable part that doesn't work on Unix
* Changes to hook up the portable runtime and bootstrapper. Includes implementing allocation using RhpNewFast as well as some floating point codegen fixes required to make the new code compile.
Removes usage of buggy dladdr API in WebAssembly, fixes order of conditional branch expressions, which fixes printing when C# optimizations are enabled, adds debug and release flags to emcc, updates WebAssembly documentation to reflect the new build flavor and linking steps. Adds thunks for RuntimeExport methods to fix linker errors for missing runtime exports.
Latest XCode errors on ordered comparison of pointer with integer
Also, fixed some unnecessary always-true comparisons that the C++ compiler emitted warnings for.
* Determinism test harness
Add a /determinism mode to runtest.cmd that will invoke ILC twice with different random determinism seeds. This causes graph expansion to be randomized based on seed (though deterministic for a given seed).
* Add Utf8String.CompareTo
Implementation taken from S.P.CoreLib. All the loop unrolling optimizations were excluded; we re-implement them if this function becomes a perf bottleneck.
* Define the mechanisms for determinism
To ensure deterministic output, all nodes that are emitted to the binary are sorted after compilation when retrieving the final marked nodes list. The overall approach is to sort nodes of the same type together (ie, all EETypeNodes come before NonGcStaticsNodes). Within nodes of the same type, a comparison function uses a CompilerComparer to compare the key identifying data of a node. For example, an EETypeNode is defined by a TypeDesc. The CompilerComparer provides a stable comparison function for comparing various type system primitives. Some nodes need to be emitted in an early and specific order for compiler correctness; this is provided by partitioning all sorted nodes into two phases: the first phase containined specifically ordered nodes, and a second phase for all other nodes whose ordering in the output binary doesn't matter.
Add SortableDependencyNode abstract class which provides the sortability layer on top of a DependencyNodeCore. Both ObjectNode and EmbeddedObjectNode derive from SortableDependencyNode to provide sortability across all nodes that are emitted to the output binary.
Introduce `ISortableSymbolNode` to provide sortability for nodes that are currently referred to in the compiler via the ISymbolNode interface. Such nodes are actually `ObjectNode` or `EmbeddedObjectNode` instances that we've lost type information for. Instances that implement `ISortableSymbolNode` redirect to the matching SortableDependencyNode methods.
Add `EmbeddedDataContainerNode` as a base class of `ArrayOfEmbeddedDataNode` to allow comparison of different instantiations.
Refactor the interface dispatch map index calculation so it's done when the dispatch map is emitted at the end of compilation. Previously it was done when the indirection cell in the dispatch map array got marked. This mechanism is incompatible with generating stable dispatch map indices. The arrays of embedded nodes now stabilize IDs as they emit their final data. This introduces an output ordering dependency - OptionalEETypeNodes must be emitted after the dispatch map since they encode dispatch map indices. Manually enforce this with C++ code generation, since it doesn't emit the real dispatch map structure and builds its own.
Modify InterfaceDispatchMapNode to use the type name in name mangling instead of an index into the dispatch map table.
Modify ObjectDumper to also amit a sha256 hash of each node's data. This dump is used to diff the map files and prove determinism.
* Fill out ordering functions for all nodes
Most of these are not very interesting. Here's the overall approach:
- Every different type of node needs a unique ClassCode. These were generated using Math.Random.
- The various metadata / native layout nodes plus arrays of EmbeddedObjectNodes get placed in the Ordered phase with specific ordering of each.
- All other nodes go in the Unordered phase
- To order nodes of the same type, the data that represents the key for the node in NodeFactory is compared. Ie, for an EEType, that would be its defining TypeDesc, whereas a FrozenStringNode is defined by the string it represents).
- The marked nodes list also contains raw DependencyNodeCore<T> nodes, which aren't emitted. Those are all shuffled after the emitted nodes and not sorted amongst themselves. That hasn't proven to be a problem with determinism and saves a bunch of hopefully unnecessary comparisons.
Added support for ldsfld/stsfld/ldsflda/ldflda including general value type support, throw on invalid branches to the first basicblock in a routine and ensure type sameness for icmp
* Fix instance method parameter management and a couple of codegen issues that showed up when more methods started compiling. Includes implementing ldfld for instance fields (progress toward #4530) and the leave opcode. This is enough to make the String.Length getter work.
* Implement neg (float/integer) and not IL instructions. Fixes#4524 and #4525
* Neg and not basic tests added to HelloWasm.
Name parameter updated in LLVM build methods to reflect the operation being built.
Use PushExpression method instead of direct push to stack.
This PR completes, adds tests for, and enables the CLR Thread Pool on Unix. It also adds Windows implementations for the low level primitives used in the CLR Thread Pool.
The thread pool is enabled by setting the MSBuild property FeaturePortableThreadPool to true, which is the default on Unix.
Supersedes #4039.
Rebased and squashed replacement for original PR #4124.
Changes that allow loading a printing a frozen string. Includes:
* Implement ldnull and ldind
* Fix brtrue and brfalse for pointers
* Fix the polarity of ceq and friends
* Change functions we can't compile into traps so that they don't break LLVM
* Put call arguments in the right order
* Miscellaneous stack canonicalization fixes (we'll have to revisit this in a more comprehensive way soon)
* Minor diagnosability improvements
This change adds support for Server GC.
To enable the Server GC for an application, the RH_UseServerGC environment variable (which is already supported by the runtime) should be set to 1.
On Alpine Linux, the default shell is ash. To build .NET Core, we
explicitly install bash.
If we run the following command:
```sh
echo 0x3F > /proc/self/coredump_filter
```
in bash, it raises the exception:
> echo: write error: Invalid argument
while with ask, it works fine.
Problem is that, when `echo` is run from `bash`, it adds a trailing
linefeed which kernel seems to reject.
Adding `-n` avoids adding the linefeed.
Original exception while building CoreRT:
```sh
chmod +x /corert/Tools/dotnetcli//dotnet
/corert/bin/Linux.x64.Debug/ILCompiler.DependencyAnalysisFramework.Tests/RunTests.sh: line 67: echo: write error: Invalid argument
/corert/Tools/dotnetcli//dotnet xunit.console.netcore.exe ILCompiler.DependencyAnalysisFramework.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nontests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
```
For static array fields, the real element type which can be a descendent of the declaring type must be added to the preinitilizeded data. The element count should also be there.
[tfs-changeset: 1674701]
* Added a dedicated wait thread to the thread pool following similar design principles as CoreCLR
* Fixed project file
* Fix missing return.
* Use the RuntimeThread APIs instead of the Interop-level APIs to create the WaitThread.
* Moved callback execution and unregistering to the thread pool
* Encapsulated signaling the user provided wait handle to ensure it only happens once.
* Handle the blocking case by not queuing the unregistration of the wait on the threadpool when the handle is invalid and instead unregistering directly.
* Rename WaitThread to ClrThreadPool.WaitThread and remove Unix suffix since (in theory), we will be able to move this to work on Windows as well in the future.
* Cache WaitHandle array and timeout and only recreate when the list of RegisteredWaitHandles is changed
* Fixed ClrThreadPool class modifiers in ClrThreadPool.WaitThread.cs
* Fixed reference to WaitThread class
* Fixed reference to WaitThread class
* Re-architected WaitThread to use a partial section of the array of wait handles to wait on and queue all removals to happen on the wait thread itself.
* Fixed compile error in WaitAny
* Make ClrThreadPool.WaitThread non-static and add scaffolding code to ensure handles can be unregistered by the user from the right WaitThread
* Fixed possible indxing errors, prevented double remove requests, and renamed some methods to be more descriptive
* Add comments
* PR feedback
* PR feedback
* Add a lock around selecting a wait thread to register the wait handle on
* ClrThreadPool is no longer static in master
* Only process removals before calling WaitAny, not after
* Try-Finally in ProcessRemovals
* Remove safeToDisposeHandleEvent as per PR feedback
* Added try-finally in UnregisterWait
* Refactor SetEvent into WaitHandle so as to not have direct dependencies on the Unix Wait Subsystem
* waitHandles.Length -> numWaitHandles in WaitAnyCore
* Added try finally in PerformCallback and added a way to ensure Unregister only returns true once
* Refactored timeout to actually account for previous waits when calculating timeout
* Account for handles that timeout while doing other work than waiting
* Changed removals aglorithm to fill nulls in while removing instead of doing so in a later loop
* Create 1st Wait Thread lazily
* Refactor RegisteredWaitHandle. Made wait threads time out. Start wait
threads outside of global lock.
* Fix wait thread accessibility
* Fix registered wait initialization to match CoreCLR
* Removed the lock from RegisteredWait in favor of a CompareExchange flag
* Replaced CompareExchanges with volatile loads and Exchanges
* Fix asserts in WaitSubsystem
* Fix bounds in safe wait handle release
* Added Wait Thread tests.
* Added more wait thread tests and fixed code style.
* Remove resolved TODO.
* Uninterruptible waits on wait handles where supported. Internalize CanUnregister. Return true on first unregister call even if automatically unregistered eariler
* Fix typos
* Add basic recursive event to handle overlapping callbacks
* Fix repeating timer timeout
* Remove dependency on Environment.TickCount from RegisteredWait
* Fixed test
* Fix race conditions on callbacks + unregister based on PR feedback
* Only timeout on change event when there are no user waits
* Fix overload call
* Remove code used for initial debugging
* Double times used in test to allow for about 50 ms window for test timing instead of 25 ms
* Move RegisteredWait into RegisteredWaitHandle
* Unregister on wait thread before callback
* Add test and code to ensure events are not observed on wait thread after an unregistration, even if it is non-blocking.
* Fixed incorrect timeout class
* Expand time frame for failing test to 100ms from 50ms. Add another assert to ensure it's working correctly
* Use event setting for test instead of timeout.
* Cleaned up leftover variables in test
* Only use sync context when wait is uninterruptible
* Make IsBlocking setter private.
* Fixed blocking unregisters.
* PR Renames
* Remove unneeded _unregisterSignaled member.
* Add debug check of _numRequestedCallbacks.
* Add argument verification for top of range.
* Clean up tests per PR feedback.
* Add CheckedWait helper.
* Added dispose-after-unregister test.
* Added block comment stating the reasoning behind not having a blocking wait on removal in unregister.
* Added finalizer to RegisteredWaitHandle.
* Changed overlapping callback test to use events instead of sleeps in the job.
* Refactor timeout times in tests and add comment to ClrThreadPool constant.
* Move updating the user's supplied unregister event to be in the callback lock.
* Added lots of comments and fixed an accessibility modifier
* Clean up tests and argument validation. Lazily create event for blocking unregistration.
* Only grab sync context if wait is interruptible.
* Fixed comment.
* Unregister event will be signaled even if handle has already been unregistered
* Fix timeout checks.
* Added test and code to make it pass for blocking unregistering a handle after it has been removed from the list but is still executing a callback.
* Add an event to wait for the handle to be removed from the list. User code will not get control back until it is safe to dispose the handle.
* Refactor internal event handles
* Changed sleep times for test
* Implement cache for internally used AutoResetEvent
* Exceptions in unregister rollback RegisteredWaitHandle state.
* Change defaults of UserUnregisterWaitHandle and UserUnregisterWaitHandleValue to null and IntPtr.Zero respectively
Once we pick up test assets after dotnet/coreclr#12537 this test will be
passing - moving the test to the category of tests that are waiting for
test update.
- Pick up latest .NET Core 2.0 CLI and buildtools
- Remove all project.json references and convert everything to msbuild projects
- Stick to vanilla .NET CLI project shape as much as possible. Minimize dependencies on buildtools special behaviors
Without this fix, the CoreRT compiler generated the following code for the
test that I am adding:
call Interfaces!_NewHelper_Interfaces_BringUpTest_SomeClass
mov rcx,rax
call
Interfaces!Interfaces_BringUpTest_SomeAbstractBaseClass__get_SomeValue
As you can see, it ends up calling an abstract method on the base class.
With this fix, the interface call is resolved correctly:
call Interfaces!_NewHelper_Interfaces_BringUpTest_SomeClass
mov rcx,rax
call Interfaces!Interfaces_BringUpTest_SomeClass__get_SomeValue
The latest project templates do not have it, and it is only good for generating warnings like the following in the detailed build log:
Project file contains ToolsVersion="4.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="14.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=293424.
[tfs-changeset: 1662024]
The code adds TypeHandleFixup/MethodAddrFixup and abstraction for fixups. It is mostly straight-forward but testing is rather annoying:
The data blobs for fixups are pointer-size and therefore dependent on CPU architecture. I haven't found anything in ILAsm that is similar to ifdef so I have to conditionally include 32-bit or 64-bit copies. I suppose we'll need to deal with the fun that is indian-ness at some point.
Updating IL is quite annoying as I don't want to write test code in IL. I end up splitting the IL in two sections, and I compile the corresponding C# code into exe, ildasm it, and incrementally copy/paste the code that I want into the IL.
I need to use internal types such as FixupRuntimeTypeHandle / NativeCallable, so those need to be manually fixed up from my internal copy to the right reference System.Private.CoreLib
Canonical EETypes weren't reporting their virtual method dependencies,
hoping that by the time they want to emit the VTable, someone else
already reported them. Fixing that by moving virtual method reporting
from `ConstructedEETypeNode` down to `EETypeNode`. At this point it's
really obvious that the current object hierarchy (where `EETypeNode`
represents an unconstructed EEType that shouldn't care about virtuals,
but handles pretty much all of it anyway) doesn't work.
Fixes#3659.
We previously didn't do an intrinsic expanstion for this and had to fall
back to slow and unreliable type loader. This implements a guaranteed
expansion of `CreateInstance<T>` using the infrastructure built for
`EETypePtrOf<T>`.
* Add `RuntimeHelpers.IsReference<T>` intrinsic that constantly expands
to 0 or 1 at compile time (this gets rid of the `default(T) == null`
workaround that requires special handling for `Nullable<T>`).
* Add `DefaultConstructorOf<T>` that expands to the default constructor
of `T` or a special marker method.
* Rewrite `CreateInstanceIntrinsic` using the new intrinsics.
Fixes#368.
* Make virtual invoke map report dependency on virtual method slot
Placing a virtual method in the map means that we'll need a slot for it.
I wasn't sure I liked this as a fix in the past, because it makes the
mapping table potentially bring unused methods into the compilation
(e.g. if the method has multiple overrides), but I convinced myself that
doing anything else would result in terrible user experience and
whatever external tool we'll use to determine reflection roots in the
future will need to apply the same rule.
* Fix release test failure
Now that I make NativeStructType implement INonEmittableType interface
struct marshalling test is failing for CPPCodeGen. The issue here
is that in CPPCodeGen we add type reference for stack values. Since
NativeStructType is INonEmittableType they hitting assertion failures.
Fixes#3616.
The crash in #3616 was from a race condition in a small section of code between setting a `WaitHandle` and unregistering a timed out `Wait` call on that `WaitHandle` on a different thread. This section of code is not locked by the `WaitMonitor`. This assert failure happened when a `WaitHandle` was signaled twice while a `Wait` was in the process of timing out between the locked sections.
However, as per @kouvel, the asserts are invalid anyway since when a thread is waiting on multiple `WaitableObject`s (like in `WaitAll`), the thread's wait info will still be in the waiters list of all of the `WaitableObject`s until they all complete. So, we just remove the asserts that were firing since they will break anyway when more multithreaded tests are added.
In order to make the Marshal APIs to work, this change generates runtime
interop data for the following cases:
. Delegates with UnmanagedFunctionPointerAttribute
. Structs with StructLayoutAttribute
. Generic parameters for Marshal generic API calls which are struct or
delegate
Turns out I got most of this right in #3318. At that time I saw RyuJIT
crashing but didn't have a checked version of it handy. Turns out it was
just an eager assert that is easy to subdue.
Reflection blocking is a size on disk optimization that prevents
generating native metadata for things that are considered
private implementation details of the runtime.
Since CoreRT is compiled ahead of time, a lot of things that
would be needed by a full VM (method names, custom attributes, etc.)
are no longer necessary at runtime. Metadata is strictly only necessary
to support reflection at runtime.
The policy I'm implementing is to consider everything private in our
implementation assemblies to be reflection blocked.
What this entails:
* Adding support for computing reflection blocked state in the compiler
* Tweak to blocking policy to allow us to express blocking state of MethodImpls
* Uncomment blocking table mapping table scanning (this broke the PInvoke test for CppCodegen and we needed a workaround)
* Opt in private assemblies to reflection blocking
The drop we read the test artifacts from doesn't contain Unix native components
necessary for running interop tests. This change consumes another drop from
jenkins depending on whether we are in Linux or OSX.
Adding proper tracking of virtual method reflection use in multifile
mode (#3437) was the last thing blocking this. Let's see if we can get
away with enabling this by default.
This change contains the following:
* AnsiCharArray Marshaller
* AnsiCharMarshaller
* Couple of bug fixes for InlineArray and String builder marshaller
This is a followup to #2390 that fixes a few things:
* vs2017 no longer needs to be passed to build.cmd (it will be
autodetected)
* Building and running tests now works
* Documentation updates to point out running from Developer Command
prompt is needed.
Fixes#3394.
It does the following:
* Enable support for correct ANSI marshalling semantics
(https://github.com/dotnet/corert/issues/2476) by calling into PInvokeMarshal ANSI string marshalling helpers.
* Split Alloc and Transform for StringBuilder marshalling
* Added test case for [Out] StringBuilder marshalling
* Added null check in array marshaller.
I found the problem with filter funclets is that the untracked variables
get reported both by the filter funclet and the enclosing method.
While the compiler makes sure that the lifetimes of other variables get
appropriately split and marked as pinned, this is not the case for
untracked variables - here we simply need to suppress the reporting if
we are indeed in a filter funclet.
My fix simply figures out whether we are indeed in a filter, and if so,
passes an appropriate flag to EnumerateLiveSlots.
I suspect that UnixNativeCodeManager.cpp/.h will need a parallel fix,
but lets first discuss whether my fix for Windows is in fact correct.
This used to work but it's now failing in the rolling build. It's easy
enough to restore enough of the functionality to get the tests passing
again.
The rest of the work (creating the delegate from shared code) is tracked
in #2796 (uncomment the commented out test line to hit a failure in the
JIT).
Update the compiler to be able to leverage the newly added RyuJIT capabilities around delegate constructions.
In the past, RyuJIT would only ask for the construction helper for the simplest cases (target method is not virtual, and no generic runtime lookup is needed). It would call the actual constructor method for everything else.
With updated RyuJIT, we can now use the helper in all cases (if the construction pattern is verifiable).
The change is bigger than I would like because:
* We needed to update `DelegateCreationInfo` to capture the fact that the target of the delegate can now be runtime determined
* There was an unimplemented feature around fat function pointers looked up from dictionaries (we couldn't express that the fat function pointer should point to the unboxing stub).
* `ShadowConcreteMethod` didn't work great with unboxing stubs. I just made a new node that is a "shadow unboxing stub".
* We needed to update both generic and nongeneric ready to run helpers to deal with the new delegate creation patterns.
* JitInterface change.
* Tests
The dynamic invoke tests were failing because the JIT was inlining all the target methods leaving nothing to actually invoke through reflection. Mark all the methods we invoke (including the empty constructors) with `[MethodImpl(MethodImplOptions.NoInlining)]`.
[tfs-changeset: 1653296]
* Add CoreRT implementation for the dynamic invoke template table which maps invoke stub name / sig to the canonical method entrypoint
* CoreRT based compilers use 32bit relative addresses to the stub containing EEType and method entry points (.NET Native uses RVAs). Set the bit on the reloc to indicate a 32bit rel reloc.
* Fix a bug in generic method template map where generic methods on structs were getting IsUnboxingStub set incorrectly
[tfs-changeset: 1653288]
* Add a simple thread pool implementation for Unix
This change adds a very primitive (temporary) implementation of Thread
Pool to allow Tasks to be used on Unix until a proper implementation is
available.
For simplicity of the state management, the thread pool pre-creates all
thread pool workers (controlled by the MaxThreadCount const) on the first
request and then uses a semaphore to release threads as new requests come
in.
In addition, this change enables all test cases in the BasicThreading test.
* Fix some inconsistencies in RuntimeThread.StartThread method due to
differences between Windows and Unix.
Support GC Statics in type loader
* Add support for CoreRT-style GC statics when creating new generic types from templates
* Pass new `BagElementKind.GCStaticEEType` through in the template layout so the type's statics can be heap-allocated
* Add managed accessors to the dynamic GC / non-GC data to EEType.cs
Implementation supports both CoreRT and ProjectX models.
Fixed interface EEType generation to unconditionally produce a dictionary slot for generic interfaces (to support static interface methods, and enable correct Reflection virtual invoke slot computation on ProjectX).
Add support for running a single test via runtest.cmd /test <Test Name>.
Example usage:
```
runtest.cmd /test BasicThreading
runtest.sh -test BasicThreading
```
* Fix a race in thread shutdown on Unix
The problem is that the _stopped event (that is used for signaling that
the thread has stopped) is allocated in the CreateThread() method after an
OS thread is created. As a result, the parent thread can be scheduled out
before it allocates an event and assigns it _stopped. At same time, the
newly created thread gets a chance to run, finishes its work promptly and
exits; while exiting, it tries to signal the _stopped event but it crashes
instead because the event has not been allocated yet. I believe the
problem with lifetime management of the _stopped event is due to the fact
that the original code relied on the value of _stopped in the HasStarted()
method to indicate whether a thread has been created/started or not. On
Windows, the HasStarted() method relied on the value of _osHandle for the
same purpose.
This change fixes this problem by:
1. Changing the HasStarted() method to use the explicit state of the tread
ThreadState.Unstarted (i.e. the _threadState field). This also makes
implementation of this method platform independent. The existing code
already sets the thread state to ThreadState.Unstarted (when an instance
of the RuntimeThread class is created) and clears the bit when the thread
starts executing managed code.
2. Changing RuntimeThread on Unix to allocate the stopped event before
thread creation to actually fix the race.
I have also partially enabled the BasicThreading test for Unix and added a
small regression test for the race condition. Full functionality of
BasicThreading will be enabled once we have ThreadPool supported on all
platforms.
* PR Feedback
* Fix issue with __cxa_thread_atexit in PAL. The change (https://github.com/dotnet/corert/pull/2531)
that introduced __cxa_thread_atexit in PAL did it incorrectly. The __cxa_thread_atexit function
should be called for each TLS object that needs its destructor to be executed at thread shutdown time.
The incorrect usage of __cxa_thread_atexit resulted in hangs in SuspendEE and random corruptions.
* Enable template generation for generic types
* Dictionary slots generated for types / methods cause that type / method to get a type loader template
* Remove the template-specific list of slots and just use the owning type's dictionary layout
* Add not-supported entry for template slots that aren't implemented yet
[tfs-changeset: 1651962]
My previous change errorneously marked these two tests as working
because I had a local hack in place that made TypeRefs to
`TypedReference` always resolve to CoreLib (makes testing possible
without fighting the build system too hard).
Turns out these two tests import `TypedReference` from System.Runtime.
Our System.Runtime doesn't have a forward for that yet.
1) Adding the StaticsInfoHashtable to track statics regions for compiled types.
2) Writing statics field data to the ReflectionFieldMapNode
3) Improving the IndirectionNode to allow it to point to an inner offset in a target symbol
4) CoreRT runtime support for static fields access
5) Collapsing of field entries in the ReflectionFieldMapNode based on canonicalization
* Run the known good ~5000 CoreCLR tests using multi-module compilation
for through-put wins. The whole suite takes under 2 hours locally, vs ~5
hours with single file mode. We should expect to see the rolling test
job reduce from its current 7 hours.
* Remove `MultiFileLeafCompilationModuleGroup` and always use
`MultiFileSharedCompilationModuleGroup` for multi-module builds. This
addresses several failing tests where pointer / array types were getting
ConstructedEETypeNodes generated due to ShouldProduceFullType returning
true incorrectly. Since it's technically possible for libraries to refer
to types in the entry-point assembly, we should always produce full
types in multi-module mode to be safe.
* Disable a GC test that fails under multi-module. The test verifies
that a couple allocated objects are moved to higher generations during
GC. This doesn't happen in multi-module, possible due to a different
pattern of allocations with more code running on the startup path and
unpredictable pinning behavior with conservative GC mode. This test had
better pass when we enable precise GC so I put it under that category.
Implement Thread for Unix:
* Use a callback from PalDetachThread to signal the thread as stopped.
* In BasicThreading test account that SafeHandle postpones disposing until the next garbage collection.
* Move reusable code from RuntimeThread.Windows.cs to RuntimeThread.cs.
* Rename *Core functions to *Internal to be close to CoreCLR implementation.
* Priority is ignored for now.
Fixes for Windows:
* Merge HasFinishedExecution and JoinInternal.
* Handle race condition between JoinInternal and SafeWaitHandle finalizer: SafeWaitHandle.DangerousAddRef may throw an ObjectDisposedException.
* Handle Interop.mincore.CreateThread returning the NULL handle.
The previous commit to add multi-module support for thread statics
(https://github.com/dotnet/corert/commit/be92d719) has a small issue
causing a link failure if the generic type containing the thread-static
field was instantiated in multiple modules.
That commit changed from referring to the thread static base directly to
referring to the type thread static index, which is looked up on a list
reached through the type manager. This mechanism ensures that even
though we put a copy of the static base in multiple object files, we'll
unify because the thread static index nodes are COMDAT folded. The
ThreadStaticsNodes themselves can't be folded since they're embedded
objects in a list, so change the naming to make them unique across
modules.
Pull request #2096 made sure that lazily built vtables of `Foo<X>` and
`Foo<Y>` contain the same entries if their canonical forms are the same,
but didn't make sure the order of the entries is the same. This can lead
to bugs at runtime where the wrong slot is used to dispatch a virtual
method call. To make sure the vtables look the same, we need to also
sort the vtable.
In Project N NUTC compiler, these types are considered "visible by
reflection" and as such, all bets as to what the code is going to do
with it are off. Whole program analysis could potentially optimize which
ones really need to be considered constructed (e.g. we often take an
EEType only to compare it with some other EEType), but that's a possible
optimization for the future.
The runtime has a special set of rules for casting array types. Array
can be cast to various non-variant generic interfaces without triggering
an exception. Note this does not violate type safety because the safety
is enforced at the array store operation level.
The fix is twofold:
* Dependency tracking: methods implementing generic interfaces on arrays
should be always considered used (same as we do for variant interfaces).
This has a pretty negative size on disk impact. The work to see if we
can do something about it without adversely affecting compilation
throughput is tracked in #1198.
* EEType emission - this is a separate issue, but related: generic
interface types that can potentially be implemented by an array need to
have the GenericVariance flag set (even if all the parameters turn out
to be non-variant). This is to make casting take a slower path at
runtime that can handle special array casts.
* Support by-ref parameters
* Add infrastructure for by-ref arg/local
* fix bugs
* make all tests pass
* add some comments
* add test
* address comments
* rename NativeType to NativeTypeKind
* Support custom stack size (maxStackSize parameter).
* OOM hardening for new thread starting. Throw ThreadStartException in case of failure.
* Create a new thread suspended and set its priority before resuming it.
* In HasFinishedExecution method handle the case when an external thread died and its Thread object was resurrected.
* Add test cases for resurrected Thread objects and maxStackSize parameter.
Also fix a couple of issues copied from CLR/CoreCLR code:
* Better handle OS threads with intermediate priorities.
* Obtain the actual thread priority from the OS instead of using the cached value.
The delegate creation code path wasn't updated for shared generics:
* We need to use canonical entrypoints and fat function pointers where appropriate
* This required adding infrastructure to track unboxing stubs from shared code.
More work will be needed around here to add support for creating delegates *from* shared code.
The CoreRT compiler and runtime have been able to handle this for a
while. The only thing missing (that required us to throw from a place)
was the type loader support.
This adds support for building ByRef types at runtime. I'm not
introducing a native layout representation though, so things like `T&`
won't compile still.
This tests that we generate the same dictionaries in multimodule
scenarios. We currently don't.
This is just so that we don't have a clean precheckin run with shared
generics enabled (because without this change, we do).
* Linux support for multi-module
* Alter build integration scripts to support different naming / tool invocations on Linux.
* Add GC / frame info for ELF to the same group as its associated method so it gets Comdat folded properly.
* Add -multimodule switch to runtest.sh to run tests against a shared pre-compiled framework library.
* Update ObjectWriter Nuget package which brings changes need to support Elf Comdat sections.
Initial RuntimeThread implementation.
To do:
* Wait for foreground threads on main thread's exit.
* Support custom stack size (maxStackSize parameter).
* Do OOM/Win32 error hardening for thread starting.
* Handle thread resurrection (check _osHandle for validness).
* Add platform-specific StartCore, JoinCore, and HasFinishedExecution methods for Unix.
Initital support for delegate marshalling.
- Enabled only in full runtime
- Doesn't work in non-windows platforms
- Only supports open static delegates
Allow invalid binaries as input files to ILC
* Add a hopefully short-term hack which allows files specified as
inputs to ILC to be actually native binaries that we ignore and proceed
with compilation. The CoreCLR drops we test against contain a mixture
of managed and native binaries in test folders in some cases.
* Add a quality of life switch to runtest.cmd (/CoreCLRSingleTest)
that allows specifying a single CoreCLR test to run by executable path.
This downloads the test blob and sets the correct environment variables
first.
* Fix regular CoreCLR tests caused by batch exploding if it had to deal
with "if not exist ("
In multi-module mode, if one obj file defines an EEType as necessary and
another defines it as constructed, they get arbitrarily Comdat folded.
If the incomplete EEType is chosen, we see problems with VTable
dispatch, GC AVs (due to lack of a GC desc). In multi-module mode, if a
type is shareable (a concrete generic instantiation, or a parameterized
type such as `String[]`) upgrade requests for the EETypeNode to
ConstructedEETypeNode.
This causes an increase in Framework.lib from 111MB to 121MB due to
rooting more full types.
When running CoreCLR tests in multi-module mode, produce the shared framework library first and then prevent each test from generating it. I found that sometimes MSBuild gets confused and thinks it needs to re-generate the lib (despite the inputs and outputs being properly specified) so this makes sure we don't have multiple tests trying to produce the same set of files.
* Allows runtest.cmd /multimodule /coreclr
* Modify build integration targets to allow for tests with multiple
assemblies. We have two set of assembly inputs to ILC: @(ManagedLibrary)
which despite being an Item contains the main managed assembly to use as
the template name for the response file and output object file base
name. @(IlcCompileInput) is the list of assemblies that are inputs to
Ilc. This change allows us to compile all the app assemblies into a
single object file.
* Specify the inputs / outputs to the MSBuild task that builds the
framework assembly obj files. This removes a page of spam per test when
MSBuild skips over the already-built obj files.
* Currently 12 tests fail due to some EH issue that needs investigating.
This also fixes an NYI in CorInfoImpl around getting a fat function pointer from non-shared code I hit while writing tests.
This needs very unfortunate special casing when processing relocs since references to fat function pointer symbols need to set the second lowest bit to indicate the fact. I tried to make the FatFunctionPointer node define a symbol at the adjusted location but this hits an object writer limitation that it can't define a new symbol in the middle of a reloc.
Returning structs has problems (I will file an issue for those). The tests are commented out.
This regressed when I replaced the "intrinsic but doesn't work for
shared generics" version of `Activator.CreateInstance<T>` with the
"non-intrinsic but works with shared generics" version. There's a
workitem tracking the codegen work to make an intrinsic version in
RyuJIT, so I'm disabling the tests against that.
- See comment at the top of WaitSubsystem.Unix.cs for details
- Refactored WaitHandle and derived classes, and RuntimeThread, to separate platform-specific functionality
- These changes have only been tested to the extent of public APIs currently available. They have not yet been tested with multiple threads, and several features such as sleep, interrupt, signal-and-wait, mutex abandoning, etc., have not yet been tested. More tests will be coming after Thread is made available in CoreRT.
I admit this required special casing in places I don't like, but the
`Address` methods on multidimensional arrays are... well, special.
The problem with the `Address` method is that it needs to do an exact
type check in it's body - the array element type that is expected at the
callsite needs to match the runtime type of the array passed in. We need
this because of array covariance (`string[`]` is allowed to be cast to
`object[,]` and we need to disallow making a `ref object` to an array
element if the runtime type of the array is not exactly `object[,]`).
This changes mirrors how Address methods get compiled on CoreCLR. The
story begins at the callsite:
1. We tell the codegen in `getCallInfo` that `Address` is a method that
requires an instantiation argument (just like e.g. static methods on
shared generic types), but disallow inlining it so that we are forced to
generate a standalone body. Method's owning type becomes the
instantiation argument.
2. We make sure the dependency tracking doesn't track the `Address`
method as a runtime determined method. It's difficult to represent it in
the RuntimeDetermined infrastructure (for the simple reason that
RuntimeDetermined types need to be DefTypes). We can get away with it
because the `Address` method doesn't actually do generic lookups anyway.
3. When we get to compile the method body, we pull the old switcheroo
and replace the method with one that has the instantiation argument in
the signature (so that we can refer to the argument from IL). We then
use the hidden argument to do the type check within the method body.
* Don't add VTable for Finalize method in multi-module
Finalize is not added to the VTable with the CoreRT runtime; instead it
is added after the VTable in conjunction with an EEType flag to save
VTable space. In multi-module library mode with eagerly built VTables,
skip the Finalize method (since it appears in the type system as a
virtual on `System.Object`).
Add a test to validate finalizers are run.
* Throw exception if code callvirts `Object.Finalize`
* Add `IsFinalize` to `MethodDesc`
The fact we could reflection enable two instantiations of a method
relied on pure luck on which invoke stub we choose to invoke the method.
If we chose the one for `object`, we could use it to invoke the method
instantiated over `string`. We ran out of luck in postcheckin.
I'm fixning the test to only have one generic method instantiation to
make it pass again to keep coverage for the other interesting cases.
Making this actually work depends on the "Use type loader to build
invoke stub dictionaries" workitem. Requires type loader.
To support calling canonical interface methods on generic valuetypes,
the compiler needs to generate unboxing+instantiating thunks that bridge
the difference between two calling conventions.
As a refresher:
* Instance methods on shared generic valuetypes expect two arguments
(aside from the arguments declared in the signature): a ByRef to the
first byte of the value of the valuetype (`this`), and a generic context
argument (EEType)
* Interface calls expect `this` to be a reference type (with the generic
context to be inferred from `this` by the callee)
Instantiating and unboxing stubs bridge this by extracting a managed
pointer out of a boxed valuetype, along with the EEType of the boxed
valuetype (to provide the generic context) before dispatching to the
instance method.
We compile them by:
* Pretending the unboxing stub is an instance method on a reference type
with the same layout as a boxed valuetype
* Having the unboxing stub load the `m_pEEType` field (to get generic
context) and a byref to the actual value (to get a `this` expected by
valuetype methods)
* Generating a call to the instance method on the valuetype through a
wrapper that has an explicit generic context parameter in it's
signature.
This is a bunch of things:
* Make the reflection dynamic invoke thunk use the transformed calli
intrinsic to do the call (to handle fat function pointers)
* Generic dictionary nodes are prefixed with the hash code, but I put
the padding in the wrong location. That's what I get for writing code
off memory without crosschecking with Project N binder...
* Make MethodCodeNode depend on it's unboxig stub. This is in the hacky
section with a big comment explaining why we need it for now. (Method
being compiled determines it's reflectability.)
* Emit a pointer to the unboxing stub in the reflection mapping table if
needed.
* Remove a another usage of GetRvaFromIndex from the type loader
* Add a test
* Enable multi-module on Windows
Enable multi-module compilation for tests in CI through a new switch:
`runtest.cmd /multimodule`. This will compile the framework assemblies
into a single Framework.lib that tests are linked against.
* Adjust several dependency nodes in the compiler to tidy up sharing of
nodes / naming for uniqueness.
* Add handling in ObjectWriter for compiler generated types and their
methods to force sharing.
* Ifdef out runtime imports for StdCall which are implemented in
Native.Interop, a library that hasn't been ported to CoreRT yet.
* Remove metadata generation for type args to an instantiation. This was
breaking type unification because we were treating type args as type
defs, even when they are present in another module and we weren't
emitting the actual EEType, causing a null type handle at runtime.
* Alter build integration MSBuild authoring to produce the framework lib
in multi-module builds on demand. By default, the lib is emitted in the
app binary folder. For test runs there are overrides to control where
the framework objs and lib are placed so they can be shared
(`FrameworkLibPath` and `FrameworkObjPath`).
* Alter test harness to take `/multimodule` switch which links all
Ryujit applicable tests against Framework.lib.
* CR Feedback
* Remove TypeSystem.Ecma reference and use IAssemblyDesc instead
* Move check for CompilerGeneratedType to ObjectWriter and check the
module against the generated assembly when determining shareability
* Always generate static bases for library modules
Fixes issue https://github.com/dotnet/corert/issues/2568 where static
fields that aren't referenced anywhere in a module compiled as a library
won't get their static base generated in the image. This causes link
failures when an app is built against the library.
Added a test that is compiled as two separate modules and linked
together.
* CR Feedback II
Fix release Windows multi-module compilation. The __managed_code_a and _managed_code_z bookends were being collapsed into one by the linker. We then told the code manager there were 0 bytes of managed code and EH broke. Fixed to give each book-end symbol a unique value so the linker won't fold them.
My upcoming PR for multi-module testing support needs the CI authoring
done one PR early since netci.groovy is always pulled from HEAD (not the
changes in a pending PR).
Call `runtest.cmd` with the new `/multimodule` switch during Windows CI
runs. Currently the switch causes the test script to immediately return
with success.
Handling of hardware exceptions had a hack to add +1 to the actual instruction IP. Windows x64 unwinder
is disassembling instructions at the IP passed in to detect method epilogs. If the bytes at IP + 1
happened to match the epilog pattern, the unwind is done as if we were in the middle of the epilog that
lead to spectacular crash.
This change is moving this adjustment to be done later for EH related things only, and not interfere
with stackwalking.
Fixes#2535
[tfs-changeset: 1645602]
The change contains the following:
- Select marshaller based on MarshalAs attribute
- Indentify In/Out/Ref and have the infrastructure to marshal based on
them.
This change contains the following:
1. Fixed out SafeHandleMarshaller with CPP codegen as Sedar pointed out.
2. Also enable PInvoke test with CPP Codegen. The generic method
ThrowIfNotEquals doesn't work with CPPCodegen. So temporary created
overrides to address this.
Add scripting for Linux / OSX to download the pre-compiled CoreCLR tests and run them.
Top200.unix.txt contains the set of top 200 tests run on Windows that also pass on Unix systems.
Author the CI netci.groovy changes along with a stubbed out `runtest.sh
-coreclr top200` implementation so that the actual PR containing the
test changes will run. This is needed to ensure we don’t break the
build since netci.groovy changes aren’t picked up in a PR (CI takes
netci.groovy from HEAD).
`dir.props` loads the parent folder `dirs.props`. This is used when
CoreRT is built as a part of another source tree. When building CoreRT
with its own build system, any MSBuild invocation needs to set
`RepoLocalBuild` to `true` to avoid dragging in an unexpected set of
MSBuild configuration. Update the the standard and CoreCLR test
harnesses to pass in `RepoLocalBuild=true` to MSBuild as a command-line
property.
This adds support for generating helpers and dictionary entries to
support thread static base lookups for generic types in shared generic
context.
I slightly tweaked how dictionary slots get emitted because the "every
slot is an `ISymbolNode`" paradigm wasn't very flexible.
The JIT calls this when it hits internal limitations to decide whether to retry with optimizations off; or whether to stay on the safe side and give up in partial trust environments.
Fixes#2462
Call PInvokeMarshal.SaveLastWin32Error after the P/Invoke call to ensure
the last error is saved in thread local variable which can be later
accessed through PInvokeMarshal.GetLastWin32Error.
Additionally, I have rename ParameterAttributes to
ParameterMetadataAttributes to avoid the confusion with
System.Reflection.ParameterAttributes.
This change also makes sure PInvoke works on Unix and also get rids of
build warning on windows.
This is the first iteration of PInvoke change which ensures right
infrastructure is in place for future PInvoke changes. It doesn't
add any new PInvoke features, but makes sure all the existing ones
are working. I have also added a testcase.
MSBuild parallelizes at the project level (though there are third party
extensions that allow parallelism at the task level). Build each of the
framework assemblies by batching on `@(IlcReference)` with a custom
target that invokes the `<MSBuild>` task.
On a hyper-threaded i7-6700 it reduced `BuildFrameworkNativeObjects` run time from 67s to 17s.
This was rather annoying, but we should be able to get a clean run with KnownGood set of test. It still does take around 4 hours to run them on my quad core i7 so we probably don't want them to run in CI.
* Static methods on generic types use the type's generic dictionary
Dependency analysis was incorrectly assuming these have their own
dictionary (they don't).
* Emit R2R generic helpers for ldvirtftn
This change implements basic support for thread static fields. Most things
are already functional, performance could definitely be improved in the
future but it should be sufficient to get things off the ground.
This code passes CoreRT and Top200 CoreCLR tests on Windows. The
BasicThreading test in this change verifies that thread static fields work
for both non-generic and generic types in the single-threaded and
multi-thread environment (using Tasks).
One thing that does not work yet is multi-module compilation because the
driver creates a separate ReadyToRun helper for a type in every module
that accesses thread statics of the type. I am currently working on fixing
this.
The existing code has already implemented a good chuck of required
functionality so this simply builds on top of that.
Each module has a ThreadStatic region. Each entry in the region points to
an EEType that represents a GC map for the thread static fields of a given
type. The index of the entry in the region is the TLS index of the type.
The TypeManager indirection node of the module has been extended to
contain the index of the module in addition to a pointer to the type
manager (which is also used for initialization on first access).
The generated ReadyToRun helpers (that return thread static base for a
type) look like this:
__GetThreadStaticBase_System_Private_CoreLib_System_Threading_ManagedThreadId:
48 8D 0D BD E0 39 00 lea rcx,[__typemanager_indirection
(07FF70D68AD58h)] <= module information (type manager, module index)
BA 03 00 00 00 mov edx,3 <= TLS index of the type
E9 33 FB E4 FF jmp
System_Private_CoreLib_Internal_Runtime_ThreadStatics__GetThreadStaticBaseForType
(07FF70D13C7D8h)
* Fix getting test root on Mac and Linux pipeline builds.
* Add clean scripts for CI to use.
CI is slightly different from a normal user scenario because
it will often be cleaning right after another build before
VBCSCompiler.exe is killed automatically.
* Add checked-in build definitions for pipebuild.
`Type.GetType` needs to be compiled specially since for inputs that are
not assembly-qualified, the method is expected to search the calling
assembly.
The way we do this is by introducing a concept of callsite-specific
intrinsics. The expansion of these intrinsics depends on the callsite
from which the intrinsic is called.
We make calls to `Type.GetType` expand to overload+assembly specific
`ILStubMethod` that thunks to a helper that consumes a "default assembly
argument" in addition to the regular arguments.
RyuJIT depends on the relocation offset delta to be stored in the code stream because of it applies additional fixups to it.
Tweak the list of Top200 tests to include coverage for this issue.
Fixes#2254
This relies on the existing logic that tries DIA first, and then falls
back to reflection metadata to find method names. If fallback fails, you
get RVAs. Conveniently, the DIA path doesn't even try to run.
Some tests have more artifacts than just the test EXE.
* Pass all DLL files to ILC (technically, we only need managed DLLs, but
this shouldn't hurt)
* Copy all DLL files to the output directory (technically, we only need
native DLLs, but this shouldn't hurt)
For now, it's for instance fields only. I decided to make a slight
change to the format to avoid a usless indirection through the external
reflerences table to get the field offset. We might want to port that to
Project N too...
In order to compile larger bodies of code (such as XUnit) without full
GVM support, emit a R2R helper at GVM callsites that fails fast at
runtime. I left the check for GVMs in-place in
`ConstructedEETypeNode.OutputVirtualSlots` to make sure we don't
actually put them in the VTable since they'll eventually be dispatched
via a lookup table.
When the generic ready to run lookup helper needs to know the generic
dictionary vtable slot, it passes the canonical type to the helper that
determines the slot. `GetNumberOfBaseSlots` needs to be able to deal
with this, and the fact that the slots might be lazily determined.
Converting to canon fixes this due to how we ensure canonically
equivalent types have the same vtable in #2096. The other option would
be to back out that change and completely revamp how we do lazy slots
for generic types.
I found this when I ran the reflection invoke test with shared generics
enabled.
We pass in /MTd flag to cl.exe when running tests but unconditionally
pass /MD for all compilations using the BuildIntegration targets. Pass a
more constrained property `UseDebugCrt` and select the appropriate
switch in the build targets.
This is basically three parts:
* Generate a mapping table of method entrypoints to their metadata.
* Generate invocation stubs to lay out `object[]` arguments passed to `MethodBase.Invoke` on the stack.
* A small test.
Only methods that were compiled are eligible to be added to the map.
Enable compiling all framework / SDK assemblies in multi-module library
mode
Add a work-around to compile System.Console as a library in PInvokeILEmitter.cs
System.Console has a p/invoke to GetLargestConsoleWindowSize which
returns a COORD struct containing two 16 byte ints. The struct return
type prevents the JIT from emitting an inline p/invoke in the IL stub we
generate and instead it treats the external p/invoke target as a managed
call and tries to compile it. Force p/invokes returning structs to use lazy p/invokes. These use Calli, which the JIT allows to always be inlined.
For p/invokes, set CorJitFlag.CORJIT_FLG_IL_STUB JIT flag, which is required to allow us to always inline p/invokes.
Address library module compilation failures due to unsuccessful type
loads by checking on EEType creation whether the type can be loaded
sufficiently to write out the EETypeNode. Code for this stolen from one of Michal's previous PRs :)
Eagerly create EETypeNodes in ReadyToRunHelperNode so the failure point
happens during the referencing method compilation, not later on when
emitting the graph
Check method signatures on library compilation method rooting
Methods whose signatures contain types that cannot be loaded (they are
in a separate, non-existent assembly for example) should be skipped
since we cannot even replace their IL body with a throw helper. The JIT
will fail trying to getClassAttribs on the argument types.
This check is done when rooting methods instead of in MethodCodeNode's
constructor because MethodCodeNodes are created during graph sweeping
for things like virtual method combined dependencies.
Add a test that verifies we can compile all framework assemblies to their own separate object files successfully.
Don't check base types for generic type defs
Only check static field layout for types with deferred cctors
Move CheckCanGenerateMethod into MultiFileCompilationModuleGroup and
scope its checks to just what the JIT requires (whether each type
contains GC Pointers).
Scope the type checking for static base ReadyToRunHelpers to just
`ComputeStaticFieldLayout
Provide a choice of different sets of the CoreCLR test tree to run:
- Top200 tests selected from the various areas of the test tree intended to give broad coverage quickly
- KnownGood tests; the set of tests validated as passing on CoreRT
- All CoreCLR tests (Many of which are known to fail)
Make /mode <ryujit|cpp> actually work with local tests (useful for getting test rsp or artifacts of Ryujit tests without the cpp mode blowing them away).
Clean up test documentation to reflect what the test harness does these days.
* Allow tests to be specified inclusively
Expand the test selection process to allow tests to be included instead
of just excluding from all found cmd files.
If at least one test is included via <IncludeList />, skip scanning the
test tree for *.cmd and only take explicitly included tests. Exclusions
can still be applied to these tests and will override inclusions.
The CoreCLR CI machine archives test binaries in a zip file to share
with other parts of the build. CoreCLRTestsURL.txt contains a specific
test drop from a built marked for permanent retention on the Jenkins CI
box.
Modify runtest.cmd to take /CoreCLR switch which will download the
CoreCLR tests, unzip them, and run them using CoreRT.
build-and-run-test.cmd is used as the test runner, and is used to
generate CoreRT binaries for the tests before executing them. CoreRT is
run via the MSBuild targets by placing a placeholder csproj next to the
test binary. To save CI storage space, the native artifacts (which are
about 30-50MB per test) are cleaned after running the test.
Currently 130 simple JIT tests are selected to run using
Top100Tests.issues.targets
Anipik
Michal Strehovský
Michal Strehovsky
Adeel Mujahid
Andon Andonov
yowl
Jeff Greene
Jan Kotas
Morgan Brown
Jeremy Koritzinsky
Toni Solarin-Sodara
Blealtan
Christian Scheuer
Wayde Reitsma
Simon Nattress
Simon Nattress
Juan Antonio Cano
Faizur Rahman
t-jekor
Sergiy Kuryata
dotnet-bot
Morgan Brown
Yi Zhang (CLR)
Fadi Hanna
sergey ignatov
Peter Sollich
Anton Lapounov
Aditya Mandaleeka
Aditya Mandaleeka
Yi Zhang (CLR)
Alan Lawrence
Koundinya Veluri
Chris Rummel
Petr Onderka
David Wrighton