llilc_checkpass compares two test results, no matter summary or verbose,
and checks if any function was successfully jitted by LLILC in base result
but failed jitting in diff result.
If the check passed, the script return 0. In case of 0, it might have
jitted more functions in diff result than in base result. The two results
are not necessary the same in this case. Counting result is reported.
If the check failed due to newly failed function, return 1.
If the check failed due to unexpected reason, return negative numbers.
usage: llilc_checkpass.py [-h] -b BASE_RESULT_PATH -d DIFF_RESULT_PATH
optional arguments:
-h, --help show this help message and exit
required arguments:
-b BASE_RESULT_PATH, --base-result-path BASE_RESULT_PATH
full path to base result
-d DIFF_RESULT_PATH, --diff-result-path DIFF_RESULT_PATH
full path to diff result
Also this change includes some supporting changes in llilc_runtest
and applyfilter:
1. In llilc_runtest result normalization, perform a summary extraction on
verbose result and a renaming on summary result.
2. applyfilter is enhanced with summary extraction so that filtering
and summary extraction can be done at the same time without the
need to scan through result file twice.
3. Summary result renaming added in applyfilter
4. Introduce const module in for symbolic constatn return code
5. Default to single quote style. Python is a little tricky when double
quotes and single quotes are mixed.
6. Create required argument section in help for llilc_runtest and llilc_checkpass
7. Add more error and exception handling
8. Remove llvm verification failure from summary
The JIT options change in https://github.com/dotnet/llilc/pull/412 prepends
COMPlus_ to DumpLLVMIR environment flag. Changes were made in Powershell
script but not in Python script. Update the python script accordingly. Also
update all COMPLus_ flags with consistent upper/lower cases.
Makes use of the new CLR/EE interface to extract environment variables.
Initial support only covers COMPLUS_AltJit and COMPLUS_DUMPLLVMIR.
* Adds Options container in the JITContext. This is used to interface
with CoreCLR and hold any state. Currently implements the AltJit
detection as well as DumpLevel(COMPlus_DUMPLLVMIR) and OptLevel.
* Includes basic method set support that will be used to implement
matching of methods.
* Adds simple utility functions to do cross platform UTF16 to UTF8
conversions. This is based on the facilities supplied by LLVM.
* Fix testing scripts to prepend 'COMPlus_' on the DUMPLLVMIR variable.
83 GC related test cases were just added to CoreCLR repository.
12 of them cannot run with LLILC at this point. Exclude them for now.
Exclusion is added in both powershell script and python script.
Passed verification with llilc_test.
Also added newline at the end of exclusion file.
This script runs CoreCLR test with specified LLILC JIT and pre-built
CoreCLR runtime. If verbose level is specified, either summary or
verbose, a result will be created in default location or a place
that is specified. In verbose case, the result will be normalized
with filter.
It is required to run this script from tests directory in a CoreCLR
repository.
To exclude undesired test cases, please edit exclusion file.
usage: llilc_runtest.py [-h] [-a {x86,x64}] [-b {debug,release}]
[-d {summary,verbose}] [-r RESULT_PATH] -j JIT_PATH -c
CORECLR_RUNTIME_PATH
optional arguments:
-h, --help show this help message and exit
-a {x86,x64}, --arch {x86,x64}
the target architure
-b {debug,release}, --build {debug,release}
release or debug build of CoreCLR run-time used
-d {summary,verbose}, --dump-level {summary,verbose}
the dump level: summary, or verbose
-r RESULT_PATH, --result-path RESULT_PATH
the path to runtest result output directory
-j JIT_PATH, --jit-path JIT_PATH
full path to jit .dll
-c CORECLR_RUNTIME_PATH, --coreclr-runtime-path CORECLR_RUNTIME_PATH
full path to CoreCLR run-time binary directory
This script assumes developer handles the work environment by themselves
and CoreCLR test has been built.
llilc_checkpass will follow.
This script is LLILC wrapper of underlying CoreCLR runtest which is
now still Windows only and about to get cross-platform. When that happens,
the implementation of this script will be updated accordingly. But the
command line interface is intended to remain the same.
Update python filter with the following changes. These chagnes are needed to
allow applying filter easily from runtest.py.
1. Add ApplyAll to apply filter recursively on a directory.
2. Wrap the original functionality into ApplyOne.
3. Make script applyfilter main function to use ApplyOne to make it compatbile with before
4. Command line parsing added.
ICastable is a new test (to us) and is failing. Disable it until we can
investigate. This enables the lab with the current set of tests with
the up todate CoreCLR.
This changes allows people that haven't been 'sourcing' the file before
to operate in the same way. Just a convience until this script is
replaced in the near term.
Modified pointers to CoreCLR binaries and tests assets to point at a
live CoreCLR enlistment. Currently I can run 'runtest' with a CoreCLR
enlistment refered to by an environment variable 'CoreCLRSource' that
points at a 'coreclr' directory.
Powershell text processing is super slow. It costed about 1000 sec
to perform verbose LLVM IR dump normalization. A script written in
Python reduced it to 100 sec, handling the same pattern. Perl script
is even faster. But considering maintainability and reducing the
number of external tools, we decide to choose Python script.
Credit should go to Joe Tremoulet who provided the python filter. There
is further potential here above ApplyFilterAll level by dividing the files
based on the first letter of their base names and using Start-Job facility.
Check this version in first so that people can get the benefit of
speed-up early.
When we have newly passing methods jitted by the new jit, we should call
out how many new methods we jit. This change updates LLILCEnv to count
the number of newly successfully jitted methods, and outputs the number
for the developer. We output the number of newly jitted methods when
there are no new failing methods and when there are newly failed methods
so the developer has all the information about their change.
As of now, only release version of CoreCLR Runtime NuGet package is
available. BuildTest and RunTest cannot be run with debug build option
yet.
Added code to give out warning message for developer when it is used.
Also documented this on GitHub wiki page here:
https://github.com/dotnet/llilc/wiki/Dev-Env-and-Test-Harness
This is a major change to runtest and checkdiff.
First, it is changed to user pre-created baseline. Instead of
huge LLVM IR baseline stored in GitHub that caused issues.
Second, summary comparison is introduced. With one line
per method, the runtest and checkdiff process is much more
swift with summary only than previous approach. No dump
runtest is also provided. Verbose dump functionality is still
there. A normalization speed improvement is expected after
this change.
Third, LLVM IR dump level is introduced into LLILCJit.
Right now it read env var directly. It will be updated
when CLR Config is supported in LLILCJit.
A GitHub wiki page is created to illustrate the usage.
https://github.com/dotnet/llilc/wiki/Dev-Env-and-Test-Harness
Below are Typical use cases:
Developer Usage
1.Update your master to latest master
2.Build your baseline JIT: BuildAll or Build
3.Create a summary baseline: RunTest -Jit baseline-jit -Result SummaryBase -Dump Summary
4.Create a verbose baseline: RunTest -Jit baseline-jit -Result VerboseBase -Dump Verbose
5.Create your own branch
6.Do the new development or fix issues
7.Build your new JIT: Build
8.RunTest -Jit new-jit
If failed with overall pass/failure and no dump, back to step 6 for fixing.
9.RunTest -Jit new-jit -Result SummaryDiff -Dump Summary
10.CheckDiff -Base SummaryBase -Diff SummaryDiff -Summary -UseDiffTool
If failed with summary, find out which method is a new failure, back to step 6 for fixing.
11.RunTest -Jit new-jit -Result VerboseDiff -Dump Verbose
12.CheckDiff -Base VerboseBase -Diff VerboseDiff -UseDiffTool
If any LLVM IR diff is not benign, back to step 6 for fixing.
13.Ready for a pull request.
Lab PR Usage
RunTest
Lab PR usage is most simplified version. It only checks for overall pass/fail and does not keep any result.
Lab Nightly Usage
1.RunTest -Jit last-known-good-jit -Result SummaryBase -Dump Summary
or a summary result of running last-know-good-jit is kept in SummaryBase
2.RunTest -Jit latest-jit -Result SummaryDiff -Dump Summary
3.CheckDiff -Base SummaryBase -Diff SummaryDiff -Summary
4.If CheckDiff result is zero, mark latest-jit as last-known-good-jit. Result SummaryDiff could be kept as SummaryBase.
5.If CheckDiff result is not zero, send out failure notice to branch owner. Branch owner should analyze the case and bring relevant developer for fixing.
Implement the family of stelem instructions.
Test pass rate goes up from
129 tests, 34982 methods, 25451 good, 9531 bad (72% good)
to
129 tests, 36144 methods, 28195 good, 7949 bad (78% good)
Closes#188.
Such calls have a special calling convention that requires the stub
address in a target-specific register.
Issue #267 tracks adding support for calls that require VSD.
Treat the GCness of the address and the GCness of the loaded value
independently.
Use GC pointer (to i8) rather than integer when asked to create a GC
pointer destination.
Using explicit updates to move the LKG forward will allow verifying that
CoreCLR test asset updates work correctly with LLILC before merging them
into LLILC master testing, avoiding unpredictable external breaks.
`entryLabel ()` was used to enumerate the optimistic tail recursion branches. Since we don't do this optimization we don't need to ask for the entry label. I added a flag `HasOptimisticTailRecursionTransform` to `ReaderBase` that the client can set if such branches are added, and simplified `fgFixRecursiveEdges` to fail only if it sees that flag and `HasLocAlloc`.
Net result is that the two cases that failed with this NYI -- Path.NormalizePath and PathHelper.GetFullPathName -- now fail handling mul.ovf, and those mul's feed locallocs. So no net gain on methods passing but we gain bit more clarity on what it will take to get them handled properly.
Implement ldelem and the family of ldelema instructions.
Fix layout of arrays of ref types. CoreCLR as well as
desktop CLR .NET 4.6 don't have the field for element type
in array layout.
Add refactor genNullCheck to genConditionalThrow that is
now also used by genBoundsCheck
Fix some comment typos.
Closes#187.
Give user a control whether to update test assets every time
when the script starts. In some cases, user does not want to
automatically update the test assets every time. By default,
the test assets update is on every time. To throttle that,
call the script with swith -NoTestUpdate, no value is needed
to pass into this parameter. It will be $True for a switch.
Generate explicit compare+branch+throw sequences to check for null
derefrence on loads and stores. LLVM IR does not support annotating
loads/stores as exception-bearing, and additionally this approach may
simplify runtime porting to other targets since we'll generate explicit
machine code for these tests and the runtime won't need to mess with
machine traps. The llilc-jit-eh.md document contains more information
about the approach/rationale.
Introduce a mock configuration flag, UseExplicitNullChecks, on the Reader
object (this leaves the old codepaths accessible to facilitate future
experiments).
Extend the existing makeStore wrapper around LLVMBuilder::CreateStore, and
add a similar makeLoad wrapper around LLVMBuilder::CreateLoad. These
wrappers take an AddressMayBeNull parameter, and generate the null check
when the address may be null and UseExplicitNullChecks is true (which it
always is).
Transitively propagate the AddressMayBeNull parameter up through any
callers that are sometimes invoked to generate loads/stores for
known-non-null pointers and sometimes for maybe-null pointers.
For each method that now takes an AddressMayBeNull parameter, default that
parameter to true and add a -NonNull sibling method that passes false;
this makes the callsites more readable.
Null checks are always emitted as an exact equality comparison against 0.
For operations such as ldfld/stfld/ldlen, the check is performed against
the base pointer (before adding the field offset), which will catch any
null object references.
This test may not exactly match the Windows machine trap behavior when
unsafe code is involved (though it does conform to the ECMA spec); in
particular, ldind/ldfld/stind/stfld with a managed pointer base that is
nonzero but in the guard page would raise a NullReferenceException from
the machine trap but will not be caught by the checks inserted here.
This is a deliberate decision made after consultation with the CoreCLR
team; the approach does guarantee that null dereferences from verifiable
code will raise NullReferenceException (verifiable code cannot produce a
nonzero managed pointer pointing into the guard page), and if unsafe code
is used to construct not-quite-zero addresses which are subsequently
dereferenced, it is acceptable for such dereferences to
FailFast/SegFault/CoreDump/etc rather than raise NullReferenceException.
I've verified that there are no diffs if I recompile with the
UseExplicitNullChecks flag set to false.
Closes#203Resolves#204Closes#63
This change allows LLILC source out of tree in specified location.
Resource validation, checkenv, and test harness description has
been updated to reflect that. Separate LLILC configuration and
build command are added for out-of-tree case. BuildAll interface
remains the same. This update is compatible with previous default
in-tree LLILC source. No change is needed for the old setting.
Passed through all steps from scratch for both default in-tree
setting and specified out-of-tree setting. This change exposed
a cmake LLVMBUILD path issue as has been handled in
https://github.com/dotnet/llilc/pull/248.
- loadPrimitiveType was overzealously asserting that the address to load
from was of pointer type; the address may also be a native integer.
- Duplicate PHI nodes were being pushed onto the operand stack in the
case that new incoming edges were added to existing PHI nodes.
Closes#245.
If a tail call opportunity doesn't have the explicit `tail` prefix the jit can just generate a normal call. This turns all of our blocking tail call failures into successes or other failure types.
It also exposed an issue in storePrimitiveType where we'd widen i16 stores to i32. So I've fixed that too.
Rebased the tests since lots more methods now get handled.
This change has two parts:
(1) Pass ALTJIT settings via the LLILCTestEnv.cmd script, to match
the change in CoreClr test infrastructure.
(2) Run CoreCLR in Conservative GC mode for LLILC tests.
Testing:
LLILC Tests passed locally
This change fixes Issue# 27
- LLVM recently reverted a previous change to raise the minimum CMake
version to 2.8.12.2. This change fixes a CMake warning that was a
result of the reversion.
- Work has begun in LLVM on opaque pointer types. The first steps have
involved changing the textual representation of the LLVM IR, which
means all tests needed new baselines.
If there is an error in downloading NuGet, we want to fail early so
that we do not try to NuGet the CLR tests when NuGet does not exit.
We want to notify the user clearly that NuGet failed to download.
NuGet uses a cache for downloaded package so that a later NuGet request
does not have to download again and uses a previously downloaded package.
However sometimes on some machine it caused some unpredicted behavior
that led to different versions of CoreCLR/CoreFX packages. Finally the
difference is reflected in LLVM IR diffs we are catching. To exclude any
of the unpredictable factors, an operation is performed to clear NuGet
cache in the beginning of a session. Also fix a bug due to delaying
diff tool check to CheckDiff.
If we are not using the difftool to do any diffs, we should not be
required to have the diff tool on the path. Therefore, we should defer
checking if sgdm.exe is available to the CheckDiff function, and only
check for it if the user has specified that they want to use the diff
tool.
We unnecessarily check volatility in call helper where we always emit call instead of intrinsic for certain cases.
I removed it and added comments below when we are about to use intrinsic.
Even for the case of call, we still need to add barrier for volatile operation.
I implement memory barrier using the existing llvm fence construct which uses MFence for sequential consistency.
Bengu Li
rhadley
Joseph Tremoulet
Michelle McDaniel
Eugene Rozenfeld
Pat Gavlin
Andy Ayers
Joseph Tremoulet
Pat Gavlin
Swaroop Sridhar
Matt Mitchell
Michelle McDaniel
Kyungwoo Lee