зеркало из https://github.com/dotnet/roslyn.git
4
How To Investigate CI Test Failures
Heejae Chang редактировал(а) эту страницу 2019-05-07 16:55:40 -07:00
Determine Which Tests Are Failing
- Click details to see more information about the failing run.
- Click on the number of errors and warnings to open Azure DevOps.
- Click on the Tests tabs to see a list of failing tests.
- Expand each run to see which tests failed.
Check for Open Issues
- Open https://github.com/dotnet/roslyn/issues
- Enter the name of the failing test and search for open issues that match the failure.
Add New Occurrence to Existing Issue
- Add a new comment and include relevent information.
Good information to include in this comment:- A link to the Azure DevOps tests results.
- Which attempt the failure occurred on.
- Which test run failed.
For instance the Debug_Async run failed on the 3rd attempt.
Creating a New Issue for Failed Integration Test
- Open the Screenshot artifacts for one of the test runs.
- Focus on the first reported integration test failure. Often these failures cascade and it can be misleading.
- Create a new issue and include the failed test name in the title
- Good information to include in the issue body:
- A link to the Azure DevOps tests results.
- Which attempt the failure occurred on.
- Which test run failed.
- Additional details outlined below.
- First check to see if a new entry appears in the *.DotNet.log file for the test. There is a MissingMethodException thrown at the beginning of every test run that will log to *.DotNet.log anything thrown by a previous test run.
If the file is new for the test you are creating (check file timestamp compared to when your tests ran) or contains a different exception than the one given in the MissingMethodException entry, then treat the root cause of the test failure as whatever that exception is.
- If the screenshot shows an interesting state (OS window, etc.), or if the failure is not identified by (2), make sure to include a screenshot from the time of failure in the issue.
- If the cause is not identified by (2), include the stack trace from the TargetInvocationException leading to the test failure.
- Add them to the Flaky Tests columns in the Test Improvements project
Here is an example from a test failure on Jenkins - it's reviewable even though the CI link is broken https://github.com/dotnet/roslyn/issues/26041
Troubleshooting Integration Test Run Failures
- Failure during Checkout because of locked file
[error] One or more errors occurred. (The process cannot access the file '...\Some.File' because it is being used by another process.)- Take note of the machine name
- note - if you can't see pool name, make sure you change this to Attempt 1
- Click on the Pool name (dotnet-external-vs2019-preview) in this case.
- Click on the Agents tab
- Find the machine with the matching name and disable it.
- From the Azure DevOps Pipeline page choose to Retry the test run.
- Take note of the machine name