Follow markdown lint
This commit is contained in:
Amaury Levé
Родитель
7ebe3619dc
Коммит
54093ba98a
|
|
@ -1,8 +1,10 @@
|
|||
### Visual Studio Test Platform Documentation
|
||||
|
||||
You've found the GitHub repository which contains the content for the Visual Studio Test Platform documentation.
|
||||
If you are looking for the Visual Studio Test Platform product GitHub repository, you can find it [here](https://github.com/Microsoft/vstest).
|
||||
|
||||
### Documentation
|
||||
|
||||
- [Test Platform Architecture](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0001-Test-Platform-Architecture.md)
|
||||
- [Test Discovery Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0002-Test-Discovery-Protocol.md)
|
||||
- [Test Execution Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0003-Test-Execution-Protocol.md)
|
||||
|
|
@ -16,23 +18,28 @@ If you are looking for the Visual Studio Test Platform product GitHub repository
|
|||
- [Troubleshooting guide](https://github.com/Microsoft/vstest-docs/blob/main/docs/troubleshooting.md)
|
||||
|
||||
### Contributing
|
||||
|
||||
There are many ways to contribute to VSTest
|
||||
|
||||
- [Submit issues](https://github.com/Microsoft/vstest-docs/issues) and help verify fixes as they are checked in.
|
||||
- Review the [documentation changes](https://github.com/Microsoft/vstest-docs/pulls).
|
||||
- Contribute new topics/information, or make changes to existing documentation.
|
||||
|
||||
### Editing docs
|
||||
|
||||
We use [docfx](https://github.com/dotnet/docfx/releases) for building this documentation. A short primer on editing this repo is below.
|
||||
First, [download](https://github.com/dotnet/docfx/releases) latest release of docfx (`docfx.zip` package) and extract it locally. We will use d:\tmp\docfx as destination for these steps.
|
||||
|
||||
Open a command prompt, git clone this repo. Use the following commands to build and run a local server.
|
||||
|
||||
```
|
||||
> cd d:\src\vstest-docs
|
||||
> d:\tmp\docfx\docfx.exe build
|
||||
> d:\tmp\docfx\docfx.exe serve
|
||||
```
|
||||
|
||||
Open http://localhost:8080/_site in a browser to see the rendering of the documentation.
|
||||
Open <http://localhost:8080/_site> in a browser to see the rendering of the documentation.
|
||||
|
||||
### Microsoft Open Source Code of Conduct
|
||||
|
||||
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
|
||||
|
|
|
|||
|
|
@ -1,24 +1,25 @@
|
|||
# Passing runsettings arguments through commandline
|
||||
|
||||
You are here because you are looking for syntax and details to pass runsettings configurations to either `vstest.console.exe` or `dotnet test` through commandline.
|
||||
|
||||
`RunSettings arguments` are used to add/update specific `runsettings configurations`. The updated `runsettings configurations` will be available to `TestPlatform` and `test extensions` (E.g. test adapter, etc.) through runsettings.
|
||||
|
||||
## Syntax
|
||||
|
||||
* `RunSettings arguments` are specified as name-value pair of the form `[name]=[value]` after `-- `. Note the space after --.
|
||||
* `RunSettings arguments` are specified as name-value pair of the form `[name]=[value]` after `--`. Note the space after --.
|
||||
* Use a space to separate multiple `[name]=[value]`.
|
||||
* All the arguments after `--` will be treated as `RunSettings arguments`, means `RunSettings arguments` should be at the end of the command line.
|
||||
|
||||
## Example
|
||||
|
||||
Passing argument `-- MSTest.MapInconclusiveToFailed=True` in (1) below is equivalent to passing argument
|
||||
Passing argument `-- MSTest.MapInconclusiveToFailed=True` in (1) below is equivalent to passing argument
|
||||
`--settings additionalargs.runsettings` in (2) below.
|
||||
|
||||
```
|
||||
```shell
|
||||
1) dotnet test -- MSTest.MapInconclusiveToFailed=True MSTest.DeploymentEnabled=False
|
||||
```
|
||||
|
||||
```
|
||||
```shell
|
||||
2) dotnet test --settings additionalargs.runsettings
|
||||
```
|
||||
|
||||
|
|
@ -35,19 +36,17 @@ where `additionalargs.runsettings` is:
|
|||
</RunSettings>
|
||||
```
|
||||
|
||||
|
||||
The syntax in (1) is another way of passing runsettings configuration and you need not author a runsetting file while using `Runsettings arguments`. More details about runsettings can be found [here](https://msdn.microsoft.com/en-us/library/jj635153.aspx).
|
||||
|
||||
The syntax in (1) is another way of passing runsettings configuration and you need not author a runsetting file while using `Runsettings arguments`. More details about runsettings can be found [here](https://msdn.microsoft.com/library/jj635153.aspx).
|
||||
|
||||
`Runsettings arguments` takes precedence over `runsettings`.
|
||||
|
||||
For example, in below command the final value for `MapInconclusiveToFailed` will be `False` and value for `DeploymentEnabled` will be unchanged, that is `False`.
|
||||
|
||||
```
|
||||
```shell
|
||||
dotnet test --settings additionalargs.runsettings -- MSTest.MapInconclusiveToFailed=False
|
||||
```
|
||||
|
||||
You can also set `TestRunParameters` using command line option:
|
||||
You can also set `TestRunParameters` using command line option:
|
||||
|
||||
```cmd
|
||||
# cmd
|
||||
|
|
@ -60,4 +59,4 @@ dotnet test --% -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\
|
|||
dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\)
|
||||
```
|
||||
|
||||
In this example, `\"myParam\"` corresponds to the name of you parameter and `\"value\"` - the value of your parameter. Note, that `\` are escaping characters and they should stay as shown above.
|
||||
In this example, `\"myParam\"` corresponds to the name of you parameter and `\"value\"` - the value of your parameter. Note, that `\` are escaping characters and they should stay as shown above.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,11 @@
|
|||
# Monitor and analyze test run
|
||||
|
||||
This document will walk you through enabling data collection for a test run covering:
|
||||
|
||||
1. A brief overview of DataCollectors.
|
||||
2. Configuring DataCollectors in TPv2.
|
||||
3. Key differences for using DataCollectors in TPv2 v/s TPv1.
|
||||
3. Instructions for using [code coverage][coverage].
|
||||
1. Configuring DataCollectors in TPv2.
|
||||
1. Key differences for using DataCollectors in TPv2 v/s TPv1.
|
||||
1. Instructions for using [code coverage][coverage].
|
||||
|
||||
> **Version note:**
|
||||
>
|
||||
|
|
@ -13,7 +15,9 @@ This document will walk you through enabling data collection for a test run cove
|
|||
[coverage]: #coverage
|
||||
|
||||
## DataCollector
|
||||
|
||||
A DataCollector is a test platform extension to monitor test run. It can be extended to perform tasks on specific test execution events. Currently, four events are exposed to DataCollector:
|
||||
|
||||
1. Session Start event.
|
||||
2. Test Case Start event
|
||||
3. Test Case End event.
|
||||
|
|
@ -25,10 +29,13 @@ Please refer [here](https://github.com/Microsoft/vstest-docs/blob/main/docs/exte
|
|||
if you're interested in the architecture of data collection.
|
||||
|
||||
## Configure DataCollectors
|
||||
|
||||
DataCollectors can be configured for monitoring test execution through runSettings, testSettings or vstest.console args.
|
||||
|
||||
### Using RunSettings<a name="Using-RunSettings"></a>
|
||||
|
||||
Below is the sample runsettings for a custom DataCollector
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<RunSettings>
|
||||
|
|
@ -49,21 +56,28 @@ Below is the sample runsettings for a custom DataCollector
|
|||
</DataCollectionRunSettings>
|
||||
</RunSettings>
|
||||
```
|
||||
|
||||
Below is the sample command for enabling DataCollectors using runsettings
|
||||
```
|
||||
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.runsettings
|
||||
```
|
||||
|
||||
### Using vstest.console args<a name="Using-vstest.console-args"></a>
|
||||
|
||||
DataCollectors can be configured and used through first class command line arguments `/collect` and `/testadapterpath`. Hence, for common DataCollection scenarios, separate runsettings file may not be required.
|
||||
|
||||
Below is the sample command to configure and use DataCollectors through vstest.console command line.
|
||||
```
|
||||
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"Code Coverage"
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"MyDataCollector" /testadapterpath:<Path to MyDataCollector assembly>
|
||||
```
|
||||
|
||||
Please note that `testadapterpath` is not required for DataCollectors shipped along with TPv2.
|
||||
|
||||
### Using TestSettings
|
||||
|
||||
While the recommended way is to use [runsettings](#Using-RunSettings) or [vstest.console args](#Using-vstest.console-args), there are few DataCollectors which only worked with testsettings.
|
||||
E.g.: `System Information` DataCollector. Below is the sample testsettings for using `System Information` DataCollector.
|
||||
|
||||
|
|
@ -83,12 +97,16 @@ E.g.: `System Information` DataCollector. Below is the sample testsettings for u
|
|||
<Properties />
|
||||
</TestSettings>
|
||||
```
|
||||
|
||||
Below is the sample command for enabling DataCollectors using testsettings
|
||||
```
|
||||
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.testsettings
|
||||
```
|
||||
|
||||
### Enable/Disable a DataCollector
|
||||
All DataCollectors configured in the .runsettings files are loaded automatically and are enabled to participate for run, unless explicitely disabled using boolean valued attribute named `enabled`.
|
||||
|
||||
All DataCollectors configured in the .runsettings files are loaded automatically and are enabled to participate for run, unless explicitly disabled using boolean valued attribute named `enabled`.
|
||||
|
||||
For example, only `MyDataCollector1` DataCollector will be enabled for a test run with
|
||||
below runsettings:
|
||||
|
|
@ -116,19 +134,22 @@ below runsettings:
|
|||
|
||||
A specific DataCollector can be explicitely enabled using the `/collect:<friendly name>` command line switch.
|
||||
|
||||
For example, below command line will enable a DataCollector named `MyDataCollector1`
|
||||
For example, below command line will enable a DataCollector named `MyDataCollector1`
|
||||
(and disable other DataCollectors mentioned in .runsettings):
|
||||
```
|
||||
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /settings:<Path to runSettings>
|
||||
```
|
||||
|
||||
|
||||
More than one DataCollectors can also be enabled using `/collect` command line switch
|
||||
|
||||
For example, below command will enable DataCollectors named `MyDataCollector1` and `MyDataCollector2`:
|
||||
```
|
||||
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /collect:MyDataCollector2 /settings:<Path to runSettings>
|
||||
```
|
||||
## Key differences for using DataCollectors in TPv2 v/s TPv1.
|
||||
|
||||
## Key differences for using DataCollectors in TPv2 v/s TPv1
|
||||
|
||||
1. In TPv1, DataCollectors are loaded from `<<VisualStudio Installation Directory>>\Common7\IDE\PrivateAssemblies\DataCollectors`.
|
||||
In TPv2, DataCollectors are loaded from `TestAdaptersPaths` specified in runSettings or `/testadapterpath` argument of `vstest.console.exe`. DataCollector assemblies must follow the naming convention *collector.dll.
|
||||
|
|
@ -138,9 +159,12 @@ In TPv2, DataCollectors are loaded from `TestAdaptersPaths` specified in runSett
|
|||
3. There are breaking changes in latest DataCollector interface. Hence, older DataCollectors need to be rebuilt against latest APIs to work with TPv2. For details, refer [here(todo)]();
|
||||
|
||||
## Working with Code Coverage<a name="coverage"></a>
|
||||
|
||||
> **Requirements:**
|
||||
> Code Coverage requires the machine to have Visual Studio 2017 Enterprise ([15.3.0](https://www.visualstudio.com/vs) or later installed and a Windows operating system.
|
||||
|
||||
### Setup a project
|
||||
|
||||
Here's a sample project file, please note the xml entity marked as `Required`. Previously, the `Microsoft.VisualStudio.CodeCoverage` was required, but is now shipped with the SDK.
|
||||
|
||||
```xml
|
||||
|
|
@ -162,49 +186,56 @@ Here's a sample project file, please note the xml entity marked as `Required`. P
|
|||
</Project>
|
||||
```
|
||||
|
||||
[coveragenuget]: https://www.nuget.org/packages/Microsoft.CodeCoverage/
|
||||
|
||||
### Analyze coverage with Visual Studio
|
||||
|
||||
> **Version note:**
|
||||
>
|
||||
>
|
||||
> Try this feature with [Visual Studio 2017 15.3.0](https://www.visualstudio.com/vs) or later.
|
||||
|
||||
Use the `Analyze Code Coverage` context menu available in `Test Explorer` tool window to start a coverage run.
|
||||
|
||||
After the coverage run is complete, a detailed report will be available in the `Code Coverage Results` tool window.
|
||||
|
||||
Please refer the documentation for additional details: https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested
|
||||
Please refer the documentation for additional details: <https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested>
|
||||
|
||||
### Collect coverage with command line runner
|
||||
|
||||
Use the following command line to collect coverage data for tests:
|
||||
|
||||
```
|
||||
```shell
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" --collect:"Code Coverage" --framework:".NETCoreApp,Version=v1.1" d:\testproject\bin\Debug\netcoreapp1.1\testproject.dll
|
||||
```
|
||||
|
||||
This will generate a `*.coverage` file in the `<Current working directory>\TestResults` directory.
|
||||
|
||||
### Event Log Data Collector
|
||||
|
||||
This document introduces Event Log DataCollector. We will start with a brief overview of Event Log DataCollector, use cases where it will be useful followed by steps to enable it.
|
||||
|
||||
#### Introduction
|
||||
|
||||
Event Log DataCollector is a Windows only DataCollector that is used to get event logs logged into Windows Event Viewer during test execution. Event logs are saved in a file `Event Log.xml` and this file is available as Attachment as part of test result report (trx).
|
||||
When enabled, Event Log DataCollector generates one `Event Log.xml` file for entire test session. `Event Log.xml` files are also generated corresponding to all test cases as well, to provide a granular view of events logged while executing a test case.
|
||||
|
||||
|
||||
More info on Event Viewer [here](https://technet.microsoft.com/en-us/library/cc938674.aspx)
|
||||
|
||||
#### Use cases for Event Log DataCollector
|
||||
Event Log DataCollector is used to get event logs as Attachment and is particularly useful for remote scenarios where logging into the machine and viewing the Event Viewer is not possible.
|
||||
|
||||
Event Log DataCollector is used to get event logs as Attachment and is particularly useful for remote scenarios where logging into the machine and viewing the Event Viewer is not possible.
|
||||
|
||||
#### Enabling Event Log DataCollector
|
||||
|
||||
There are two ways of enabling Event Log DataCollector for a test run:
|
||||
##### 1. Using vstest.console argument.
|
||||
|
||||
##### 1. Using vstest.console argument
|
||||
|
||||
Use the following command to enable Event Log DataCollector with default configuration:
|
||||
|
||||
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /testadapterpath:<<Path to test adapter>> /collect:"Event Log"
|
||||
|
||||
##### 2. Using runsettings.
|
||||
##### 2. Using runsettings
|
||||
|
||||
Below runsettings can be used to enable Event Log DataCollector.
|
||||
|
||||
```xml
|
||||
|
|
@ -225,7 +256,8 @@ Below runsettings can be used to enable Event Log DataCollector.
|
|||
</DataCollectionRunSettings>
|
||||
</RunSettings>
|
||||
```
|
||||
The above runsettings will collect event logs from `System` and `Application` event logs which are logged as `Error` or `Warning` and event source is specified as `CustomEventSource`. `MaxEventLogEntriesToCollect` specifies the upper limit on the events that are logged in `Event Log.xml` file corresponding to test cases. There is no upper limit on number of events logged in `Event Log.xml` file for test session.
|
||||
|
||||
The above runsettings will collect event logs from `System` and `Application` event logs which are logged as `Error` or `Warning` and event source is specified as `CustomEventSource`. `MaxEventLogEntriesToCollect` specifies the upper limit on the events that are logged in `Event Log.xml` file corresponding to test cases. There is no upper limit on number of events logged in `Event Log.xml` file for test session.
|
||||
|
||||
In default configuration (through vstest.console.exe args or when <Configuration> section is empty in runsettings), `System`, `Application` and `Security` logs with entry types `Error`, `Warning` or `FailureAudit` and with any event source are collected. Default value of `MaxEventLogEntriesToCollect` is 50000. There is no upper limit on number of events logged in `Event Log.xml` file for test session.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
# Configure a test run
|
||||
|
||||
This document covers configuration of a test run in the test platform.
|
||||
|
||||
## Overview
|
||||
|
||||
There are three different ways to configure various aspects of a test run.
|
||||
|
||||
1. **Using command line arguments**
|
||||
|
|
@ -10,13 +12,13 @@ test` command line. For example, `--framework` can specify the runtime framework
|
|||
version, or `--platform` can specify the architecture of test run (`x86` or
|
||||
`x64`).
|
||||
|
||||
2. **Using a runsettings file**
|
||||
1. **Using a runsettings file**
|
||||
User can specify a `runsettings` file to configure test run. For example:
|
||||
|
||||
* `> vstest.console.exe --settings:test.runsettings test.dll`
|
||||
* `> dotnet test -s test.runsettings`
|
||||
|
||||
3. **Using command line runsettings parameters**
|
||||
1. **Using command line runsettings parameters**
|
||||
Various elements of a `runsettings` file can also specified as command line
|
||||
parameters directly. For example, consider following `runsettings`:
|
||||
|
||||
|
|
@ -31,20 +33,21 @@ parameters directly. For example, consider following `runsettings`:
|
|||
The `DisableAppDomain` settings can be changed in the following way:
|
||||
`> vstest.console --settings:test.runsettings -- RunConfiguration.DisableAppDomain=true`
|
||||
|
||||
Or, `> dotnet test -s test.runsettings -- RunConfiguration.DisableAppDomain=true` if
|
||||
Or, `> dotnet test -s test.runsettings -- RunConfiguration.DisableAppDomain=true` if
|
||||
you're using dotnet test to execute tests.
|
||||
|
||||
Order of priority (in case of conflicting settings) is as follows:
|
||||
|
||||
1. If a command line switch is available, it takes priority. E.g. `/platform`
|
||||
wins over `<TargetPlatform>` specified in the `runsettings` file.
|
||||
2. If a command line runsettings is available, it takes priority over the
|
||||
1. If a command line runsettings is available, it takes priority over the
|
||||
content of `runsettings` file. E.g. in case of `dotnet test -s
|
||||
test.runsettings -- RunConfiguration.TargetPlatform=x86`, platform is set to
|
||||
`x86` even if `test.runsettings` specifies `x64` as platform.
|
||||
3. If a `runsettings` file is provided, it is used for the test run.
|
||||
1. If a `runsettings` file is provided, it is used for the test run.
|
||||
|
||||
## Run settings
|
||||
|
||||
A `*.runsettings` file is used to configure various aspects of a test discovery
|
||||
and execution with `vstest.console.exe` or the `Test Explorer` in VS. An editor
|
||||
using the test platform, can specify a `runsettings` as an `xml` in the `Test
|
||||
|
|
@ -53,28 +56,30 @@ Discovery` or `Test Run` requests (details are in Editor API Specification).
|
|||
The `runsettings` file is a xml file with following sections:
|
||||
|
||||
1. Run Configuration
|
||||
2. Data Collection
|
||||
3. Runtime Parameters
|
||||
4. Adapter Configuration
|
||||
5. Legacy Settings
|
||||
1. Data Collection
|
||||
1. Runtime Parameters
|
||||
1. Adapter Configuration
|
||||
1. Legacy Settings
|
||||
|
||||
We will cover these sections in detail later in the document. Let's discuss few
|
||||
core principles for runsettings.
|
||||
|
||||
### Principles
|
||||
|
||||
1. Any runner (CLI, IDE, Editor) can use run settings to configure the test run
|
||||
2. These runners need to take care of merging user provided runsettings with
|
||||
1. These runners need to take care of merging user provided runsettings with
|
||||
their own configuration parameters. E.g. an IDE may provide `TargetPlatform`
|
||||
as an UI configuration, at the same time an user can also provide the setting
|
||||
in a `runsettings` file. It is the IDE's responsibility to disambiguate.
|
||||
3. Run settings are always created at the start of a test run. They are
|
||||
1. Run settings are always created at the start of a test run. They are
|
||||
immutable for rest of the run.
|
||||
4. Every test platform extension should get the _effective_ runsettings for a
|
||||
1. Every test platform extension should get the _effective_ runsettings for a
|
||||
test run via the test platform APIs. It may use it to read its own settings
|
||||
or take decisions (e.g. an adapter needs to disable appdomains if
|
||||
`DisableAppDomain` setting is provided via run settings).
|
||||
|
||||
### Sample
|
||||
|
||||
Given below is a complete `runsettings` file with all available options. Each
|
||||
option is briefly annotated, details are available in a later section of this
|
||||
document.
|
||||
|
|
@ -180,6 +185,7 @@ document.
|
|||
```
|
||||
|
||||
### Section 1: Run Configuration
|
||||
|
||||
```xml
|
||||
<RunSettings>
|
||||
<RunConfiguration>
|
||||
|
|
@ -196,7 +202,8 @@ Available elements are:
|
|||
|
||||
1. **Test settings**
|
||||
|
||||
*Example*
|
||||
_Example_
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<RunSettings>
|
||||
|
|
@ -212,8 +219,8 @@ Available elements are:
|
|||
</RunConfiguration>
|
||||
</RunSettings>
|
||||
```
|
||||
|
||||
*Description*
|
||||
|
||||
_Description_
|
||||
|
||||
| Settings Element | Type | Function |
|
||||
|-------------------|--------|-------------------------------------------------------------------------------------------------|
|
||||
|
|
@ -227,6 +234,7 @@ Available elements are:
|
|||
| ExecutionThreadApartmentState | string | Apartment state of thread which calls adapter's RunTests and Cancel APIs. Possible values: (MTA, STA). default is STA for .NET Full and MTA for .NET Core. STA supported only for .NET Full **Required Version: 15.5+.** [More details.](#execution-thread-apartment-state) |
|
||||
|
||||
Examples of valid `TargetFrameworkVersion`:
|
||||
|
||||
* .NETCoreApp, Version=v1.0
|
||||
* .NETCoreApp, Version=v1.1
|
||||
* .NETFramework, Version=v4.5
|
||||
|
|
@ -238,7 +246,8 @@ These settings are a hint to adapters to behave in a particular way. These are
|
|||
determined by runners based on user configuration. It is also possible for an
|
||||
user to provide these settings, if they want to tweak a run.
|
||||
|
||||
*Example*
|
||||
_Example_
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<RunSettings>
|
||||
|
|
@ -251,7 +260,7 @@ user to provide these settings, if they want to tweak a run.
|
|||
</RunSettings>
|
||||
```
|
||||
|
||||
*Description*
|
||||
_Description_
|
||||
|
||||
| Settings Element | Type | Function |
|
||||
|--------------------------|------|------------------------------------------------------------------------------------|
|
||||
|
|
@ -264,7 +273,8 @@ user to provide these settings, if they want to tweak a run.
|
|||
An IDE/Editor can set these settings to change the behavior of test platform.
|
||||
They are not actionable for an adapter.
|
||||
|
||||
*Example*
|
||||
_Example_
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<RunSettings>
|
||||
|
|
@ -274,13 +284,14 @@ They are not actionable for an adapter.
|
|||
</RunSettings>
|
||||
```
|
||||
|
||||
*Description*
|
||||
_Description_
|
||||
|
||||
| Settings Element | Type | Function |
|
||||
|------------------|------|-------------------------------------------------------------------------------------------------------------|
|
||||
| BatchSize | int | Configures the frequency of run statistics. Discovered/Test results are send once `n` tests are accumulated |
|
||||
|
||||
### Section 2: Data Collection
|
||||
|
||||
```xml
|
||||
<RunSettings>
|
||||
<!-- Configurations for out-of-process data collectors -->
|
||||
|
|
@ -313,7 +324,8 @@ Data collector extension authors may use the content within the specific
|
|||
for their data collector. For example, the code coverage data collector uses
|
||||
following section.
|
||||
|
||||
*Example*
|
||||
_Example_
|
||||
|
||||
```xml
|
||||
<RunSettings>
|
||||
<!-- Configurations for data collectors -->
|
||||
|
|
@ -341,15 +353,16 @@ following section.
|
|||
</DataCollectors>
|
||||
</DataCollectionRunSettings>
|
||||
</RunSettings>
|
||||
```
|
||||
```
|
||||
|
||||
*Description*
|
||||
_Description_
|
||||
|
||||
| Settings Element | Type | Function |
|
||||
|------------------|--------|---------------------------------------------------------------------------|
|
||||
| DataCollector | string | Provides a data collector information. See below for required attributes. |
|
||||
|
||||
Required attributes:
|
||||
|
||||
1. `friendlyName` provides a common name for the data collector. It is declared
|
||||
by a data collector implementation.
|
||||
2. `enabled` is used to enable/disable a data collector. If `true` the data
|
||||
|
|
@ -375,6 +388,7 @@ These settings can be used for Ordered tests and MSTest v1 based tests when runn
|
|||
Users can now migrate testsettings to runsettings using [SettingsMigrator](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0023-TestSettings-Deprecation.md#migration)
|
||||
|
||||
Here is an example:
|
||||
|
||||
```xml
|
||||
<Runsettings>
|
||||
<MSTest>
|
||||
|
|
@ -421,6 +435,7 @@ Here is an example:
|
|||
```
|
||||
|
||||
#### Run Parameters
|
||||
|
||||
```xml
|
||||
<!-- Parameters used by tests at runtime -->
|
||||
<TestRunParameters>
|
||||
|
|
@ -439,6 +454,7 @@ available to tests during runtime.
|
|||
> `TestContext` API.
|
||||
|
||||
#### Adapter Configuration
|
||||
|
||||
An adapter may provide a section in runsettings for users. See [mstest config][]
|
||||
and [nunit config][] for more details.
|
||||
|
||||
|
|
@ -446,10 +462,13 @@ and [nunit config][] for more details.
|
|||
[nunit config]: TODO
|
||||
|
||||
# Execution thread apartment state
|
||||
|
||||
This section explains usage of ExecutionThreadApartmentState element in runsettings and testplatform behavior for same.
|
||||
|
||||
### Usage:
|
||||
#### using runsettings file:
|
||||
### Usage
|
||||
|
||||
#### using runsettings file
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<RunSettings>
|
||||
|
|
@ -459,22 +478,24 @@ This section explains usage of ExecutionThreadApartmentState element in runsetti
|
|||
</RunSettings>
|
||||
```
|
||||
|
||||
#### using command line:
|
||||
#### using command line
|
||||
|
||||
vstest.console.exe a.dll -- RunConfiguration.ExecutionThreadApartmentState=STA
|
||||
|
||||
dotnet test -f net46 -- RunConfiguration.ExecutionThreadApartmentState=STA
|
||||
|
||||
### History
|
||||
|
||||
In Test Platform V1 ExecutionThreadApartmentState property can be set from vstest.executionengine*.exe.config file. Default value is `STA`.
|
||||
|
||||
### Behavior
|
||||
|
||||
In Test platform V2 ExecutionThreadApartmentState property default value is `MTA` for .NET Core and `STA` for .NET Full. `STA` value is only supported for .NET Framework.
|
||||
Warning should be shown on trying to set value `STA` for .NET Core and UAP10.0 frameworks tests.
|
||||
|
||||
* To support adapters which depends on thread test platform creates may need STA apartment state to run UI tests. `ExecutionThreadApartmentState` option can be used to set apartment state. Example: MSTest v1, MSTest v2 and MSCPPTest adapters.
|
||||
|
||||
- To support adapters which depends on thread test platform creates may need STA apartment state to run UI tests. `ExecutionThreadApartmentState` option can be used to set apartment state. Example: MSTest v1, MSTest v2 and MSCPPTest adapters.
|
||||
|
||||
- The recommended way to make the tests run in STA thread is using custom attributes that adapter provides.
|
||||
* The recommended way to make the tests run in STA thread is using custom attributes that adapter provides.
|
||||
|
||||
| Adapter | Attribute|
|
||||
|-----------|-----------|
|
||||
|
|
|
|||
|
|
@ -4,24 +4,27 @@ This article will help you build, test and try out local builds of the VS test
|
|||
platform.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Please ensure you have a `.net 4.6.2` or higher installed on the machine.
|
||||
|
||||
Clone the repository to a local directory. Rest of this article assumes
|
||||
`/src/vstest` as the location of source enlistment.
|
||||
|
||||
```
|
||||
```shell
|
||||
> git clone https://github.com/Microsoft/vstest.git
|
||||
```
|
||||
|
||||
If you're planning to use **Visual Studio** as development environment, please
|
||||
install `VS 2022` with .NET Desktop development workload, and install individual component `.NET Portable Library targeting pack`.
|
||||
|
||||
If you're _not_ planning to use **Visual Studio** and only use CLI. You will need to install [.Net 46 targeting pack](https://www.microsoft.com/en-us/download/details.aspx?id=48136). The download link has two msis. Both needs to be installed. Otherwise build will fail asking to install net 46.
|
||||
If you're _not_ planning to use **Visual Studio** and only use CLI. You will need to install [.Net 46 targeting pack](https://www.microsoft.com/download/details.aspx?id=48136). The download link has two msis. Both needs to be installed. Otherwise build will fail asking to install net 46.
|
||||
|
||||
### Unix requirements
|
||||
|
||||
Install common tools
|
||||
|
||||
```shell
|
||||
$ sudo apt install libcurl4-openssl-dev
|
||||
sudo apt install libcurl4-openssl-dev
|
||||
```
|
||||
|
||||
Follow the instructions on [Mono Installation][mono-linux] page to install latest bits.
|
||||
|
|
@ -29,6 +32,7 @@ Follow the instructions on [Mono Installation][mono-linux] page to install lates
|
|||
[mono-linux]: http://www.mono-project.com/download/#download-lin
|
||||
|
||||
Rest of the article will provide steps for VS and CLI/Editors development.
|
||||
|
||||
## Build
|
||||
|
||||
### Building with Visual Studio
|
||||
|
|
@ -44,7 +48,7 @@ Binaries for each assembly are produced in the
|
|||
|
||||
To build the repository, run the following command:
|
||||
|
||||
```
|
||||
```shell
|
||||
> cd /src/vstest
|
||||
> build.cmd
|
||||
```
|
||||
|
|
@ -71,7 +75,7 @@ assemblies can be found at
|
|||
To build a particular configuration, use the `-c` option. E.g. to trigger a
|
||||
release build use
|
||||
|
||||
```
|
||||
```shell
|
||||
> build.cmd -c Release
|
||||
```
|
||||
|
||||
|
|
@ -82,18 +86,18 @@ For other options, check `/src/vstest/scripts/build.ps1`.
|
|||
There are following sets of tests:
|
||||
|
||||
* Unit tests
|
||||
- Very fast tests primarily validating individual units
|
||||
- Named as `<UnitUnderTest>.UnitTests` where UnitUnderTest is any product
|
||||
* Very fast tests primarily validating individual units
|
||||
* Named as `<UnitUnderTest>.UnitTests` where UnitUnderTest is any product
|
||||
assembly
|
||||
* Smoke tests
|
||||
- Slower end to end tests. Typically these cover P0 scenarios (99% of users
|
||||
* Slower end to end tests. Typically these cover P0 scenarios (99% of users
|
||||
will use these, if these are broken, PR will not be merged)
|
||||
- Driven by a real `vstest.console` executable
|
||||
- Named as `Microsoft.TestPlatform.SmokeTests`
|
||||
* Driven by a real `vstest.console` executable
|
||||
* Named as `Microsoft.TestPlatform.SmokeTests`
|
||||
* End to end tests
|
||||
- Slower end to end tests for extensive coverage
|
||||
- Driven by a real `vstest.console` executable
|
||||
- Named as `Microsoft.TestPlatform.AcceptanceTests`
|
||||
* Slower end to end tests for extensive coverage
|
||||
* Driven by a real `vstest.console` executable
|
||||
* Named as `Microsoft.TestPlatform.AcceptanceTests`
|
||||
|
||||
As a principle, most of tests are unit tests (~70-80%), few smoke tests
|
||||
(~20-15%), fewer acceptance tests (~10-5%).
|
||||
|
|
@ -114,7 +118,7 @@ run only unit tests. For running smoke tests, use the `project:"Smoke"` filter.
|
|||
|
||||
To execute tests, run the following command:
|
||||
|
||||
```
|
||||
```shell
|
||||
> cd /src/vstest
|
||||
> test.cmd
|
||||
```
|
||||
|
|
@ -122,27 +126,27 @@ To execute tests, run the following command:
|
|||
By default, only unit tests are run. To run the smoke tests, provide the `-p`
|
||||
option to `test.cmd` to set test assembly pattern:
|
||||
|
||||
```
|
||||
```shell
|
||||
> test.cmd -p smoke
|
||||
```
|
||||
|
||||
The `-p` option can be used to run tests for any assembly as well. E.g.
|
||||
following command will run tests for *datacollector*:
|
||||
following command will run tests for _datacollector_:
|
||||
|
||||
```
|
||||
```shell
|
||||
> test.cmd -p datacollector
|
||||
```
|
||||
|
||||
Tests for a particular configuration can be run with following command. By
|
||||
default, `Debug` configuration is run.
|
||||
|
||||
```
|
||||
```shell
|
||||
> test.cmd -c release
|
||||
```
|
||||
|
||||
If you want to run a particular test. Eg: Test Name that contains Blame in Acceptance test
|
||||
|
||||
```
|
||||
```shell
|
||||
> test.cmd -p accept -f net451 -filter blame
|
||||
```
|
||||
|
||||
|
|
@ -156,7 +160,7 @@ using previous instructions.
|
|||
Visual Studio 2017 RC ships with the test platform `vsix` generated by build.cmd. To use the locally
|
||||
built version, use the following steps in a developer command prompt.
|
||||
|
||||
```
|
||||
```shell
|
||||
> vsixinstaller /src/vstest/artifacts/Debug/TestPlatform.vsix
|
||||
# (replace Debug with Release as appropriate)
|
||||
```
|
||||
|
|
@ -168,7 +172,7 @@ A `netcoreapp` target of vstest.console is dropped at
|
|||
executed with any dotnet executable. You may choose to use the dotnet-cli we
|
||||
have in `/src/vstest/tools/dotnet/dotnet` as well :)
|
||||
|
||||
```
|
||||
```shell
|
||||
> /src/vstest/tools/dotnet/dotnet /src/vstest/artifacts/Debug/netcoreapp1.0/vstest.console.dll /?
|
||||
```
|
||||
|
||||
|
|
@ -178,7 +182,7 @@ A `net46` target of vstest.console is dropped at
|
|||
`src/vstest/artifacts/<Configuration>/net46/win7-x64`. It can be run directly as
|
||||
follows:
|
||||
|
||||
```
|
||||
```shell
|
||||
> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe /?
|
||||
```
|
||||
|
||||
|
|
@ -188,7 +192,7 @@ Try to isolate the failure scenario. In the best case it's just a command line
|
|||
that can demonstrate a bug. For example, a bug in discovery of tests can show up
|
||||
in following command line:
|
||||
|
||||
```
|
||||
```shell
|
||||
> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe mytest.dll /listTests /tests:*&&*#Ed
|
||||
```
|
||||
|
||||
|
|
@ -199,7 +203,8 @@ Another add a `Debugger.Launch` at the process launch points. E.g.
|
|||
CoreCLR for netcoreapp scenario) and step through the code.
|
||||
|
||||
## Running Tests with TPv2 using Test Explorer
|
||||
Test Platform (TPv2) is packaged as a `vsix` with VS 2017 RC releases and lights up the .NET core and Live Unit Testing scenarios. It currently does not support UWP & data collector scenarios (code coverage & fakes). It is placed @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform"`.
|
||||
|
||||
Test Platform (TPv2) is packaged as a `vsix` with VS 2017 RC releases and lights up the .NET core and Live Unit Testing scenarios. It currently does not support UWP & data collector scenarios (code coverage & fakes). It is placed @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform"`.
|
||||
|
||||
Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow"`.
|
||||
|
||||
|
|
@ -215,4 +220,5 @@ To use TPv2 for desktop - place `testplatform.config` @ `"%programfiles(x86)%\Mi
|
|||
</appSettings>
|
||||
</configuration>
|
||||
````
|
||||
If `feature.net35` and `feature.net40` are set to `true`, VS Test Explorer will use TPv2 for desktop flow for tests targetting net35 and net40 frameworks respectively. If `feature.datacollector` is set to `true`, VS Test Explorer will use TPv2 when data collectors are enabled.
|
||||
|
||||
If `feature.net35` and `feature.net40` are set to `true`, VS Test Explorer will use TPv2 for desktop flow for tests targeting net35 and net40 frameworks respectively. If `feature.datacollector` is set to `true`, VS Test Explorer will use TPv2 when data collectors are enabled.
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ This document outlines troubleshooting and diagnosis instructions for test platf
|
|||
The console test runner (`vstest.console`) provides a `diag` switch to collect
|
||||
traces to a log file. Invoke the runner using below command line:
|
||||
|
||||
```
|
||||
```shell
|
||||
> vstest.console testApp.dll --diag:log.txt
|
||||
```
|
||||
|
||||
|
|
@ -19,7 +19,7 @@ it doesn't exist.
|
|||
|
||||
Trace level can be changed using below command line:
|
||||
|
||||
```
|
||||
```shell
|
||||
> vstest.console testApp.dll --diag:log.txt;tracelevel=verbose
|
||||
```
|
||||
|
||||
|
|
@ -29,13 +29,14 @@ Allowed values for tracelevel are: off, error, warning, info and verbose.
|
|||
|
||||
The `--diag` option is supported on the `dotnet test` command as well. This will also produce same
|
||||
set of log files: `log.txt` and `log.*.txt`.
|
||||
```
|
||||
|
||||
```shell
|
||||
> dotnet test --diag:log.txt
|
||||
```
|
||||
|
||||
To get traces for VSTest build task, enable the following environment variable:
|
||||
|
||||
```
|
||||
```shell
|
||||
> set VSTEST_BUILD_TRACE=1
|
||||
or
|
||||
> $env:VSTEST_BUILD_TRACE=1 # powershell
|
||||
|
|
@ -50,7 +51,7 @@ a `vstest.console.exe.config` configuration file in the same directory as
|
|||
|
||||
This technique is useful to capture logs for IDE/Editor scenarios.
|
||||
|
||||
```
|
||||
```shell
|
||||
> where vstest.console
|
||||
d:\tmp\vstest.console.exe
|
||||
|
||||
|
|
@ -59,7 +60,7 @@ d:\tmp\vstest.console.exe
|
|||
|
||||
Add the following content to the config file (`vstest.console.exe.config` or `testhost.exe.config`).
|
||||
|
||||
```
|
||||
```shell
|
||||
<configuration>
|
||||
<system.diagnostics>
|
||||
<sources>
|
||||
|
|
@ -97,10 +98,12 @@ Add the following content to the config file (`vstest.console.exe.config` or `te
|
|||
```
|
||||
|
||||
## Collect trace in VSTS CI (VSTest Task)
|
||||
|
||||
Add a variable to the build definition named `System.Debug` with value `True`. This can
|
||||
be done while a new build is queued.
|
||||
|
||||
This variable will ensure:
|
||||
|
||||
* Exact command line of `vstest.console.exe` is reported. It can be used to isolate if there
|
||||
are errors in translating user inputs in the task to command line options of vstest.console.exe.
|
||||
* The task invokes `vstest.console.exe` with `/diag` option. The log files can be uploaded
|
||||
|
|
@ -112,7 +115,7 @@ The runner and test host processes support waiting for debugger attach. You can
|
|||
enable the following environment variable to keep `vstest.console.exe` waiting for
|
||||
debugger attach:
|
||||
|
||||
```
|
||||
```shell
|
||||
> set VSTEST_RUNNER_DEBUG=1
|
||||
> vstest.console testApp.dll
|
||||
Waiting for debugger attach...
|
||||
|
|
@ -120,10 +123,10 @@ Process Id: 51928, Name: vstest.console
|
|||
```
|
||||
|
||||
For test host process (`testhost.exe`, or `dotnet exec testhost.dll` depending on
|
||||
target framework) use the following environment variable.
|
||||
target framework) use the following environment variable.
|
||||
Execution will halt until debugger is attached.
|
||||
|
||||
```
|
||||
```shell
|
||||
> set VSTEST_HOST_DEBUG=1
|
||||
> vstest.console testApp.dll
|
||||
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
# TestCase filter
|
||||
|
||||
This document will help you to selectively execute tests based on filtering conditions through `--filter` for `dotnet test` and `--testcasefilter` for `vstest.console.exe`.
|
||||
|
||||
### Syntax
|
||||
|
||||
`dotnet test --filter <Expression>` or
|
||||
`vstest.console.exe --testcasefilter:<Expression>`
|
||||
|
||||
|
|
@ -20,6 +22,7 @@ supported by popular unit test frameworks.
|
|||
| Xunit | <ul><li>FullyQualifiedName</li><li>DisplayName</li><li>Traits</li></ul> |
|
||||
|
||||
Allowed **operators**:
|
||||
|
||||
* `=` implies an exact match
|
||||
* `!=` implies an exact not match
|
||||
* `~` implies a contains lookup
|
||||
|
|
@ -27,7 +30,7 @@ Allowed **operators**:
|
|||
|
||||
**Value** is a string. All the lookups are case insensitive.
|
||||
|
||||
**Escape Sequences** must be used to represent characters in the value that have special meanings in the filter, i.e. filter operators.
|
||||
**Escape Sequences** must be used to represent characters in the value that have special meanings in the filter, i.e. filter operators.
|
||||
|
||||
| Escape Sequence | Represents |
|
||||
| -------------- | -------------------- |
|
||||
|
|
@ -44,13 +47,16 @@ A helper method `Microsoft.VisualStudio.TestPlatform.ObjectModel.Utilities.Filte
|
|||
is also available by referencing the `Microsoft.VisualStudio.TestPlatform.ObjectModel` NuGet package, which can be used to escape strings programatically.
|
||||
|
||||
Expressions can be joined with boolean operators. The following boolean operators are supported:
|
||||
|
||||
* `|` implies a boolean `OR`
|
||||
* `&` implies a boolean `AND`
|
||||
|
||||
## Examples
|
||||
The following examples use `dotnet test`, if you're using `vstest.console.exe` replace `--filter ` with `--testcasefilter:`.
|
||||
|
||||
The following examples use `dotnet test`, if you're using `vstest.console.exe` replace `--filter` with `--testcasefilter:`.
|
||||
|
||||
### MSTest
|
||||
|
||||
```CSharp
|
||||
namespace MSTestNamespace
|
||||
{
|
||||
|
|
@ -88,8 +94,8 @@ namespace MSTestNamespace
|
|||
|
||||
| Expression | What it does? |
|
||||
| ---------- | ------------- |
|
||||
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1\|TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName __or__ TestCategory is CategoryA. |
|
||||
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName __and__ TestCategory is CategoryA. |
|
||||
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1\|TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName **or** TestCategory is CategoryA. |
|
||||
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName **and** TestCategory is CategoryA. |
|
||||
| `dotnet test --filter "(FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `UnitTestClass1` and TestCategory is CategoryA or Priority is 1. |
|
||||
|
||||
### xUnit
|
||||
|
|
@ -101,6 +107,7 @@ namespace MSTestNamespace
|
|||
| `dotnet test --filter DisplayName~TestClass1` | Runs tests whose display name contains `TestClass1`. |
|
||||
|
||||
#### Using traits for filter
|
||||
|
||||
```CSharp
|
||||
namespace XUnitNamespace
|
||||
{
|
||||
|
|
@ -123,6 +130,7 @@ namespace XUnitNamespace
|
|||
}
|
||||
|
||||
```
|
||||
|
||||
In above code we defined traits with keys `Category` and `Priority` which can be used for filtering.
|
||||
|
||||
| Expression | What it does? |
|
||||
|
|
@ -134,6 +142,6 @@ In above code we defined traits with keys `Category` and `Priority` which can be
|
|||
|
||||
| Expression | What it does? |
|
||||
| ---------- | ------------- |
|
||||
| `dotnet test --filter "FullyQualifiedName~TestClass1\|Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName __or__ Category is Nightly. |
|
||||
| `dotnet test --filter "FullyQualifiedName~TestClass1&Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName __and__ Category is Nightly. |
|
||||
| `dotnet test --filter "FullyQualifiedName~TestClass1\|Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName **or** Category is Nightly. |
|
||||
| `dotnet test --filter "FullyQualifiedName~TestClass1&Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName **and** Category is Nightly. |
|
||||
| `dotnet test --filter "(FullyQualifiedName~TestClass1&Category=Nightly)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `TestClass1` and Category is CategoryA or Priority is 1. |
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
# Reporting test results
|
||||
|
||||
Test discovery, execution results in a test run can be controlled with test
|
||||
loggers. This document will cover details on installation, usage and authoring
|
||||
of test loggers.
|
||||
|
||||
|
||||
## Test loggers
|
||||
|
||||
A test logger is a test platform extension to control reporting of test results.
|
||||
It can perform tasks when a test run message, individual test
|
||||
results or the test run completion events are reported by the test platform.
|
||||
|
|
@ -17,6 +18,7 @@ services. Default inputs to a test logger can be provided in the command line.
|
|||
if you're interested in the architecture of a test logger.
|
||||
|
||||
### Available test loggers
|
||||
|
||||
| Scenario | Nuget Package | Source Repository |
|
||||
| -------- | ------------- | ----------------- |
|
||||
| Local, CI, CD | Inbuilt | [Trx Logger][] |
|
||||
|
|
@ -49,6 +51,7 @@ if you're interested in the architecture of a test logger.
|
|||
Want to add your logger? Please send a PR with changes in this doc.
|
||||
|
||||
## Acquisition
|
||||
|
||||
A test logger should be made available as a NuGet package (preferred), or as
|
||||
a zip file (for e.g. loggers for C++ etc.).
|
||||
|
||||
|
|
@ -59,16 +62,17 @@ them in the same directory as the test assemblies.
|
|||
If the test logger is made available as a zip file, it should be extracted
|
||||
to one of the following locations:
|
||||
|
||||
1. the `Extensions` folder along side `vstest.console.exe`. E.g. in case of
|
||||
1. the `Extensions` folder along side `vstest.console.exe`. E.g. in case of
|
||||
dotnet-cli, the path could be `/sdk/<version>/Extensions` directory.
|
||||
2. any well known location on the filesystem
|
||||
|
||||
|
||||
> Version Note: new in 15.1
|
||||
In case of #2, user can specify the full path to the location using `/TestAdapterPath:<path>`
|
||||
command line switch. Test platform will locate extensions from the provided
|
||||
directory.
|
||||
|
||||
## Naming
|
||||
|
||||
Test platform will look for assemblies named `*.testlogger.dll` when it's trying
|
||||
to load test loggers.
|
||||
|
||||
|
|
@ -76,23 +80,31 @@ to load test loggers.
|
|||
> For 15.0 version, the test loggers are also discovered from *.testadapter.dll
|
||||
|
||||
## Create a test logger
|
||||
|
||||
Go through the following steps to create your own logger
|
||||
|
||||
1) Add a nuget reference of package `Microsoft.TestPlatform.ObjectModel`.
|
||||
2) Implement ITestLoggerWithParameters (or ITestLogger, if your logger is not expecting any parameter). [Logger Example](https://github.com/spekt/xunit.testlogger/blob/master/src/Xunit.Xml.TestLogger/XunitXmlTestLogger.cs#L19)
|
||||
3) Name your logger assemlby `*.testlogger.dll`. [Detailed](https://github.com/Microsoft/vstest-docs/blob/main/docs/report.md#naming)
|
||||
|
||||
## Enable a test logger
|
||||
|
||||
A test logger must be explicitly enabled using the command line. E.g.
|
||||
```
|
||||
|
||||
```shell
|
||||
vstest.console test_project.dll /logger:mylogger
|
||||
```
|
||||
|
||||
Where `mylogger` is the LoggerUri or FriendlyName of the logger.
|
||||
|
||||
## Configure reporting
|
||||
|
||||
Additional arguments to a logger can also be passed in the command line. E.g.
|
||||
```
|
||||
|
||||
```shell
|
||||
vstest.console test_project.dll /logger:mylogger;Setting=Value
|
||||
```
|
||||
|
||||
Where `mylogger` is the LoggerUri or FriendlyName of the logger.
|
||||
`Setting` is the name of the additional argument and `Value`is its value.
|
||||
|
||||
|
|
@ -101,11 +113,12 @@ It is upto the logger implementation to support additional arguments.
|
|||
## Syntax of default loggers
|
||||
|
||||
### 1) Console logger
|
||||
|
||||
Console logger is the default logger and it is used to output the test results into console window.
|
||||
|
||||
#### Syntax
|
||||
|
||||
```
|
||||
```shell
|
||||
For dotnet test or dotnet vstest :
|
||||
--logger:console[;verbosity=<Defaults to "minimal">]
|
||||
|
||||
|
|
@ -116,7 +129,8 @@ Argument "verbosity" define the verbosity level of console logger. Allowed value
|
|||
```
|
||||
|
||||
#### Example
|
||||
```
|
||||
|
||||
```shell
|
||||
vstest.console.exe Tests.dll /logger:"console;verbosity=normal"
|
||||
|
||||
If you are using "dotnet test", then use the following command
|
||||
|
|
@ -129,10 +143,12 @@ dotnet test Tests.csproj -v normal
|
|||
```
|
||||
|
||||
### 2) Trx logger
|
||||
|
||||
Trx logger is used to log test results into a Visual Studio Test Results File (TRX).
|
||||
|
||||
#### Syntax
|
||||
```
|
||||
|
||||
```shell
|
||||
/logger:trx [;LogFileName=<Defaults to unique file name>]
|
||||
|
||||
Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.
|
||||
|
|
@ -141,7 +157,8 @@ Where "LogFileName" can be absolute or relative path. If path is relative, it wi
|
|||
#### Examples
|
||||
|
||||
Suppose the current working directory is "c:\tempDirecory".
|
||||
```
|
||||
|
||||
```shell
|
||||
1) vstest.console.exe Tests.dll /logger:trx
|
||||
trx file will get generated in location "c:\tempDirecory\TestResults"
|
||||
|
||||
|
|
@ -153,10 +170,12 @@ trx file will be "c:\temp\logFile.txt"
|
|||
```
|
||||
|
||||
### 3) Html logger
|
||||
|
||||
Html logger is used to log test results into a html file.
|
||||
|
||||
#### Syntax
|
||||
```
|
||||
|
||||
```shell
|
||||
/logger:html [;LogFileName=<Defaults to unique file name>]
|
||||
|
||||
Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.
|
||||
|
|
@ -166,7 +185,8 @@ Where "LogFileName" can be absolute or relative path. If path is relative, it wi
|
|||
#### Examples
|
||||
|
||||
Suppose the current working directory is "c:\tempDirecory".
|
||||
```
|
||||
|
||||
```shell
|
||||
1) vstest.console.exe Tests.dll /logger:html
|
||||
Html file will get generated in location "c:\tempDirecory\TestResults"
|
||||
|
||||
|
|
@ -177,6 +197,6 @@ Html file will be "c:\tempDirecory\TestResults\relativeDir\logFile.html"
|
|||
Html file will be "c:\temp\logFile.html"
|
||||
```
|
||||
|
||||
|
||||
## Related links
|
||||
|
||||
TODO: link to author a test logger
|
||||
|
|
|
|||
|
|
@ -1,17 +1,21 @@
|
|||
# Testplatform Migration Known issues
|
||||
|
||||
Here are the current known issues you may face when running tests, along with available workarounds.
|
||||
|
||||
# Known Issues
|
||||
@# Known Issues
|
||||
|
||||
- [Change in Thread.CurrentPrincipal value.](CurrentPrincipal)
|
||||
- [Change in test execution processes name.](processname)
|
||||
|
||||
### <b>Change in Thread.CurrentPrincipal value.</b><a name="CurrentPrincipal"></a>
|
||||
|
||||
- <b>Issue:</b> <br/>
|
||||
Tests that depend on `Thread.CurrentPrincipal` may fail. This is due to change in inter process commincation in Testplatform.
|
||||
- <b>Workaround:</b> <br/>
|
||||
Use an alternative like `System.Security.Principal.WindowsIdentity.GetCurrent()`
|
||||
|
||||
### <b>Change in test execution processes name.</b><a name="processname"></a>
|
||||
|
||||
- <b>Issue:</b> <br/>
|
||||
Tests that depend on the name of the currnet running process may fail.
|
||||
- <b>Workaround:</b> <br/>
|
||||
|
|
|
|||
|
|
@ -4,12 +4,15 @@ The goal of this document is to help the test platform users to collect useful i
|
|||
|
||||
## Dotnet CLI
|
||||
|
||||
#### Collect logs and crash dump
|
||||
```bash
|
||||
### Collect logs and crash dump
|
||||
|
||||
```shell
|
||||
dotnet test --diag:log.txt --blame-crash --blame-crash-dump-type full
|
||||
```
|
||||
|
||||
At the end of the execution you'll find the list of artifacts generated with the link to the file dump:
|
||||
```bash
|
||||
|
||||
```shell
|
||||
Starting test execution, please wait...
|
||||
Logging Vstest Diagnostics in file: C:\git\issue\bug\log.txt
|
||||
A total of 1 test files matched the specified pattern.
|
||||
|
|
@ -20,12 +23,15 @@ Attachments:
|
|||
C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\Sequence_bfcc4d8558654413a3fb2f5164695bf6.xml
|
||||
C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\dotnet.exe_11876_1660721586_crashdump.dmp
|
||||
```
|
||||
|
||||
You'll find 3 files for logs(runner, datacollector, host), datacollector one can be missing.
|
||||
```bash
|
||||
|
||||
```shell
|
||||
-a---- 8/17/2022 9:33 AM 30001 log.datacollector.22-08-17_09-32-54_50516_1.txt
|
||||
-a---- 8/17/2022 9:33 AM 37222 log.host.22-08-17_09-32-55_24965_7.txt
|
||||
-a---- 8/17/2022 9:33 AM 200345 log.txt
|
||||
```
|
||||
|
||||
## Azure DevOps
|
||||
|
||||
### @VSTest2 task
|
||||
|
|
@ -245,11 +251,12 @@ jobs:
|
|||
|
||||
Note: these logs could contain sensitive information (paths, project name...). Make sure to clean them or use the Visual Studio `Send Feedback` button. Don't put anything you want to keep private in the title or content of the initial report, which is public. Instead, say that you'll send details privately in a separate comment. Once the problem report is created, it's now possible to specify who can see your replies and attachments.
|
||||
|
||||
## Use procdump https://docs.microsoft.com/it-it/sysinternals/downloads/procdump on Windows
|
||||
## Use procdump <https://docs.microsoft.com/it-it/sysinternals/downloads/procdump> on Windows
|
||||
|
||||
Sometimes it's not possible to take the dump using test platform tool because the crash happen before we're able to attach to the process to take the dump self. In that situation we need a way to register for dump at process startup level.
|
||||
To achieve it we can use procdump that will install machine wide Just-in-time (AeDebug) debugger.
|
||||
```bash
|
||||
|
||||
```shell
|
||||
PS C:\tools\Procdump> .\procdump.exe -i C:\tools\Procdump\dumps
|
||||
|
||||
ProcDump v10.11 - Sysinternals process dump utility
|
||||
|
|
@ -268,7 +275,9 @@ Set to:
|
|||
|
||||
ProcDump is now set as the Just-in-time (AeDebug) debugger.
|
||||
```
|
||||
|
||||
After you can run your application and in case of crash a dump will be automatically taken inside the `C:\tools\Procdump\dumps` directory.
|
||||
|
||||
```
|
||||
PS C:\tools\Procdump> ls C:\tools\Procdump\dumps
|
||||
|
||||
|
|
@ -280,8 +289,11 @@ Mode LastWriteTime Length Name
|
|||
---- ------------- ------ ----
|
||||
-a---- 8/17/2022 9:42 AM 6161605 dotnet.exe_220817_094234.dmp
|
||||
```
|
||||
You can unistall the automatic generation running at the end of the collection phase
|
||||
```bash
|
||||
|
||||
You can unistall the automatic generation running at the end of the collection phase
|
||||
|
||||
```shell
|
||||
.\procdump.exe -u
|
||||
```
|
||||
|
||||
Keep in mind that this mode will collect machine wide crash so every process in the machine that will crash will collect a dump in the folder.
|
||||
|
|
|
|||
|
|
@ -1,53 +1,69 @@
|
|||
# Issue tracking
|
||||
|
||||
The vstest project tracks issues and feature requests using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md) for the vstest repository.
|
||||
|
||||
## Submitting an Issue
|
||||
|
||||
First, please do a search in [issues](https://github.com/Microsoft/vstest/issues) to see if the issue or feature request has already been filed. Use this [query](https://github.com/Microsoft/vstest/issues?q=is%3Aopen+is%3Aissue+sort%3Areactions-%2B1-desc) to search for the most popular feature requests.
|
||||
If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment.
|
||||
|
||||
- 👍 - upvote
|
||||
- 👎 - downvote
|
||||
|
||||
If you cannot find an existing issue that describes your issue, submit it using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md). Remember to follow the instruction mentioned therein carefully.
|
||||
|
||||
## Issue triage
|
||||
|
||||
Please follow the guidelines in the issue template when filing an issue or a pull request.
|
||||
New issues or pull requests submitted by the community are triaged by a team member using the inbox query below.
|
||||
|
||||
## Inbox query
|
||||
|
||||
The [inbox query](https://github.com/Microsoft/vstest/issues?utf8=%E2%9C%93&q=is%3Aopen%20no%3Aassignee%20-label%3Abacklog%20-label%3Aenhancement) will return the following:
|
||||
|
||||
- Open issues or pull requests that are not enhancements and that have no owner assigned.
|
||||
|
||||
## Initial triage - Issue tagging
|
||||
|
||||
Issues will then be tagged as follows:
|
||||
|
||||
- Is the issue ***invalid***? It will be closed, and the reason will be provided.
|
||||
- Is the issue a ***general question***, like *are data collector events synchronous*? It will be tagged as a ***question***.
|
||||
- Is the issue a feature request or an enhancement. It will be tagged as an ***enhancement***.
|
||||
- Else, the issue will be tagged as a ***bug***.
|
||||
|
||||
## Secondary triage – assignment and follow through
|
||||
|
||||
As and when an issue get assigned to a team member, the following secondary triage will happen
|
||||
|
||||
- When a team member picks up an issue, they will first assign it to themselves.
|
||||
- Ensure that the issue has an appropriate tag (***question***, ***enhancement***, ***bug***).
|
||||
- If an issue needs a repro, tag it with ***need-repro*** and ask for a repro in a comment.
|
||||
|
||||
## Ongoing issue management
|
||||
|
||||
- Issues tagged ***need-repro*** info will be closed if no additional information is provided for 7 days.
|
||||
|
||||
Team members will strive to resolve every bug within a stipulated period of time – we would like that to be a period of 14 days from the date the issue was filed.
|
||||
|
||||
# Community contributions - up-for-grabs
|
||||
|
||||
We strongly encourage the community to contribute enhancements and bug-fixes. Therefore, team members will look to add as much information in the issues as required so that you can make an effective contribution. Such issues will be tagged ***up-for-grabs***.
|
||||
|
||||
# Planning
|
||||
|
||||
## Triage
|
||||
|
||||
Bugs and enhancements will be assigned a milestone, and within a milestone they will be assigned a priority. The priority dictates the order in which issues should be addressed. A important bug (something that we think is critical for the milestone) is to be addressed before the other bugs.
|
||||
|
||||
To find out when a bug fix will be available in an update, then please check the milestone that is assigned to the issue.
|
||||
Please see Issue Tracking for a description of the different workflows we are using.
|
||||
|
||||
## Milestone planning
|
||||
|
||||
We typically plan for a quarter and establish a set of themes we want to work towards, and prioritize enhancements and bug-fixes accordingly.
|
||||
During the planning process we prioritize items as follows:
|
||||
|
||||
- Important ***bugs*** - crashes, regressions, and issues that do not have a reasonable workaround.
|
||||
- ***Enhancements*** that have many reactions.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,19 +5,25 @@ We aim to continuously deliver improvements that will ship with Visual Studio an
|
|||
Over the past several quarters, we have made many enhancements - from introducing support for Mono, to refactoring the platform to make it ready to support device testing, to performance improvements, to enabling robust C++ support, to improved documentation, and more. For a complete list see here: [Release Notes](https://github.com/Microsoft/vstest-docs/blob/main/docs/releases.md)
|
||||
|
||||
## Roadmap
|
||||
|
||||
We typically plan for a quarter, and establish a set of themes we want to work towards. Here are the themes we will work on this quarter.
|
||||
|
||||
### Reach: Enable leveraging your vstest experience across all supported platforms
|
||||
|
||||
Over the course of the next phase of execution we will make TPV2 the "default" for all scenarios across Visual Studio and Visual Studio Team Services (VSTS) – i.e. extending it to .NET Framework, UWP, and the VSTest task in VSTS. We will ship a standalone package that can be potentially used in other CI systems even. This is a big switch. We will strive to maintain backwards compat, and publish migration guides for the few features that require to be migrated, and help you in the migration.
|
||||
|
||||
### Performance: At scale
|
||||
|
||||
Performance has been an area where we have received feedback, and made strong progress as well. It will continue to remain a focus. We will look to make improvements across the pipeline from the Test Explorer to the framework adapters, to enhance the overall end to end performance.
|
||||
|
||||
### UWP, Win10 IoT Core Support
|
||||
|
||||
UWP is the application platform for Windows 10, to reach all Windows 10 devices – PC, tablet, phone, Xbox, HoloLens, Surface Hub and more. The vstest engine is architected so that it can be extended to support new application platforms. Such extensions will come from teams who understand their platforms the best, and integrated with vstest. To drive home this point, vstest will be extended to support testing UWP applications. In particular we will light up support for Win10 IoT Core.
|
||||
|
||||
### Code Coverage for .NET Core
|
||||
|
||||
This has been a clear ask from the community, and we are working towards enabling this support. The code coverage infrastructure consumes information from PDB files. Specifically with regard to .NET Core, it now needs to understand the new portable PDB format. We are working cross-team to introduce this support in order to light up code coverage support for .NET Core.
|
||||
|
||||
## Summary
|
||||
|
||||
These are examples of the work we will be focusing on this quarter. We will provide details through individual issues. Follow along, and let us know what you think. We look forward to working with you!
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче