This commit is contained in:
Amaury Levé 2022-12-05 15:16:53 +01:00
Родитель 7ebe3619dc
Коммит 54093ba98a
12 изменённых файлов: 255 добавлений и 121 удалений

Просмотреть файл

@ -1,8 +1,10 @@
### Visual Studio Test Platform Documentation ### Visual Studio Test Platform Documentation
You've found the GitHub repository which contains the content for the Visual Studio Test Platform documentation. You've found the GitHub repository which contains the content for the Visual Studio Test Platform documentation.
If you are looking for the Visual Studio Test Platform product GitHub repository, you can find it [here](https://github.com/Microsoft/vstest). If you are looking for the Visual Studio Test Platform product GitHub repository, you can find it [here](https://github.com/Microsoft/vstest).
### Documentation ### Documentation
- [Test Platform Architecture](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0001-Test-Platform-Architecture.md) - [Test Platform Architecture](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0001-Test-Platform-Architecture.md)
- [Test Discovery Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0002-Test-Discovery-Protocol.md) - [Test Discovery Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0002-Test-Discovery-Protocol.md)
- [Test Execution Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0003-Test-Execution-Protocol.md) - [Test Execution Protocol](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0003-Test-Execution-Protocol.md)
@ -16,23 +18,28 @@ If you are looking for the Visual Studio Test Platform product GitHub repository
- [Troubleshooting guide](https://github.com/Microsoft/vstest-docs/blob/main/docs/troubleshooting.md) - [Troubleshooting guide](https://github.com/Microsoft/vstest-docs/blob/main/docs/troubleshooting.md)
### Contributing ### Contributing
There are many ways to contribute to VSTest There are many ways to contribute to VSTest
- [Submit issues](https://github.com/Microsoft/vstest-docs/issues) and help verify fixes as they are checked in. - [Submit issues](https://github.com/Microsoft/vstest-docs/issues) and help verify fixes as they are checked in.
- Review the [documentation changes](https://github.com/Microsoft/vstest-docs/pulls). - Review the [documentation changes](https://github.com/Microsoft/vstest-docs/pulls).
- Contribute new topics/information, or make changes to existing documentation. - Contribute new topics/information, or make changes to existing documentation.
### Editing docs ### Editing docs
We use [docfx](https://github.com/dotnet/docfx/releases) for building this documentation. A short primer on editing this repo is below. We use [docfx](https://github.com/dotnet/docfx/releases) for building this documentation. A short primer on editing this repo is below.
First, [download](https://github.com/dotnet/docfx/releases) latest release of docfx (`docfx.zip` package) and extract it locally. We will use d:\tmp\docfx as destination for these steps. First, [download](https://github.com/dotnet/docfx/releases) latest release of docfx (`docfx.zip` package) and extract it locally. We will use d:\tmp\docfx as destination for these steps.
Open a command prompt, git clone this repo. Use the following commands to build and run a local server. Open a command prompt, git clone this repo. Use the following commands to build and run a local server.
``` ```
> cd d:\src\vstest-docs > cd d:\src\vstest-docs
> d:\tmp\docfx\docfx.exe build > d:\tmp\docfx\docfx.exe build
> d:\tmp\docfx\docfx.exe serve > d:\tmp\docfx\docfx.exe serve
``` ```
Open http://localhost:8080/_site in a browser to see the rendering of the documentation. Open <http://localhost:8080/_site> in a browser to see the rendering of the documentation.
### Microsoft Open Source Code of Conduct ### Microsoft Open Source Code of Conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

Просмотреть файл

@ -1,24 +1,25 @@
# Passing runsettings arguments through commandline # Passing runsettings arguments through commandline
You are here because you are looking for syntax and details to pass runsettings configurations to either `vstest.console.exe` or `dotnet test` through commandline. You are here because you are looking for syntax and details to pass runsettings configurations to either `vstest.console.exe` or `dotnet test` through commandline.
`RunSettings arguments` are used to add/update specific `runsettings configurations`. The updated `runsettings configurations` will be available to `TestPlatform` and `test extensions` (E.g. test adapter, etc.) through runsettings. `RunSettings arguments` are used to add/update specific `runsettings configurations`. The updated `runsettings configurations` will be available to `TestPlatform` and `test extensions` (E.g. test adapter, etc.) through runsettings.
## Syntax ## Syntax
* `RunSettings arguments` are specified as name-value pair of the form `[name]=[value]` after `-- `. Note the space after --. * `RunSettings arguments` are specified as name-value pair of the form `[name]=[value]` after `--`. Note the space after --.
* Use a space to separate multiple `[name]=[value]`. * Use a space to separate multiple `[name]=[value]`.
* All the arguments after `--` will be treated as `RunSettings arguments`, means `RunSettings arguments` should be at the end of the command line. * All the arguments after `--` will be treated as `RunSettings arguments`, means `RunSettings arguments` should be at the end of the command line.
## Example ## Example
Passing argument `-- MSTest.MapInconclusiveToFailed=True` in (1) below is equivalent to passing argument Passing argument `-- MSTest.MapInconclusiveToFailed=True` in (1) below is equivalent to passing argument
`--settings additionalargs.runsettings` in (2) below. `--settings additionalargs.runsettings` in (2) below.
``` ```shell
1) dotnet test -- MSTest.MapInconclusiveToFailed=True MSTest.DeploymentEnabled=False 1) dotnet test -- MSTest.MapInconclusiveToFailed=True MSTest.DeploymentEnabled=False
``` ```
``` ```shell
2) dotnet test --settings additionalargs.runsettings 2) dotnet test --settings additionalargs.runsettings
``` ```
@ -35,19 +36,17 @@ where `additionalargs.runsettings` is:
</RunSettings> </RunSettings>
``` ```
The syntax in (1) is another way of passing runsettings configuration and you need not author a runsetting file while using `Runsettings arguments`. More details about runsettings can be found [here](https://msdn.microsoft.com/library/jj635153.aspx).
The syntax in (1) is another way of passing runsettings configuration and you need not author a runsetting file while using `Runsettings arguments`. More details about runsettings can be found [here](https://msdn.microsoft.com/en-us/library/jj635153.aspx).
`Runsettings arguments` takes precedence over `runsettings`. `Runsettings arguments` takes precedence over `runsettings`.
For example, in below command the final value for `MapInconclusiveToFailed` will be `False` and value for `DeploymentEnabled` will be unchanged, that is `False`. For example, in below command the final value for `MapInconclusiveToFailed` will be `False` and value for `DeploymentEnabled` will be unchanged, that is `False`.
``` ```shell
dotnet test --settings additionalargs.runsettings -- MSTest.MapInconclusiveToFailed=False dotnet test --settings additionalargs.runsettings -- MSTest.MapInconclusiveToFailed=False
``` ```
You can also set `TestRunParameters` using command line option: You can also set `TestRunParameters` using command line option:
```cmd ```cmd
# cmd # cmd
@ -60,4 +59,4 @@ dotnet test --% -- TestRunParameters.Parameter(name=\"myParam\", value=\"value\
dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\) dotnet test -- TestRunParameters.Parameter\(name=\"myParam\",\ value=\"value\"\)
``` ```
In this example, `\"myParam\"` corresponds to the name of you parameter and `\"value\"` - the value of your parameter. Note, that `\` are escaping characters and they should stay as shown above. In this example, `\"myParam\"` corresponds to the name of you parameter and `\"value\"` - the value of your parameter. Note, that `\` are escaping characters and they should stay as shown above.

Просмотреть файл

@ -1,9 +1,11 @@
# Monitor and analyze test run # Monitor and analyze test run
This document will walk you through enabling data collection for a test run covering: This document will walk you through enabling data collection for a test run covering:
1. A brief overview of DataCollectors. 1. A brief overview of DataCollectors.
2. Configuring DataCollectors in TPv2. 1. Configuring DataCollectors in TPv2.
3. Key differences for using DataCollectors in TPv2 v/s TPv1. 1. Key differences for using DataCollectors in TPv2 v/s TPv1.
3. Instructions for using [code coverage][coverage]. 1. Instructions for using [code coverage][coverage].
> **Version note:** > **Version note:**
> >
@ -13,7 +15,9 @@ This document will walk you through enabling data collection for a test run cove
[coverage]: #coverage [coverage]: #coverage
## DataCollector ## DataCollector
A DataCollector is a test platform extension to monitor test run. It can be extended to perform tasks on specific test execution events. Currently, four events are exposed to DataCollector: A DataCollector is a test platform extension to monitor test run. It can be extended to perform tasks on specific test execution events. Currently, four events are exposed to DataCollector:
1. Session Start event. 1. Session Start event.
2. Test Case Start event 2. Test Case Start event
3. Test Case End event. 3. Test Case End event.
@ -25,10 +29,13 @@ Please refer [here](https://github.com/Microsoft/vstest-docs/blob/main/docs/exte
if you're interested in the architecture of data collection. if you're interested in the architecture of data collection.
## Configure DataCollectors ## Configure DataCollectors
DataCollectors can be configured for monitoring test execution through runSettings, testSettings or vstest.console args. DataCollectors can be configured for monitoring test execution through runSettings, testSettings or vstest.console args.
### Using RunSettings<a name="Using-RunSettings"></a> ### Using RunSettings<a name="Using-RunSettings"></a>
Below is the sample runsettings for a custom DataCollector Below is the sample runsettings for a custom DataCollector
```xml ```xml
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<RunSettings> <RunSettings>
@ -49,21 +56,28 @@ Below is the sample runsettings for a custom DataCollector
</DataCollectionRunSettings> </DataCollectionRunSettings>
</RunSettings> </RunSettings>
``` ```
Below is the sample command for enabling DataCollectors using runsettings Below is the sample command for enabling DataCollectors using runsettings
```
```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.runsettings > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.runsettings
``` ```
### Using vstest.console args<a name="Using-vstest.console-args"></a> ### Using vstest.console args<a name="Using-vstest.console-args"></a>
DataCollectors can be configured and used through first class command line arguments `/collect` and `/testadapterpath`. Hence, for common DataCollection scenarios, separate runsettings file may not be required. DataCollectors can be configured and used through first class command line arguments `/collect` and `/testadapterpath`. Hence, for common DataCollection scenarios, separate runsettings file may not be required.
Below is the sample command to configure and use DataCollectors through vstest.console command line. Below is the sample command to configure and use DataCollectors through vstest.console command line.
```
```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"Code Coverage" > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"Code Coverage"
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"MyDataCollector" /testadapterpath:<Path to MyDataCollector assembly> > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:"MyDataCollector" /testadapterpath:<Path to MyDataCollector assembly>
``` ```
Please note that `testadapterpath` is not required for DataCollectors shipped along with TPv2. Please note that `testadapterpath` is not required for DataCollectors shipped along with TPv2.
### Using TestSettings ### Using TestSettings
While the recommended way is to use [runsettings](#Using-RunSettings) or [vstest.console args](#Using-vstest.console-args), there are few DataCollectors which only worked with testsettings. While the recommended way is to use [runsettings](#Using-RunSettings) or [vstest.console args](#Using-vstest.console-args), there are few DataCollectors which only worked with testsettings.
E.g.: `System Information` DataCollector. Below is the sample testsettings for using `System Information` DataCollector. E.g.: `System Information` DataCollector. Below is the sample testsettings for using `System Information` DataCollector.
@ -83,12 +97,16 @@ E.g.: `System Information` DataCollector. Below is the sample testsettings for u
<Properties /> <Properties />
</TestSettings> </TestSettings>
``` ```
Below is the sample command for enabling DataCollectors using testsettings Below is the sample command for enabling DataCollectors using testsettings
```
```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.testsettings > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /settings:datacollection.testsettings
``` ```
### Enable/Disable a DataCollector ### Enable/Disable a DataCollector
All DataCollectors configured in the .runsettings files are loaded automatically and are enabled to participate for run, unless explicitely disabled using boolean valued attribute named `enabled`.
All DataCollectors configured in the .runsettings files are loaded automatically and are enabled to participate for run, unless explicitly disabled using boolean valued attribute named `enabled`.
For example, only `MyDataCollector1` DataCollector will be enabled for a test run with For example, only `MyDataCollector1` DataCollector will be enabled for a test run with
below runsettings: below runsettings:
@ -116,19 +134,22 @@ below runsettings:
A specific DataCollector can be explicitely enabled using the `/collect:<friendly name>` command line switch. A specific DataCollector can be explicitely enabled using the `/collect:<friendly name>` command line switch.
For example, below command line will enable a DataCollector named `MyDataCollector1` For example, below command line will enable a DataCollector named `MyDataCollector1`
(and disable other DataCollectors mentioned in .runsettings): (and disable other DataCollectors mentioned in .runsettings):
```
```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /settings:<Path to runSettings> > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /settings:<Path to runSettings>
``` ```
More than one DataCollectors can also be enabled using `/collect` command line switch More than one DataCollectors can also be enabled using `/collect` command line switch
For example, below command will enable DataCollectors named `MyDataCollector1` and `MyDataCollector2`: For example, below command will enable DataCollectors named `MyDataCollector1` and `MyDataCollector2`:
```
```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /collect:MyDataCollector2 /settings:<Path to runSettings> > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /collect:MyDataCollector1 /collect:MyDataCollector2 /settings:<Path to runSettings>
``` ```
## Key differences for using DataCollectors in TPv2 v/s TPv1.
## Key differences for using DataCollectors in TPv2 v/s TPv1
1. In TPv1, DataCollectors are loaded from `<<VisualStudio Installation Directory>>\Common7\IDE\PrivateAssemblies\DataCollectors`. 1. In TPv1, DataCollectors are loaded from `<<VisualStudio Installation Directory>>\Common7\IDE\PrivateAssemblies\DataCollectors`.
In TPv2, DataCollectors are loaded from `TestAdaptersPaths` specified in runSettings or `/testadapterpath` argument of `vstest.console.exe`. DataCollector assemblies must follow the naming convention *collector.dll. In TPv2, DataCollectors are loaded from `TestAdaptersPaths` specified in runSettings or `/testadapterpath` argument of `vstest.console.exe`. DataCollector assemblies must follow the naming convention *collector.dll.
@ -138,9 +159,12 @@ In TPv2, DataCollectors are loaded from `TestAdaptersPaths` specified in runSett
3. There are breaking changes in latest DataCollector interface. Hence, older DataCollectors need to be rebuilt against latest APIs to work with TPv2. For details, refer [here(todo)](); 3. There are breaking changes in latest DataCollector interface. Hence, older DataCollectors need to be rebuilt against latest APIs to work with TPv2. For details, refer [here(todo)]();
## Working with Code Coverage<a name="coverage"></a> ## Working with Code Coverage<a name="coverage"></a>
> **Requirements:** > **Requirements:**
> Code Coverage requires the machine to have Visual Studio 2017 Enterprise ([15.3.0](https://www.visualstudio.com/vs) or later installed and a Windows operating system. > Code Coverage requires the machine to have Visual Studio 2017 Enterprise ([15.3.0](https://www.visualstudio.com/vs) or later installed and a Windows operating system.
### Setup a project ### Setup a project
Here's a sample project file, please note the xml entity marked as `Required`. Previously, the `Microsoft.VisualStudio.CodeCoverage` was required, but is now shipped with the SDK. Here's a sample project file, please note the xml entity marked as `Required`. Previously, the `Microsoft.VisualStudio.CodeCoverage` was required, but is now shipped with the SDK.
```xml ```xml
@ -162,49 +186,56 @@ Here's a sample project file, please note the xml entity marked as `Required`. P
</Project> </Project>
``` ```
[coveragenuget]: https://www.nuget.org/packages/Microsoft.CodeCoverage/
### Analyze coverage with Visual Studio ### Analyze coverage with Visual Studio
> **Version note:** > **Version note:**
> >
> Try this feature with [Visual Studio 2017 15.3.0](https://www.visualstudio.com/vs) or later. > Try this feature with [Visual Studio 2017 15.3.0](https://www.visualstudio.com/vs) or later.
Use the `Analyze Code Coverage` context menu available in `Test Explorer` tool window to start a coverage run. Use the `Analyze Code Coverage` context menu available in `Test Explorer` tool window to start a coverage run.
After the coverage run is complete, a detailed report will be available in the `Code Coverage Results` tool window. After the coverage run is complete, a detailed report will be available in the `Code Coverage Results` tool window.
Please refer the documentation for additional details: https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested Please refer the documentation for additional details: <https://docs.microsoft.com/en-us/visualstudio/test/using-code-coverage-to-determine-how-much-code-is-being-tested>
### Collect coverage with command line runner ### Collect coverage with command line runner
Use the following command line to collect coverage data for tests: Use the following command line to collect coverage data for tests:
``` ```shell
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" --collect:"Code Coverage" --framework:".NETCoreApp,Version=v1.1" d:\testproject\bin\Debug\netcoreapp1.1\testproject.dll > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" --collect:"Code Coverage" --framework:".NETCoreApp,Version=v1.1" d:\testproject\bin\Debug\netcoreapp1.1\testproject.dll
``` ```
This will generate a `*.coverage` file in the `<Current working directory>\TestResults` directory. This will generate a `*.coverage` file in the `<Current working directory>\TestResults` directory.
### Event Log Data Collector ### Event Log Data Collector
This document introduces Event Log DataCollector. We will start with a brief overview of Event Log DataCollector, use cases where it will be useful followed by steps to enable it. This document introduces Event Log DataCollector. We will start with a brief overview of Event Log DataCollector, use cases where it will be useful followed by steps to enable it.
#### Introduction #### Introduction
Event Log DataCollector is a Windows only DataCollector that is used to get event logs logged into Windows Event Viewer during test execution. Event logs are saved in a file `Event Log.xml` and this file is available as Attachment as part of test result report (trx). Event Log DataCollector is a Windows only DataCollector that is used to get event logs logged into Windows Event Viewer during test execution. Event logs are saved in a file `Event Log.xml` and this file is available as Attachment as part of test result report (trx).
When enabled, Event Log DataCollector generates one `Event Log.xml` file for entire test session. `Event Log.xml` files are also generated corresponding to all test cases as well, to provide a granular view of events logged while executing a test case. When enabled, Event Log DataCollector generates one `Event Log.xml` file for entire test session. `Event Log.xml` files are also generated corresponding to all test cases as well, to provide a granular view of events logged while executing a test case.
More info on Event Viewer [here](https://technet.microsoft.com/en-us/library/cc938674.aspx) More info on Event Viewer [here](https://technet.microsoft.com/en-us/library/cc938674.aspx)
#### Use cases for Event Log DataCollector #### Use cases for Event Log DataCollector
Event Log DataCollector is used to get event logs as Attachment and is particularly useful for remote scenarios where logging into the machine and viewing the Event Viewer is not possible.
Event Log DataCollector is used to get event logs as Attachment and is particularly useful for remote scenarios where logging into the machine and viewing the Event Viewer is not possible.
#### Enabling Event Log DataCollector #### Enabling Event Log DataCollector
There are two ways of enabling Event Log DataCollector for a test run: There are two ways of enabling Event Log DataCollector for a test run:
##### 1. Using vstest.console argument.
##### 1. Using vstest.console argument
Use the following command to enable Event Log DataCollector with default configuration: Use the following command to enable Event Log DataCollector with default configuration:
> "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /testadapterpath:<<Path to test adapter>> /collect:"Event Log" > "%vsinstalldir%\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" test_project.dll /testadapterpath:<<Path to test adapter>> /collect:"Event Log"
##### 2. Using runsettings. ##### 2. Using runsettings
Below runsettings can be used to enable Event Log DataCollector. Below runsettings can be used to enable Event Log DataCollector.
```xml ```xml
@ -225,7 +256,8 @@ Below runsettings can be used to enable Event Log DataCollector.
</DataCollectionRunSettings> </DataCollectionRunSettings>
</RunSettings> </RunSettings>
``` ```
The above runsettings will collect event logs from `System` and `Application` event logs which are logged as `Error` or `Warning` and event source is specified as `CustomEventSource`. `MaxEventLogEntriesToCollect` specifies the upper limit on the events that are logged in `Event Log.xml` file corresponding to test cases. There is no upper limit on number of events logged in `Event Log.xml` file for test session.
The above runsettings will collect event logs from `System` and `Application` event logs which are logged as `Error` or `Warning` and event source is specified as `CustomEventSource`. `MaxEventLogEntriesToCollect` specifies the upper limit on the events that are logged in `Event Log.xml` file corresponding to test cases. There is no upper limit on number of events logged in `Event Log.xml` file for test session.
In default configuration (through vstest.console.exe args or when <Configuration> section is empty in runsettings), `System`, `Application` and `Security` logs with entry types `Error`, `Warning` or `FailureAudit` and with any event source are collected. Default value of `MaxEventLogEntriesToCollect` is 50000. There is no upper limit on number of events logged in `Event Log.xml` file for test session. In default configuration (through vstest.console.exe args or when <Configuration> section is empty in runsettings), `System`, `Application` and `Security` logs with entry types `Error`, `Warning` or `FailureAudit` and with any event source are collected. Default value of `MaxEventLogEntriesToCollect` is 50000. There is no upper limit on number of events logged in `Event Log.xml` file for test session.

Просмотреть файл

@ -1,7 +1,9 @@
# Configure a test run # Configure a test run
This document covers configuration of a test run in the test platform. This document covers configuration of a test run in the test platform.
## Overview ## Overview
There are three different ways to configure various aspects of a test run. There are three different ways to configure various aspects of a test run.
1. **Using command line arguments** 1. **Using command line arguments**
@ -10,13 +12,13 @@ test` command line. For example, `--framework` can specify the runtime framework
version, or `--platform` can specify the architecture of test run (`x86` or version, or `--platform` can specify the architecture of test run (`x86` or
`x64`). `x64`).
2. **Using a runsettings file** 1. **Using a runsettings file**
User can specify a `runsettings` file to configure test run. For example: User can specify a `runsettings` file to configure test run. For example:
* `> vstest.console.exe --settings:test.runsettings test.dll` * `> vstest.console.exe --settings:test.runsettings test.dll`
* `> dotnet test -s test.runsettings` * `> dotnet test -s test.runsettings`
3. **Using command line runsettings parameters** 1. **Using command line runsettings parameters**
Various elements of a `runsettings` file can also specified as command line Various elements of a `runsettings` file can also specified as command line
parameters directly. For example, consider following `runsettings`: parameters directly. For example, consider following `runsettings`:
@ -31,20 +33,21 @@ parameters directly. For example, consider following `runsettings`:
The `DisableAppDomain` settings can be changed in the following way: The `DisableAppDomain` settings can be changed in the following way:
`> vstest.console --settings:test.runsettings -- RunConfiguration.DisableAppDomain=true` `> vstest.console --settings:test.runsettings -- RunConfiguration.DisableAppDomain=true`
Or, `> dotnet test -s test.runsettings -- RunConfiguration.DisableAppDomain=true` if Or, `> dotnet test -s test.runsettings -- RunConfiguration.DisableAppDomain=true` if
you're using dotnet test to execute tests. you're using dotnet test to execute tests.
Order of priority (in case of conflicting settings) is as follows: Order of priority (in case of conflicting settings) is as follows:
1. If a command line switch is available, it takes priority. E.g. `/platform` 1. If a command line switch is available, it takes priority. E.g. `/platform`
wins over `<TargetPlatform>` specified in the `runsettings` file. wins over `<TargetPlatform>` specified in the `runsettings` file.
2. If a command line runsettings is available, it takes priority over the 1. If a command line runsettings is available, it takes priority over the
content of `runsettings` file. E.g. in case of `dotnet test -s content of `runsettings` file. E.g. in case of `dotnet test -s
test.runsettings -- RunConfiguration.TargetPlatform=x86`, platform is set to test.runsettings -- RunConfiguration.TargetPlatform=x86`, platform is set to
`x86` even if `test.runsettings` specifies `x64` as platform. `x86` even if `test.runsettings` specifies `x64` as platform.
3. If a `runsettings` file is provided, it is used for the test run. 1. If a `runsettings` file is provided, it is used for the test run.
## Run settings ## Run settings
A `*.runsettings` file is used to configure various aspects of a test discovery A `*.runsettings` file is used to configure various aspects of a test discovery
and execution with `vstest.console.exe` or the `Test Explorer` in VS. An editor and execution with `vstest.console.exe` or the `Test Explorer` in VS. An editor
using the test platform, can specify a `runsettings` as an `xml` in the `Test using the test platform, can specify a `runsettings` as an `xml` in the `Test
@ -53,28 +56,30 @@ Discovery` or `Test Run` requests (details are in Editor API Specification).
The `runsettings` file is a xml file with following sections: The `runsettings` file is a xml file with following sections:
1. Run Configuration 1. Run Configuration
2. Data Collection 1. Data Collection
3. Runtime Parameters 1. Runtime Parameters
4. Adapter Configuration 1. Adapter Configuration
5. Legacy Settings 1. Legacy Settings
We will cover these sections in detail later in the document. Let's discuss few We will cover these sections in detail later in the document. Let's discuss few
core principles for runsettings. core principles for runsettings.
### Principles ### Principles
1. Any runner (CLI, IDE, Editor) can use run settings to configure the test run 1. Any runner (CLI, IDE, Editor) can use run settings to configure the test run
2. These runners need to take care of merging user provided runsettings with 1. These runners need to take care of merging user provided runsettings with
their own configuration parameters. E.g. an IDE may provide `TargetPlatform` their own configuration parameters. E.g. an IDE may provide `TargetPlatform`
as an UI configuration, at the same time an user can also provide the setting as an UI configuration, at the same time an user can also provide the setting
in a `runsettings` file. It is the IDE's responsibility to disambiguate. in a `runsettings` file. It is the IDE's responsibility to disambiguate.
3. Run settings are always created at the start of a test run. They are 1. Run settings are always created at the start of a test run. They are
immutable for rest of the run. immutable for rest of the run.
4. Every test platform extension should get the _effective_ runsettings for a 1. Every test platform extension should get the _effective_ runsettings for a
test run via the test platform APIs. It may use it to read its own settings test run via the test platform APIs. It may use it to read its own settings
or take decisions (e.g. an adapter needs to disable appdomains if or take decisions (e.g. an adapter needs to disable appdomains if
`DisableAppDomain` setting is provided via run settings). `DisableAppDomain` setting is provided via run settings).
### Sample ### Sample
Given below is a complete `runsettings` file with all available options. Each Given below is a complete `runsettings` file with all available options. Each
option is briefly annotated, details are available in a later section of this option is briefly annotated, details are available in a later section of this
document. document.
@ -180,6 +185,7 @@ document.
``` ```
### Section 1: Run Configuration ### Section 1: Run Configuration
```xml ```xml
<RunSettings> <RunSettings>
<RunConfiguration> <RunConfiguration>
@ -196,7 +202,8 @@ Available elements are:
1. **Test settings** 1. **Test settings**
*Example* _Example_
```xml ```xml
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<RunSettings> <RunSettings>
@ -212,8 +219,8 @@ Available elements are:
</RunConfiguration> </RunConfiguration>
</RunSettings> </RunSettings>
``` ```
*Description* _Description_
| Settings Element | Type | Function | | Settings Element | Type | Function |
|-------------------|--------|-------------------------------------------------------------------------------------------------| |-------------------|--------|-------------------------------------------------------------------------------------------------|
@ -227,6 +234,7 @@ Available elements are:
| ExecutionThreadApartmentState | string | Apartment state of thread which calls adapter's RunTests and Cancel APIs. Possible values: (MTA, STA). default is STA for .NET Full and MTA for .NET Core. STA supported only for .NET Full **Required Version: 15.5+.** [More details.](#execution-thread-apartment-state) | | ExecutionThreadApartmentState | string | Apartment state of thread which calls adapter's RunTests and Cancel APIs. Possible values: (MTA, STA). default is STA for .NET Full and MTA for .NET Core. STA supported only for .NET Full **Required Version: 15.5+.** [More details.](#execution-thread-apartment-state) |
Examples of valid `TargetFrameworkVersion`: Examples of valid `TargetFrameworkVersion`:
* .NETCoreApp, Version=v1.0 * .NETCoreApp, Version=v1.0
* .NETCoreApp, Version=v1.1 * .NETCoreApp, Version=v1.1
* .NETFramework, Version=v4.5 * .NETFramework, Version=v4.5
@ -238,7 +246,8 @@ These settings are a hint to adapters to behave in a particular way. These are
determined by runners based on user configuration. It is also possible for an determined by runners based on user configuration. It is also possible for an
user to provide these settings, if they want to tweak a run. user to provide these settings, if they want to tweak a run.
*Example* _Example_
```xml ```xml
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<RunSettings> <RunSettings>
@ -251,7 +260,7 @@ user to provide these settings, if they want to tweak a run.
</RunSettings> </RunSettings>
``` ```
*Description* _Description_
| Settings Element | Type | Function | | Settings Element | Type | Function |
|--------------------------|------|------------------------------------------------------------------------------------| |--------------------------|------|------------------------------------------------------------------------------------|
@ -264,7 +273,8 @@ user to provide these settings, if they want to tweak a run.
An IDE/Editor can set these settings to change the behavior of test platform. An IDE/Editor can set these settings to change the behavior of test platform.
They are not actionable for an adapter. They are not actionable for an adapter.
*Example* _Example_
```xml ```xml
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<RunSettings> <RunSettings>
@ -274,13 +284,14 @@ They are not actionable for an adapter.
</RunSettings> </RunSettings>
``` ```
*Description* _Description_
| Settings Element | Type | Function | | Settings Element | Type | Function |
|------------------|------|-------------------------------------------------------------------------------------------------------------| |------------------|------|-------------------------------------------------------------------------------------------------------------|
| BatchSize | int | Configures the frequency of run statistics. Discovered/Test results are send once `n` tests are accumulated | | BatchSize | int | Configures the frequency of run statistics. Discovered/Test results are send once `n` tests are accumulated |
### Section 2: Data Collection ### Section 2: Data Collection
```xml ```xml
<RunSettings> <RunSettings>
<!-- Configurations for out-of-process data collectors --> <!-- Configurations for out-of-process data collectors -->
@ -313,7 +324,8 @@ Data collector extension authors may use the content within the specific
for their data collector. For example, the code coverage data collector uses for their data collector. For example, the code coverage data collector uses
following section. following section.
*Example* _Example_
```xml ```xml
<RunSettings> <RunSettings>
<!-- Configurations for data collectors --> <!-- Configurations for data collectors -->
@ -341,15 +353,16 @@ following section.
</DataCollectors> </DataCollectors>
</DataCollectionRunSettings> </DataCollectionRunSettings>
</RunSettings> </RunSettings>
``` ```
*Description* _Description_
| Settings Element | Type | Function | | Settings Element | Type | Function |
|------------------|--------|---------------------------------------------------------------------------| |------------------|--------|---------------------------------------------------------------------------|
| DataCollector | string | Provides a data collector information. See below for required attributes. | | DataCollector | string | Provides a data collector information. See below for required attributes. |
Required attributes: Required attributes:
1. `friendlyName` provides a common name for the data collector. It is declared 1. `friendlyName` provides a common name for the data collector. It is declared
by a data collector implementation. by a data collector implementation.
2. `enabled` is used to enable/disable a data collector. If `true` the data 2. `enabled` is used to enable/disable a data collector. If `true` the data
@ -375,6 +388,7 @@ These settings can be used for Ordered tests and MSTest v1 based tests when runn
Users can now migrate testsettings to runsettings using [SettingsMigrator](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0023-TestSettings-Deprecation.md#migration) Users can now migrate testsettings to runsettings using [SettingsMigrator](https://github.com/Microsoft/vstest-docs/blob/main/RFCs/0023-TestSettings-Deprecation.md#migration)
Here is an example: Here is an example:
```xml ```xml
<Runsettings> <Runsettings>
<MSTest> <MSTest>
@ -421,6 +435,7 @@ Here is an example:
``` ```
#### Run Parameters #### Run Parameters
```xml ```xml
<!-- Parameters used by tests at runtime --> <!-- Parameters used by tests at runtime -->
<TestRunParameters> <TestRunParameters>
@ -439,6 +454,7 @@ available to tests during runtime.
> `TestContext` API. > `TestContext` API.
#### Adapter Configuration #### Adapter Configuration
An adapter may provide a section in runsettings for users. See [mstest config][] An adapter may provide a section in runsettings for users. See [mstest config][]
and [nunit config][] for more details. and [nunit config][] for more details.
@ -446,10 +462,13 @@ and [nunit config][] for more details.
[nunit config]: TODO [nunit config]: TODO
# Execution thread apartment state # Execution thread apartment state
This section explains usage of ExecutionThreadApartmentState element in runsettings and testplatform behavior for same. This section explains usage of ExecutionThreadApartmentState element in runsettings and testplatform behavior for same.
### Usage: ### Usage
#### using runsettings file:
#### using runsettings file
```xml ```xml
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<RunSettings> <RunSettings>
@ -459,22 +478,24 @@ This section explains usage of ExecutionThreadApartmentState element in runsetti
</RunSettings> </RunSettings>
``` ```
#### using command line: #### using command line
vstest.console.exe a.dll -- RunConfiguration.ExecutionThreadApartmentState=STA vstest.console.exe a.dll -- RunConfiguration.ExecutionThreadApartmentState=STA
dotnet test -f net46 -- RunConfiguration.ExecutionThreadApartmentState=STA dotnet test -f net46 -- RunConfiguration.ExecutionThreadApartmentState=STA
### History ### History
In Test Platform V1 ExecutionThreadApartmentState property can be set from vstest.executionengine*.exe.config file. Default value is `STA`. In Test Platform V1 ExecutionThreadApartmentState property can be set from vstest.executionengine*.exe.config file. Default value is `STA`.
### Behavior ### Behavior
In Test platform V2 ExecutionThreadApartmentState property default value is `MTA` for .NET Core and `STA` for .NET Full. `STA` value is only supported for .NET Framework. In Test platform V2 ExecutionThreadApartmentState property default value is `MTA` for .NET Core and `STA` for .NET Full. `STA` value is only supported for .NET Framework.
Warning should be shown on trying to set value `STA` for .NET Core and UAP10.0 frameworks tests. Warning should be shown on trying to set value `STA` for .NET Core and UAP10.0 frameworks tests.
* To support adapters which depends on thread test platform creates may need STA apartment state to run UI tests. `ExecutionThreadApartmentState` option can be used to set apartment state. Example: MSTest v1, MSTest v2 and MSCPPTest adapters.
- To support adapters which depends on thread test platform creates may need STA apartment state to run UI tests. `ExecutionThreadApartmentState` option can be used to set apartment state. Example: MSTest v1, MSTest v2 and MSCPPTest adapters. * The recommended way to make the tests run in STA thread is using custom attributes that adapter provides.
- The recommended way to make the tests run in STA thread is using custom attributes that adapter provides.
| Adapter | Attribute| | Adapter | Attribute|
|-----------|-----------| |-----------|-----------|

Просмотреть файл

@ -4,24 +4,27 @@ This article will help you build, test and try out local builds of the VS test
platform. platform.
## Prerequisites ## Prerequisites
Please ensure you have a `.net 4.6.2` or higher installed on the machine. Please ensure you have a `.net 4.6.2` or higher installed on the machine.
Clone the repository to a local directory. Rest of this article assumes Clone the repository to a local directory. Rest of this article assumes
`/src/vstest` as the location of source enlistment. `/src/vstest` as the location of source enlistment.
``` ```shell
> git clone https://github.com/Microsoft/vstest.git > git clone https://github.com/Microsoft/vstest.git
``` ```
If you're planning to use **Visual Studio** as development environment, please If you're planning to use **Visual Studio** as development environment, please
install `VS 2022` with .NET Desktop development workload, and install individual component `.NET Portable Library targeting pack`. install `VS 2022` with .NET Desktop development workload, and install individual component `.NET Portable Library targeting pack`.
If you're _not_ planning to use **Visual Studio** and only use CLI. You will need to install [.Net 46 targeting pack](https://www.microsoft.com/en-us/download/details.aspx?id=48136). The download link has two msis. Both needs to be installed. Otherwise build will fail asking to install net 46. If you're _not_ planning to use **Visual Studio** and only use CLI. You will need to install [.Net 46 targeting pack](https://www.microsoft.com/download/details.aspx?id=48136). The download link has two msis. Both needs to be installed. Otherwise build will fail asking to install net 46.
### Unix requirements ### Unix requirements
Install common tools Install common tools
```shell ```shell
$ sudo apt install libcurl4-openssl-dev sudo apt install libcurl4-openssl-dev
``` ```
Follow the instructions on [Mono Installation][mono-linux] page to install latest bits. Follow the instructions on [Mono Installation][mono-linux] page to install latest bits.
@ -29,6 +32,7 @@ Follow the instructions on [Mono Installation][mono-linux] page to install lates
[mono-linux]: http://www.mono-project.com/download/#download-lin [mono-linux]: http://www.mono-project.com/download/#download-lin
Rest of the article will provide steps for VS and CLI/Editors development. Rest of the article will provide steps for VS and CLI/Editors development.
## Build ## Build
### Building with Visual Studio ### Building with Visual Studio
@ -44,7 +48,7 @@ Binaries for each assembly are produced in the
To build the repository, run the following command: To build the repository, run the following command:
``` ```shell
> cd /src/vstest > cd /src/vstest
> build.cmd > build.cmd
``` ```
@ -71,7 +75,7 @@ assemblies can be found at
To build a particular configuration, use the `-c` option. E.g. to trigger a To build a particular configuration, use the `-c` option. E.g. to trigger a
release build use release build use
``` ```shell
> build.cmd -c Release > build.cmd -c Release
``` ```
@ -82,18 +86,18 @@ For other options, check `/src/vstest/scripts/build.ps1`.
There are following sets of tests: There are following sets of tests:
* Unit tests * Unit tests
- Very fast tests primarily validating individual units * Very fast tests primarily validating individual units
- Named as `<UnitUnderTest>.UnitTests` where UnitUnderTest is any product * Named as `<UnitUnderTest>.UnitTests` where UnitUnderTest is any product
assembly assembly
* Smoke tests * Smoke tests
- Slower end to end tests. Typically these cover P0 scenarios (99% of users * Slower end to end tests. Typically these cover P0 scenarios (99% of users
will use these, if these are broken, PR will not be merged) will use these, if these are broken, PR will not be merged)
- Driven by a real `vstest.console` executable * Driven by a real `vstest.console` executable
- Named as `Microsoft.TestPlatform.SmokeTests` * Named as `Microsoft.TestPlatform.SmokeTests`
* End to end tests * End to end tests
- Slower end to end tests for extensive coverage * Slower end to end tests for extensive coverage
- Driven by a real `vstest.console` executable * Driven by a real `vstest.console` executable
- Named as `Microsoft.TestPlatform.AcceptanceTests` * Named as `Microsoft.TestPlatform.AcceptanceTests`
As a principle, most of tests are unit tests (~70-80%), few smoke tests As a principle, most of tests are unit tests (~70-80%), few smoke tests
(~20-15%), fewer acceptance tests (~10-5%). (~20-15%), fewer acceptance tests (~10-5%).
@ -114,7 +118,7 @@ run only unit tests. For running smoke tests, use the `project:"Smoke"` filter.
To execute tests, run the following command: To execute tests, run the following command:
``` ```shell
> cd /src/vstest > cd /src/vstest
> test.cmd > test.cmd
``` ```
@ -122,27 +126,27 @@ To execute tests, run the following command:
By default, only unit tests are run. To run the smoke tests, provide the `-p` By default, only unit tests are run. To run the smoke tests, provide the `-p`
option to `test.cmd` to set test assembly pattern: option to `test.cmd` to set test assembly pattern:
``` ```shell
> test.cmd -p smoke > test.cmd -p smoke
``` ```
The `-p` option can be used to run tests for any assembly as well. E.g. The `-p` option can be used to run tests for any assembly as well. E.g.
following command will run tests for *datacollector*: following command will run tests for _datacollector_:
``` ```shell
> test.cmd -p datacollector > test.cmd -p datacollector
``` ```
Tests for a particular configuration can be run with following command. By Tests for a particular configuration can be run with following command. By
default, `Debug` configuration is run. default, `Debug` configuration is run.
``` ```shell
> test.cmd -c release > test.cmd -c release
``` ```
If you want to run a particular test. Eg: Test Name that contains Blame in Acceptance test If you want to run a particular test. Eg: Test Name that contains Blame in Acceptance test
``` ```shell
> test.cmd -p accept -f net451 -filter blame > test.cmd -p accept -f net451 -filter blame
``` ```
@ -156,7 +160,7 @@ using previous instructions.
Visual Studio 2017 RC ships with the test platform `vsix` generated by build.cmd. To use the locally Visual Studio 2017 RC ships with the test platform `vsix` generated by build.cmd. To use the locally
built version, use the following steps in a developer command prompt. built version, use the following steps in a developer command prompt.
``` ```shell
> vsixinstaller /src/vstest/artifacts/Debug/TestPlatform.vsix > vsixinstaller /src/vstest/artifacts/Debug/TestPlatform.vsix
# (replace Debug with Release as appropriate) # (replace Debug with Release as appropriate)
``` ```
@ -168,7 +172,7 @@ A `netcoreapp` target of vstest.console is dropped at
executed with any dotnet executable. You may choose to use the dotnet-cli we executed with any dotnet executable. You may choose to use the dotnet-cli we
have in `/src/vstest/tools/dotnet/dotnet` as well :) have in `/src/vstest/tools/dotnet/dotnet` as well :)
``` ```shell
> /src/vstest/tools/dotnet/dotnet /src/vstest/artifacts/Debug/netcoreapp1.0/vstest.console.dll /? > /src/vstest/tools/dotnet/dotnet /src/vstest/artifacts/Debug/netcoreapp1.0/vstest.console.dll /?
``` ```
@ -178,7 +182,7 @@ A `net46` target of vstest.console is dropped at
`src/vstest/artifacts/<Configuration>/net46/win7-x64`. It can be run directly as `src/vstest/artifacts/<Configuration>/net46/win7-x64`. It can be run directly as
follows: follows:
``` ```shell
> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe /? > /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe /?
``` ```
@ -188,7 +192,7 @@ Try to isolate the failure scenario. In the best case it's just a command line
that can demonstrate a bug. For example, a bug in discovery of tests can show up that can demonstrate a bug. For example, a bug in discovery of tests can show up
in following command line: in following command line:
``` ```shell
> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe mytest.dll /listTests /tests:*&&*#Ed > /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe mytest.dll /listTests /tests:*&&*#Ed
``` ```
@ -199,7 +203,8 @@ Another add a `Debugger.Launch` at the process launch points. E.g.
CoreCLR for netcoreapp scenario) and step through the code. CoreCLR for netcoreapp scenario) and step through the code.
## Running Tests with TPv2 using Test Explorer ## Running Tests with TPv2 using Test Explorer
Test Platform (TPv2) is packaged as a `vsix` with VS 2017 RC releases and lights up the .NET core and Live Unit Testing scenarios. It currently does not support UWP & data collector scenarios (code coverage & fakes). It is placed @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform"`.
Test Platform (TPv2) is packaged as a `vsix` with VS 2017 RC releases and lights up the .NET core and Live Unit Testing scenarios. It currently does not support UWP & data collector scenarios (code coverage & fakes). It is placed @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform"`.
Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow"`. Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ `"%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow"`.
@ -215,4 +220,5 @@ To use TPv2 for desktop - place `testplatform.config` @ `"%programfiles(x86)%\Mi
</appSettings> </appSettings>
</configuration> </configuration>
```` ````
If `feature.net35` and `feature.net40` are set to `true`, VS Test Explorer will use TPv2 for desktop flow for tests targetting net35 and net40 frameworks respectively. If `feature.datacollector` is set to `true`, VS Test Explorer will use TPv2 when data collectors are enabled.
If `feature.net35` and `feature.net40` are set to `true`, VS Test Explorer will use TPv2 for desktop flow for tests targeting net35 and net40 frameworks respectively. If `feature.datacollector` is set to `true`, VS Test Explorer will use TPv2 when data collectors are enabled.

Просмотреть файл

@ -7,7 +7,7 @@ This document outlines troubleshooting and diagnosis instructions for test platf
The console test runner (`vstest.console`) provides a `diag` switch to collect The console test runner (`vstest.console`) provides a `diag` switch to collect
traces to a log file. Invoke the runner using below command line: traces to a log file. Invoke the runner using below command line:
``` ```shell
> vstest.console testApp.dll --diag:log.txt > vstest.console testApp.dll --diag:log.txt
``` ```
@ -19,7 +19,7 @@ it doesn't exist.
Trace level can be changed using below command line: Trace level can be changed using below command line:
``` ```shell
> vstest.console testApp.dll --diag:log.txt;tracelevel=verbose > vstest.console testApp.dll --diag:log.txt;tracelevel=verbose
``` ```
@ -29,13 +29,14 @@ Allowed values for tracelevel are: off, error, warning, info and verbose.
The `--diag` option is supported on the `dotnet test` command as well. This will also produce same The `--diag` option is supported on the `dotnet test` command as well. This will also produce same
set of log files: `log.txt` and `log.*.txt`. set of log files: `log.txt` and `log.*.txt`.
```
```shell
> dotnet test --diag:log.txt > dotnet test --diag:log.txt
``` ```
To get traces for VSTest build task, enable the following environment variable: To get traces for VSTest build task, enable the following environment variable:
``` ```shell
> set VSTEST_BUILD_TRACE=1 > set VSTEST_BUILD_TRACE=1
or or
> $env:VSTEST_BUILD_TRACE=1 # powershell > $env:VSTEST_BUILD_TRACE=1 # powershell
@ -50,7 +51,7 @@ a `vstest.console.exe.config` configuration file in the same directory as
This technique is useful to capture logs for IDE/Editor scenarios. This technique is useful to capture logs for IDE/Editor scenarios.
``` ```shell
> where vstest.console > where vstest.console
d:\tmp\vstest.console.exe d:\tmp\vstest.console.exe
@ -59,7 +60,7 @@ d:\tmp\vstest.console.exe
Add the following content to the config file (`vstest.console.exe.config` or `testhost.exe.config`). Add the following content to the config file (`vstest.console.exe.config` or `testhost.exe.config`).
``` ```shell
<configuration> <configuration>
<system.diagnostics> <system.diagnostics>
<sources> <sources>
@ -97,10 +98,12 @@ Add the following content to the config file (`vstest.console.exe.config` or `te
``` ```
## Collect trace in VSTS CI (VSTest Task) ## Collect trace in VSTS CI (VSTest Task)
Add a variable to the build definition named `System.Debug` with value `True`. This can Add a variable to the build definition named `System.Debug` with value `True`. This can
be done while a new build is queued. be done while a new build is queued.
This variable will ensure: This variable will ensure:
* Exact command line of `vstest.console.exe` is reported. It can be used to isolate if there * Exact command line of `vstest.console.exe` is reported. It can be used to isolate if there
are errors in translating user inputs in the task to command line options of vstest.console.exe. are errors in translating user inputs in the task to command line options of vstest.console.exe.
* The task invokes `vstest.console.exe` with `/diag` option. The log files can be uploaded * The task invokes `vstest.console.exe` with `/diag` option. The log files can be uploaded
@ -112,7 +115,7 @@ The runner and test host processes support waiting for debugger attach. You can
enable the following environment variable to keep `vstest.console.exe` waiting for enable the following environment variable to keep `vstest.console.exe` waiting for
debugger attach: debugger attach:
``` ```shell
> set VSTEST_RUNNER_DEBUG=1 > set VSTEST_RUNNER_DEBUG=1
> vstest.console testApp.dll > vstest.console testApp.dll
Waiting for debugger attach... Waiting for debugger attach...
@ -120,10 +123,10 @@ Process Id: 51928, Name: vstest.console
``` ```
For test host process (`testhost.exe`, or `dotnet exec testhost.dll` depending on For test host process (`testhost.exe`, or `dotnet exec testhost.dll` depending on
target framework) use the following environment variable. target framework) use the following environment variable.
Execution will halt until debugger is attached. Execution will halt until debugger is attached.
``` ```shell
> set VSTEST_HOST_DEBUG=1 > set VSTEST_HOST_DEBUG=1
> vstest.console testApp.dll > vstest.console testApp.dll
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0 Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0

Просмотреть файл

@ -1,7 +1,9 @@
# TestCase filter # TestCase filter
This document will help you to selectively execute tests based on filtering conditions through `--filter` for `dotnet test` and `--testcasefilter` for `vstest.console.exe`. This document will help you to selectively execute tests based on filtering conditions through `--filter` for `dotnet test` and `--testcasefilter` for `vstest.console.exe`.
### Syntax ### Syntax
`dotnet test --filter <Expression>` or `dotnet test --filter <Expression>` or
`vstest.console.exe --testcasefilter:<Expression>` `vstest.console.exe --testcasefilter:<Expression>`
@ -20,6 +22,7 @@ supported by popular unit test frameworks.
| Xunit | <ul><li>FullyQualifiedName</li><li>DisplayName</li><li>Traits</li></ul> | | Xunit | <ul><li>FullyQualifiedName</li><li>DisplayName</li><li>Traits</li></ul> |
Allowed **operators**: Allowed **operators**:
* `=` implies an exact match * `=` implies an exact match
* `!=` implies an exact not match * `!=` implies an exact not match
* `~` implies a contains lookup * `~` implies a contains lookup
@ -27,7 +30,7 @@ Allowed **operators**:
**Value** is a string. All the lookups are case insensitive. **Value** is a string. All the lookups are case insensitive.
**Escape Sequences** must be used to represent characters in the value that have special meanings in the filter, i.e. filter operators. **Escape Sequences** must be used to represent characters in the value that have special meanings in the filter, i.e. filter operators.
| Escape Sequence | Represents | | Escape Sequence | Represents |
| -------------- | -------------------- | | -------------- | -------------------- |
@ -44,13 +47,16 @@ A helper method `Microsoft.VisualStudio.TestPlatform.ObjectModel.Utilities.Filte
is also available by referencing the `Microsoft.VisualStudio.TestPlatform.ObjectModel` NuGet package, which can be used to escape strings programatically. is also available by referencing the `Microsoft.VisualStudio.TestPlatform.ObjectModel` NuGet package, which can be used to escape strings programatically.
Expressions can be joined with boolean operators. The following boolean operators are supported: Expressions can be joined with boolean operators. The following boolean operators are supported:
* `|` implies a boolean `OR` * `|` implies a boolean `OR`
* `&` implies a boolean `AND` * `&` implies a boolean `AND`
## Examples ## Examples
The following examples use `dotnet test`, if you're using `vstest.console.exe` replace `--filter ` with `--testcasefilter:`.
The following examples use `dotnet test`, if you're using `vstest.console.exe` replace `--filter` with `--testcasefilter:`.
### MSTest ### MSTest
```CSharp ```CSharp
namespace MSTestNamespace namespace MSTestNamespace
{ {
@ -88,8 +94,8 @@ namespace MSTestNamespace
| Expression | What it does? | | Expression | What it does? |
| ---------- | ------------- | | ---------- | ------------- |
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1\|TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName __or__ TestCategory is CategoryA. | | `dotnet test --filter "FullyQualifiedName~UnitTestClass1\|TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName **or** TestCategory is CategoryA. |
| `dotnet test --filter "FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName __and__ TestCategory is CategoryA. | | `dotnet test --filter "FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA"` | Runs tests which have `UnitTestClass1` in FullyQualifiedName **and** TestCategory is CategoryA. |
| `dotnet test --filter "(FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `UnitTestClass1` and TestCategory is CategoryA or Priority is 1. | | `dotnet test --filter "(FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `UnitTestClass1` and TestCategory is CategoryA or Priority is 1. |
### xUnit ### xUnit
@ -101,6 +107,7 @@ namespace MSTestNamespace
| `dotnet test --filter DisplayName~TestClass1` | Runs tests whose display name contains `TestClass1`. | | `dotnet test --filter DisplayName~TestClass1` | Runs tests whose display name contains `TestClass1`. |
#### Using traits for filter #### Using traits for filter
```CSharp ```CSharp
namespace XUnitNamespace namespace XUnitNamespace
{ {
@ -123,6 +130,7 @@ namespace XUnitNamespace
} }
``` ```
In above code we defined traits with keys `Category` and `Priority` which can be used for filtering. In above code we defined traits with keys `Category` and `Priority` which can be used for filtering.
| Expression | What it does? | | Expression | What it does? |
@ -134,6 +142,6 @@ In above code we defined traits with keys `Category` and `Priority` which can be
| Expression | What it does? | | Expression | What it does? |
| ---------- | ------------- | | ---------- | ------------- |
| `dotnet test --filter "FullyQualifiedName~TestClass1\|Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName __or__ Category is Nightly. | | `dotnet test --filter "FullyQualifiedName~TestClass1\|Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName **or** Category is Nightly. |
| `dotnet test --filter "FullyQualifiedName~TestClass1&Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName __and__ Category is Nightly. | | `dotnet test --filter "FullyQualifiedName~TestClass1&Category=Nightly"` | Runs tests which have `TestClass1` in FullyQualifiedName **and** Category is Nightly. |
| `dotnet test --filter "(FullyQualifiedName~TestClass1&Category=Nightly)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `TestClass1` and Category is CategoryA or Priority is 1. | | `dotnet test --filter "(FullyQualifiedName~TestClass1&Category=Nightly)\|Priority=1"` | Runs tests which have either FullyQualifiedName contains `TestClass1` and Category is CategoryA or Priority is 1. |

Просмотреть файл

@ -1,10 +1,11 @@
# Reporting test results # Reporting test results
Test discovery, execution results in a test run can be controlled with test Test discovery, execution results in a test run can be controlled with test
loggers. This document will cover details on installation, usage and authoring loggers. This document will cover details on installation, usage and authoring
of test loggers. of test loggers.
## Test loggers ## Test loggers
A test logger is a test platform extension to control reporting of test results. A test logger is a test platform extension to control reporting of test results.
It can perform tasks when a test run message, individual test It can perform tasks when a test run message, individual test
results or the test run completion events are reported by the test platform. results or the test run completion events are reported by the test platform.
@ -17,6 +18,7 @@ services. Default inputs to a test logger can be provided in the command line.
if you're interested in the architecture of a test logger. if you're interested in the architecture of a test logger.
### Available test loggers ### Available test loggers
| Scenario | Nuget Package | Source Repository | | Scenario | Nuget Package | Source Repository |
| -------- | ------------- | ----------------- | | -------- | ------------- | ----------------- |
| Local, CI, CD | Inbuilt | [Trx Logger][] | | Local, CI, CD | Inbuilt | [Trx Logger][] |
@ -49,6 +51,7 @@ if you're interested in the architecture of a test logger.
Want to add your logger? Please send a PR with changes in this doc. Want to add your logger? Please send a PR with changes in this doc.
## Acquisition ## Acquisition
A test logger should be made available as a NuGet package (preferred), or as A test logger should be made available as a NuGet package (preferred), or as
a zip file (for e.g. loggers for C++ etc.). a zip file (for e.g. loggers for C++ etc.).
@ -59,16 +62,17 @@ them in the same directory as the test assemblies.
If the test logger is made available as a zip file, it should be extracted If the test logger is made available as a zip file, it should be extracted
to one of the following locations: to one of the following locations:
1. the `Extensions` folder along side `vstest.console.exe`. E.g. in case of 1. the `Extensions` folder along side `vstest.console.exe`. E.g. in case of
dotnet-cli, the path could be `/sdk/<version>/Extensions` directory. dotnet-cli, the path could be `/sdk/<version>/Extensions` directory.
2. any well known location on the filesystem 2. any well known location on the filesystem
> Version Note: new in 15.1 > Version Note: new in 15.1
In case of #2, user can specify the full path to the location using `/TestAdapterPath:<path>` In case of #2, user can specify the full path to the location using `/TestAdapterPath:<path>`
command line switch. Test platform will locate extensions from the provided command line switch. Test platform will locate extensions from the provided
directory. directory.
## Naming ## Naming
Test platform will look for assemblies named `*.testlogger.dll` when it's trying Test platform will look for assemblies named `*.testlogger.dll` when it's trying
to load test loggers. to load test loggers.
@ -76,23 +80,31 @@ to load test loggers.
> For 15.0 version, the test loggers are also discovered from *.testadapter.dll > For 15.0 version, the test loggers are also discovered from *.testadapter.dll
## Create a test logger ## Create a test logger
Go through the following steps to create your own logger Go through the following steps to create your own logger
1) Add a nuget reference of package `Microsoft.TestPlatform.ObjectModel`. 1) Add a nuget reference of package `Microsoft.TestPlatform.ObjectModel`.
2) Implement ITestLoggerWithParameters (or ITestLogger, if your logger is not expecting any parameter). [Logger Example](https://github.com/spekt/xunit.testlogger/blob/master/src/Xunit.Xml.TestLogger/XunitXmlTestLogger.cs#L19) 2) Implement ITestLoggerWithParameters (or ITestLogger, if your logger is not expecting any parameter). [Logger Example](https://github.com/spekt/xunit.testlogger/blob/master/src/Xunit.Xml.TestLogger/XunitXmlTestLogger.cs#L19)
3) Name your logger assemlby `*.testlogger.dll`. [Detailed](https://github.com/Microsoft/vstest-docs/blob/main/docs/report.md#naming) 3) Name your logger assemlby `*.testlogger.dll`. [Detailed](https://github.com/Microsoft/vstest-docs/blob/main/docs/report.md#naming)
## Enable a test logger ## Enable a test logger
A test logger must be explicitly enabled using the command line. E.g. A test logger must be explicitly enabled using the command line. E.g.
```
```shell
vstest.console test_project.dll /logger:mylogger vstest.console test_project.dll /logger:mylogger
``` ```
Where `mylogger` is the LoggerUri or FriendlyName of the logger. Where `mylogger` is the LoggerUri or FriendlyName of the logger.
## Configure reporting ## Configure reporting
Additional arguments to a logger can also be passed in the command line. E.g. Additional arguments to a logger can also be passed in the command line. E.g.
```
```shell
vstest.console test_project.dll /logger:mylogger;Setting=Value vstest.console test_project.dll /logger:mylogger;Setting=Value
``` ```
Where `mylogger` is the LoggerUri or FriendlyName of the logger. Where `mylogger` is the LoggerUri or FriendlyName of the logger.
`Setting` is the name of the additional argument and `Value`is its value. `Setting` is the name of the additional argument and `Value`is its value.
@ -101,11 +113,12 @@ It is upto the logger implementation to support additional arguments.
## Syntax of default loggers ## Syntax of default loggers
### 1) Console logger ### 1) Console logger
Console logger is the default logger and it is used to output the test results into console window. Console logger is the default logger and it is used to output the test results into console window.
#### Syntax #### Syntax
``` ```shell
For dotnet test or dotnet vstest : For dotnet test or dotnet vstest :
--logger:console[;verbosity=<Defaults to "minimal">] --logger:console[;verbosity=<Defaults to "minimal">]
@ -116,7 +129,8 @@ Argument "verbosity" define the verbosity level of console logger. Allowed value
``` ```
#### Example #### Example
```
```shell
vstest.console.exe Tests.dll /logger:"console;verbosity=normal" vstest.console.exe Tests.dll /logger:"console;verbosity=normal"
If you are using "dotnet test", then use the following command If you are using "dotnet test", then use the following command
@ -129,10 +143,12 @@ dotnet test Tests.csproj -v normal
``` ```
### 2) Trx logger ### 2) Trx logger
Trx logger is used to log test results into a Visual Studio Test Results File (TRX). Trx logger is used to log test results into a Visual Studio Test Results File (TRX).
#### Syntax #### Syntax
```
```shell
/logger:trx [;LogFileName=<Defaults to unique file name>] /logger:trx [;LogFileName=<Defaults to unique file name>]
Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory. Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.
@ -141,7 +157,8 @@ Where "LogFileName" can be absolute or relative path. If path is relative, it wi
#### Examples #### Examples
Suppose the current working directory is "c:\tempDirecory". Suppose the current working directory is "c:\tempDirecory".
```
```shell
1) vstest.console.exe Tests.dll /logger:trx 1) vstest.console.exe Tests.dll /logger:trx
trx file will get generated in location "c:\tempDirecory\TestResults" trx file will get generated in location "c:\tempDirecory\TestResults"
@ -153,10 +170,12 @@ trx file will be "c:\temp\logFile.txt"
``` ```
### 3) Html logger ### 3) Html logger
Html logger is used to log test results into a html file. Html logger is used to log test results into a html file.
#### Syntax #### Syntax
```
```shell
/logger:html [;LogFileName=<Defaults to unique file name>] /logger:html [;LogFileName=<Defaults to unique file name>]
Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory. Where "LogFileName" can be absolute or relative path. If path is relative, it will be relative to "TestResults" directory, created under current working directory.
@ -166,7 +185,8 @@ Where "LogFileName" can be absolute or relative path. If path is relative, it wi
#### Examples #### Examples
Suppose the current working directory is "c:\tempDirecory". Suppose the current working directory is "c:\tempDirecory".
```
```shell
1) vstest.console.exe Tests.dll /logger:html 1) vstest.console.exe Tests.dll /logger:html
Html file will get generated in location "c:\tempDirecory\TestResults" Html file will get generated in location "c:\tempDirecory\TestResults"
@ -177,6 +197,6 @@ Html file will be "c:\tempDirecory\TestResults\relativeDir\logFile.html"
Html file will be "c:\temp\logFile.html" Html file will be "c:\temp\logFile.html"
``` ```
## Related links ## Related links
TODO: link to author a test logger TODO: link to author a test logger

Просмотреть файл

@ -1,17 +1,21 @@
# Testplatform Migration Known issues # Testplatform Migration Known issues
Here are the current known issues you may face when running tests, along with available workarounds. Here are the current known issues you may face when running tests, along with available workarounds.
# Known Issues @# Known Issues
- [Change in Thread.CurrentPrincipal value.](CurrentPrincipal) - [Change in Thread.CurrentPrincipal value.](CurrentPrincipal)
- [Change in test execution processes name.](processname) - [Change in test execution processes name.](processname)
### <b>Change in Thread.CurrentPrincipal value.</b><a name="CurrentPrincipal"></a> ### <b>Change in Thread.CurrentPrincipal value.</b><a name="CurrentPrincipal"></a>
- <b>Issue:</b> <br/> - <b>Issue:</b> <br/>
Tests that depend on `Thread.CurrentPrincipal` may fail. This is due to change in inter process commincation in Testplatform. Tests that depend on `Thread.CurrentPrincipal` may fail. This is due to change in inter process commincation in Testplatform.
- <b>Workaround:</b> <br/> - <b>Workaround:</b> <br/>
Use an alternative like `System.Security.Principal.WindowsIdentity.GetCurrent()` Use an alternative like `System.Security.Principal.WindowsIdentity.GetCurrent()`
### <b>Change in test execution processes name.</b><a name="processname"></a> ### <b>Change in test execution processes name.</b><a name="processname"></a>
- <b>Issue:</b> <br/> - <b>Issue:</b> <br/>
Tests that depend on the name of the currnet running process may fail. Tests that depend on the name of the currnet running process may fail.
- <b>Workaround:</b> <br/> - <b>Workaround:</b> <br/>

Просмотреть файл

@ -4,12 +4,15 @@ The goal of this document is to help the test platform users to collect useful i
## Dotnet CLI ## Dotnet CLI
#### Collect logs and crash dump ### Collect logs and crash dump
```bash
```shell
dotnet test --diag:log.txt --blame-crash --blame-crash-dump-type full dotnet test --diag:log.txt --blame-crash --blame-crash-dump-type full
``` ```
At the end of the execution you'll find the list of artifacts generated with the link to the file dump: At the end of the execution you'll find the list of artifacts generated with the link to the file dump:
```bash
```shell
Starting test execution, please wait... Starting test execution, please wait...
Logging Vstest Diagnostics in file: C:\git\issue\bug\log.txt Logging Vstest Diagnostics in file: C:\git\issue\bug\log.txt
A total of 1 test files matched the specified pattern. A total of 1 test files matched the specified pattern.
@ -20,12 +23,15 @@ Attachments:
C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\Sequence_bfcc4d8558654413a3fb2f5164695bf6.xml C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\Sequence_bfcc4d8558654413a3fb2f5164695bf6.xml
C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\dotnet.exe_11876_1660721586_crashdump.dmp C:\git\issues\bug\TestResults\620c075b-e035-41d2-b950-159f57abc604\dotnet.exe_11876_1660721586_crashdump.dmp
``` ```
You'll find 3 files for logs(runner, datacollector, host), datacollector one can be missing. You'll find 3 files for logs(runner, datacollector, host), datacollector one can be missing.
```bash
```shell
-a---- 8/17/2022 9:33 AM 30001 log.datacollector.22-08-17_09-32-54_50516_1.txt -a---- 8/17/2022 9:33 AM 30001 log.datacollector.22-08-17_09-32-54_50516_1.txt
-a---- 8/17/2022 9:33 AM 37222 log.host.22-08-17_09-32-55_24965_7.txt -a---- 8/17/2022 9:33 AM 37222 log.host.22-08-17_09-32-55_24965_7.txt
-a---- 8/17/2022 9:33 AM 200345 log.txt -a---- 8/17/2022 9:33 AM 200345 log.txt
``` ```
## Azure DevOps ## Azure DevOps
### @VSTest2 task ### @VSTest2 task
@ -245,11 +251,12 @@ jobs:
Note: these logs could contain sensitive information (paths, project name...). Make sure to clean them or use the Visual Studio `Send Feedback` button. Don't put anything you want to keep private in the title or content of the initial report, which is public. Instead, say that you'll send details privately in a separate comment. Once the problem report is created, it's now possible to specify who can see your replies and attachments. Note: these logs could contain sensitive information (paths, project name...). Make sure to clean them or use the Visual Studio `Send Feedback` button. Don't put anything you want to keep private in the title or content of the initial report, which is public. Instead, say that you'll send details privately in a separate comment. Once the problem report is created, it's now possible to specify who can see your replies and attachments.
## Use procdump https://docs.microsoft.com/it-it/sysinternals/downloads/procdump on Windows ## Use procdump <https://docs.microsoft.com/it-it/sysinternals/downloads/procdump> on Windows
Sometimes it's not possible to take the dump using test platform tool because the crash happen before we're able to attach to the process to take the dump self. In that situation we need a way to register for dump at process startup level. Sometimes it's not possible to take the dump using test platform tool because the crash happen before we're able to attach to the process to take the dump self. In that situation we need a way to register for dump at process startup level.
To achieve it we can use procdump that will install machine wide Just-in-time (AeDebug) debugger. To achieve it we can use procdump that will install machine wide Just-in-time (AeDebug) debugger.
```bash
```shell
PS C:\tools\Procdump> .\procdump.exe -i C:\tools\Procdump\dumps PS C:\tools\Procdump> .\procdump.exe -i C:\tools\Procdump\dumps
ProcDump v10.11 - Sysinternals process dump utility ProcDump v10.11 - Sysinternals process dump utility
@ -268,7 +275,9 @@ Set to:
ProcDump is now set as the Just-in-time (AeDebug) debugger. ProcDump is now set as the Just-in-time (AeDebug) debugger.
``` ```
After you can run your application and in case of crash a dump will be automatically taken inside the `C:\tools\Procdump\dumps` directory. After you can run your application and in case of crash a dump will be automatically taken inside the `C:\tools\Procdump\dumps` directory.
``` ```
PS C:\tools\Procdump> ls C:\tools\Procdump\dumps PS C:\tools\Procdump> ls C:\tools\Procdump\dumps
@ -280,8 +289,11 @@ Mode LastWriteTime Length Name
---- ------------- ------ ---- ---- ------------- ------ ----
-a---- 8/17/2022 9:42 AM 6161605 dotnet.exe_220817_094234.dmp -a---- 8/17/2022 9:42 AM 6161605 dotnet.exe_220817_094234.dmp
``` ```
You can unistall the automatic generation running at the end of the collection phase
```bash You can unistall the automatic generation running at the end of the collection phase
```shell
.\procdump.exe -u .\procdump.exe -u
``` ```
Keep in mind that this mode will collect machine wide crash so every process in the machine that will crash will collect a dump in the folder. Keep in mind that this mode will collect machine wide crash so every process in the machine that will crash will collect a dump in the folder.

Просмотреть файл

@ -1,53 +1,69 @@
# Issue tracking # Issue tracking
The vstest project tracks issues and feature requests using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md) for the vstest repository. The vstest project tracks issues and feature requests using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md) for the vstest repository.
## Submitting an Issue ## Submitting an Issue
First, please do a search in [issues](https://github.com/Microsoft/vstest/issues) to see if the issue or feature request has already been filed. Use this [query](https://github.com/Microsoft/vstest/issues?q=is%3Aopen+is%3Aissue+sort%3Areactions-%2B1-desc) to search for the most popular feature requests. First, please do a search in [issues](https://github.com/Microsoft/vstest/issues) to see if the issue or feature request has already been filed. Use this [query](https://github.com/Microsoft/vstest/issues?q=is%3Aopen+is%3Aissue+sort%3Areactions-%2B1-desc) to search for the most popular feature requests.
If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment. If you find your issue already exists, make relevant comments and add your [reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments). Use a reaction in place of a "+1" comment.
- 👍 - upvote - 👍 - upvote
- 👎 - downvote - 👎 - downvote
If you cannot find an existing issue that describes your issue, submit it using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md). Remember to follow the instruction mentioned therein carefully. If you cannot find an existing issue that describes your issue, submit it using the [issue template](https://github.com/Microsoft/vstest/blob/main/.github/ISSUE_TEMPLATE.md). Remember to follow the instruction mentioned therein carefully.
## Issue triage ## Issue triage
Please follow the guidelines in the issue template when filing an issue or a pull request. Please follow the guidelines in the issue template when filing an issue or a pull request.
New issues or pull requests submitted by the community are triaged by a team member using the inbox query below. New issues or pull requests submitted by the community are triaged by a team member using the inbox query below.
## Inbox query ## Inbox query
The [inbox query](https://github.com/Microsoft/vstest/issues?utf8=%E2%9C%93&q=is%3Aopen%20no%3Aassignee%20-label%3Abacklog%20-label%3Aenhancement) will return the following: The [inbox query](https://github.com/Microsoft/vstest/issues?utf8=%E2%9C%93&q=is%3Aopen%20no%3Aassignee%20-label%3Abacklog%20-label%3Aenhancement) will return the following:
- Open issues or pull requests that are not enhancements and that have no owner assigned. - Open issues or pull requests that are not enhancements and that have no owner assigned.
## Initial triage - Issue tagging ## Initial triage - Issue tagging
Issues will then be tagged as follows: Issues will then be tagged as follows:
- Is the issue ***invalid***? It will be closed, and the reason will be provided. - Is the issue ***invalid***? It will be closed, and the reason will be provided.
- Is the issue a ***general question***, like *are data collector events synchronous*? It will be tagged as a ***question***. - Is the issue a ***general question***, like *are data collector events synchronous*? It will be tagged as a ***question***.
- Is the issue a feature request or an enhancement. It will be tagged as an ***enhancement***. - Is the issue a feature request or an enhancement. It will be tagged as an ***enhancement***.
- Else, the issue will be tagged as a ***bug***. - Else, the issue will be tagged as a ***bug***.
## Secondary triage – assignment and follow through ## Secondary triage – assignment and follow through
As and when an issue get assigned to a team member, the following secondary triage will happen As and when an issue get assigned to a team member, the following secondary triage will happen
- When a team member picks up an issue, they will first assign it to themselves. - When a team member picks up an issue, they will first assign it to themselves.
- Ensure that the issue has an appropriate tag (***question***, ***enhancement***, ***bug***). - Ensure that the issue has an appropriate tag (***question***, ***enhancement***, ***bug***).
- If an issue needs a repro, tag it with ***need-repro*** and ask for a repro in a comment. - If an issue needs a repro, tag it with ***need-repro*** and ask for a repro in a comment.
## Ongoing issue management ## Ongoing issue management
- Issues tagged ***need-repro*** info will be closed if no additional information is provided for 7 days. - Issues tagged ***need-repro*** info will be closed if no additional information is provided for 7 days.
Team members will strive to resolve every bug within a stipulated period of time – we would like that to be a period of 14 days from the date the issue was filed. Team members will strive to resolve every bug within a stipulated period of time – we would like that to be a period of 14 days from the date the issue was filed.
# Community contributions - up-for-grabs # Community contributions - up-for-grabs
We strongly encourage the community to contribute enhancements and bug-fixes. Therefore, team members will look to add as much information in the issues as required so that you can make an effective contribution. Such issues will be tagged ***up-for-grabs***. We strongly encourage the community to contribute enhancements and bug-fixes. Therefore, team members will look to add as much information in the issues as required so that you can make an effective contribution. Such issues will be tagged ***up-for-grabs***.
# Planning # Planning
## Triage ## Triage
Bugs and enhancements will be assigned a milestone, and within a milestone they will be assigned a priority. The priority dictates the order in which issues should be addressed. A important bug (something that we think is critical for the milestone) is to be addressed before the other bugs. Bugs and enhancements will be assigned a milestone, and within a milestone they will be assigned a priority. The priority dictates the order in which issues should be addressed. A important bug (something that we think is critical for the milestone) is to be addressed before the other bugs.
To find out when a bug fix will be available in an update, then please check the milestone that is assigned to the issue. To find out when a bug fix will be available in an update, then please check the milestone that is assigned to the issue.
Please see Issue Tracking for a description of the different workflows we are using. Please see Issue Tracking for a description of the different workflows we are using.
## Milestone planning ## Milestone planning
We typically plan for a quarter and establish a set of themes we want to work towards, and prioritize enhancements and bug-fixes accordingly. We typically plan for a quarter and establish a set of themes we want to work towards, and prioritize enhancements and bug-fixes accordingly.
During the planning process we prioritize items as follows: During the planning process we prioritize items as follows:
- Important ***bugs*** - crashes, regressions, and issues that do not have a reasonable workaround. - Important ***bugs*** - crashes, regressions, and issues that do not have a reasonable workaround.
- ***Enhancements*** that have many reactions. - ***Enhancements*** that have many reactions.

Просмотреть файл

@ -5,19 +5,25 @@ We aim to continuously deliver improvements that will ship with Visual Studio an
Over the past several quarters, we have made many enhancements - from introducing support for Mono, to refactoring the platform to make it ready to support device testing, to performance improvements, to enabling robust C++ support, to improved documentation, and more. For a complete list see here: [Release Notes](https://github.com/Microsoft/vstest-docs/blob/main/docs/releases.md) Over the past several quarters, we have made many enhancements - from introducing support for Mono, to refactoring the platform to make it ready to support device testing, to performance improvements, to enabling robust C++ support, to improved documentation, and more. For a complete list see here: [Release Notes](https://github.com/Microsoft/vstest-docs/blob/main/docs/releases.md)
## Roadmap ## Roadmap
We typically plan for a quarter, and establish a set of themes we want to work towards. Here are the themes we will work on this quarter. We typically plan for a quarter, and establish a set of themes we want to work towards. Here are the themes we will work on this quarter.
### Reach: Enable leveraging your vstest experience across all supported platforms ### Reach: Enable leveraging your vstest experience across all supported platforms
Over the course of the next phase of execution we will make TPV2 the "default" for all scenarios across Visual Studio and Visual Studio Team Services (VSTS) – i.e. extending it to .NET Framework, UWP, and the VSTest task in VSTS. We will ship a standalone package that can be potentially used in other CI systems even. This is a big switch. We will strive to maintain backwards compat, and publish migration guides for the few features that require to be migrated, and help you in the migration. Over the course of the next phase of execution we will make TPV2 the "default" for all scenarios across Visual Studio and Visual Studio Team Services (VSTS) – i.e. extending it to .NET Framework, UWP, and the VSTest task in VSTS. We will ship a standalone package that can be potentially used in other CI systems even. This is a big switch. We will strive to maintain backwards compat, and publish migration guides for the few features that require to be migrated, and help you in the migration.
### Performance: At scale ### Performance: At scale
Performance has been an area where we have received feedback, and made strong progress as well. It will continue to remain a focus. We will look to make improvements across the pipeline from the Test Explorer to the framework adapters, to enhance the overall end to end performance. Performance has been an area where we have received feedback, and made strong progress as well. It will continue to remain a focus. We will look to make improvements across the pipeline from the Test Explorer to the framework adapters, to enhance the overall end to end performance.
### UWP, Win10 IoT Core Support ### UWP, Win10 IoT Core Support
UWP is the application platform for Windows 10, to reach all Windows 10 devices – PC, tablet, phone, Xbox, HoloLens, Surface Hub and more. The vstest engine is architected so that it can be extended to support new application platforms. Such extensions will come from teams who understand their platforms the best, and integrated with vstest. To drive home this point, vstest will be extended to support testing UWP applications. In particular we will light up support for Win10 IoT Core. UWP is the application platform for Windows 10, to reach all Windows 10 devices – PC, tablet, phone, Xbox, HoloLens, Surface Hub and more. The vstest engine is architected so that it can be extended to support new application platforms. Such extensions will come from teams who understand their platforms the best, and integrated with vstest. To drive home this point, vstest will be extended to support testing UWP applications. In particular we will light up support for Win10 IoT Core.
### Code Coverage for .NET Core ### Code Coverage for .NET Core
This has been a clear ask from the community, and we are working towards enabling this support. The code coverage infrastructure consumes information from PDB files. Specifically with regard to .NET Core, it now needs to understand the new portable PDB format. We are working cross-team to introduce this support in order to light up code coverage support for .NET Core. This has been a clear ask from the community, and we are working towards enabling this support. The code coverage infrastructure consumes information from PDB files. Specifically with regard to .NET Core, it now needs to understand the new portable PDB format. We are working cross-team to introduce this support in order to light up code coverage support for .NET Core.
## Summary ## Summary
These are examples of the work we will be focusing on this quarter. We will provide details through individual issues. Follow along, and let us know what you think. We look forward to working with you! These are examples of the work we will be focusing on this quarter. We will provide details through individual issues. Follow along, and let us know what you think. We look forward to working with you!