microsoft
/
testfx-docs
зеркало из https://github.com/microsoft/testfx-docs.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Formatting updates (#58) 

* code formatting

* small improvemens
This commit is contained in:
Hans De Mulder 2018-10-19 12:53:08 +02:00 коммит произвёл pvlakshm
Родитель 44a1170581
Коммит 9caffd8974
8 изменённых файлов: 392 добавлений и 386 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

90
RFCs/001-Framework-Extensibility-Trait-Attributes.md
Просмотреть файл

@ -4,7 +4,7 @@
This details the MSTest V2 framework extensibility for attributes that are traits for a test.
## Motivation
It is a requirement for teams to have trait attributes which are strongly typed over a test method as opposed to just a KeyvaluePair<string,string>. This allows teams to have a standard across the team which is less error prone and more natural to specify. Users can also filter tests based on the values of these attributes.
It is a requirement for teams to have trait attributes which are strongly typed over a test method as opposed to just a `KeyvaluePair<string, string>`. This allows teams to have a standard across the team which is less error prone and more natural to specify. Users can also filter tests based on the values of these attributes.
## Detailed Design
@ -15,64 +15,64 @@ It is a requirement for teams to have trait attributes which are strongly typed
4. This extensibility should also guarantee that attributes in MSTest V1 which are brought into MSTest V2 with this extensibility model should be source compatible.
### Proposed solution
The test framework currently has a TestProperty attribute which can be used to define custom traits as a KeyValuePair<string,string>. The definition of this attribute is as below:
The test framework currently has a TestProperty attribute which can be used to define custom traits as a `KeyValuePair<string, string>`. The definition of this attribute is as below:
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public sealed class TestPropertyAttribute : Attribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public sealed class TestPropertyAttribute : Attribute
{
/// <summary>
/// Initializes a new instance of the <see cref="TestPropertyAttribute"/> class.
/// </summary>
/// <param name="name">
/// The name.
/// </param>
/// <param name="value">
/// The value.
/// </param>
public TestPropertyAttribute(string name, string value)
{
/// <summary>
/// Initializes a new instance of the <see cref="TestPropertyAttribute"/> class.
/// </summary>
/// <param name="name">
/// The name.
/// </param>
/// <param name="value">
/// The value.
/// </param>
public TestPropertyAttribute(string name, string value)
{
this.Name = name;
this.Value = value;
}
/// <summary>
/// Gets the name.
/// </summary>
public string Name { get; }
/// <summary>
/// Gets the value.
/// </summary>
public string Value { get; }
this.Name = name;
this.Value = value;
}
/// <summary>
/// Gets the name.
/// </summary>
public string Name { get; }
/// <summary>
/// Gets the value.
/// </summary>
public string Value { get; }
}
```
This TestProperty is also filled into the TestPlatform's TestCase object which makes it available for reporting in the various loggers that can be plugged into the TestPlatform.
To provide extension writers with the ability to have strongly typed attributes to achieve what TestProperty above achieves, the proposal is to make TestProperty a non-sealed class allowing classes to extend it like below:
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class CustomTraitPropertyAttribute : TestPropertyAttribute
[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class CustomTraitPropertyAttribute : TestPropertyAttribute
{
/// <summary>
/// Initializes a new instance of the <see cref="CustomTraitPropertyAttribute"/> class.
/// </summary>
/// <param name="traitData">
/// Data associated with trait property
/// </param>
public CustomTraitPropertyAttribute(int traitData)
: base ("CustomTraitProperty", traitData.ToString())
{
/// <summary>
/// Initializes a new instance of the <see cref="CustomTraitPropertyAttribute"/> class.
/// </summary>
/// <param name="traitData">
/// Data associated with trait property
/// </param>
public CustomTraitPropertyAttribute(int traitData)
: base ("CustomTraitProperty", traitData.ToString())
{
}
}
}
```
And test methods would be decorated in a much more convenient form as below:
```csharp
[TestMethod]
[CustomTraitProperty(234)]
public void TestMethod()
{
}
[TestMethod]
[CustomTraitProperty(234)]
public void TestMethod()
{
}
```
Users can then filter tests in VS IDE Test Explorer based on this Test Property. The query string that would filter the test above would look like

53
RFCs/002-Framework-Extensibility-Custom-Assertions.md
Просмотреть файл

@ -16,51 +16,52 @@ Often times, the default set of assertion APIs are not sufficient to satisfy a w
2. Users of custom assertions should be able to acquire and use them with ease.
### Proposed solution
Here is a solution that is both easily pluggable and acquirable:
Here is a solution that is both easily pluggable and acquirable:
The test frameworks Assertion class should be a non-static singleton with a C# Property('That') for accessing the instance:
```csharp
public class Assert{
public static Assert That{
get
{
// ...
}
The test frameworks Assertion class should be a non-static singleton with a C# Property('That') for accessing the instance:
```csharp
public class Assert
{
public static Assert That
{
get
{
// ...
}
}
}
```
Extension writers can then add C# extension methods for the Assertion class like below:
```csharp
public static class SampleAssertExtensions
public static class SampleAssertExtensions
{
public static void IsOfType<T>(this Assert assert, object obj)
{
public static void IsOfType<T>(this Assert assert, object obj)
if(obj is T)
{
if(obj is T)
{
return;
}
throw new AssertFailedException("Type does not match");
return;
}
throw new AssertFailedException("Type does not match");
}
}
```
And consumers of this extension can consume it in their test code with the below simple syntax:
```csharp
using SampleAssertExtensionsNamespace;
public void TestMethod
{
// ...
Assert.That.IsOfType<Dog>(animal);
}
using SampleAssertExtensionsNamespace;
public void TestMethod
{
// ...
Assert.That.IsOfType<Dog>(animal);
}
```
#### Benefits for custom assertion writers
1. Leverages the default C# constructs - No new interfaces/objects to understand and extend.
2. Extensions can be organized under a verb. For instance assertions expecting exceptions can be organized under the Throws verb like
```
```csharp
Assert.That.Throws.InnerException
Assert.That.Throws.SystemException
Assert.That.Throws.ExceptionWithMessage
@ -75,4 +76,4 @@ Assert.That.IsNotNull(animal).And.IsOfType<Cat>(animal)
2. Readable - Using linq type expressions enhances readability.
## Open questions
1. How important are combined asserts in a single Assert statement (Assert.That.Something.And.Something) and how much of this should be available in-box?
1. How important are combined asserts in a single Assert statement (`Assert.That.Something.And.Something`) and how much of this should be available in-box?

443
RFCs/003-Customize-Running-Tests.md
Просмотреть файл

@ -1,221 +1,222 @@
# RFC 003- Framework Extensibility for Custom Test Execution
## Summary
This document deals with how test runs can be customized using the MSTest V2 Framework extensibility.
## Motivation
The default workflow for running tests in MSTest V2 involves creating an instance of a TestClass and invoking a TestMethod in it. There are multiple instances where this workflow is required to be tweaked so specific tests are runnable. Some tests require to be run on a UI Thread, some others need to be parameterized. This requires that the Test Framework provide extensibility points so that test authors have the ability to run their tests differently.
## Detailed Design
The execution flow can broadly be extended at two levels:
1. Test Method level
2. Test Class level
The sections below details how one can customize execution at these two points.
### Test Method level
Customizing test method level execution is simple - Extend the `TestMethodAttribute`. The `TestMethodAttribute` has the following signature:
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestMethodAttribute : Attribute
{
/// <summary>
/// Executes a test method.
/// </summary>
/// <param name="testMethod">The test method to execute.</param>
/// <returns>An array of TestResult objects that represent the outcome(s) of the test.</returns>
/// <remarks>Extensions can override this method to customize running a test method.</remarks>
public virtual TestResult[] Execute(ITestMethod testMethod) { }
}
```
Extension writers would only need to override the `Execute` method to gain control on how a test is run. The `ITestMethod` instance allows one to get more context of the method under execution. The test method can be executed using `ITestMethod.Invoke()` or by just calling `base.Execute()` on the TestMethodAttribute to go back through the default flow of the Framework.
```csharp
/// <summary>
/// TestMethod for execution.
/// </summary>
public interface ITestMethod
{
/// <summary>
/// Gets the name of test method.
/// </summary>
string TestMethodName { get; }
/// <summary>
/// Gets the name of test class.
/// </summary>
string TestClassName { get; }
/// <summary>
/// Gets the return type of test method.
/// </summary>
Type ReturnType { get; }
/// <summary>
/// Gets the parameters of test method.
/// </summary>
ParameterInfo[] ParameterTypes { get; }
/// <summary>
/// Gets the methodInfo for test method.
/// </summary>
/// <remarks>
/// This is just to retrieve additional information about the method.
/// Do not directly invoke the method using MethodInfo. Use ITestMethod.Invoke instead.
/// </remarks>
MethodInfo MethodInfo { get; }
/// <summary>
/// Invokes the test method.
/// </summary>
/// <param name="arguments">
/// Arguments to pass to test method. (E.g. For data driven)
/// </param>
/// <returns>
/// Result of test method invocation.
/// </returns>
/// <remarks>
/// This call handles asynchronous test methods as well.
/// </remarks>
TestResult Invoke(object[] arguments);
/// <summary>
/// Get all attributes of the test method.
/// </summary>
/// <param name="inherit">
/// Whether attribute defined in parent class is valid.
/// </param>
/// <returns>
/// All attributes.
/// </returns>
Attribute[] GetAllAttributes(bool inherit);
/// <summary>
/// Get attribute of specific type.
/// </summary>
/// <typeparam name="AttributeType"> System.Attribute type. </typeparam>
/// <param name="inherit">
/// Whether attribute defined in parent class is valid.
/// </param>
/// <returns>
/// The attributes of the specified type.
/// </returns>
AttributeType[] GetAttributes<AttributeType>(bool inherit)
where AttributeType : Attribute;
}
```
From a test authors perspective, the test method would now be adorned with the type that extends `TestMethodAttribute` to light up the extended functionality.
Let us take a very simple example to apply this extensibility on - the task is to validate the stability of a test scenario, that is ensure that the test for that scenario passes always when run 'n' number of times.
We start by declaring an `IterativeTestMethodAttribute` that extends `TestMethodAttribute`. We then override `TestMethodAttribute.Execute()` to run the test 'n' number of times.
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class IterativeTestMethodAttribute : TestMethodAttribute
{
private int stabilityThreshold;
public IterativeTestMethodAttribute(int stabilityThreshold)
{
this.stabilityThreshold = stabilityThreshold;
}
public override TestResult[] Execute(ITestMethod testMethod)
{
var results = new List<TestResult>();
for(int count = 0; count < this.stabilityThreshold; count++)
{
var currentResults = base.Execute(testMethod);
results.AddRange(currentResults);
}
return results.ToArray();
}
}
```
From a test authors perspective, the test method would now be adorned with a `IterativeTestMethodAttribute` instead.
```csharp
[TestClass]
public class LongRunningScenarios()
{
[IterativeTestMethod(5)]
public void LongRunningTest()
{
}
}
```
### Test Class level
Scaling up the test method level extensibility gets one to a position of customizing execution of all test methods under a unit, which in this case is a TestClass. One can do so by extending the `TestClassAttribute`.
```csharp
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class TestClassAttribute : Attribute
{
/// <summary>
/// Gets a test method attribute that enables running this test.
/// </summary>
/// <param name="testMethodAttribute">The test method attribute instance defined on this method.</param>
/// <returns>The <see cref="TestMethodAttribute"/> to be used to run this test.</returns>
/// <remarks>Extensions can override this method to customize how all methods in a class are run.</remarks>
public virtual TestMethodAttribute GetTestMethodAttribute(TestMethodAttribute testMethodAttribute) { }
}
```
Overriding `GetTestMethodAttribute()` allows extensions to provide a custom `TestMethodAttribute` that specifies how a specific method is run as detailed in the Test Method level extensibility section above.
From a test authors perspective, the test class would now be adorned with the type that extends `TestClassAttribute` to light up the extended functionality.
To explain this better, lets go back to the example of running a test method 'n' number of times to determine the stability of a scenario. The task now is scaled up to ensure all test methods in a unit are stable.
We start by declaring an `IterativeTestClassAttribute` that extends `TestClassAttribute`. We then extend `GetTestMethodAttribute()` to return an `IterativeTestMethodAttribute`.
```csharp
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class IterativeTestClassAttribute : TestClassAttribute
{
private int stabilityThreshold;
public IterativeTestClassAttribute(int stabilityThreshold)
{
this.stabilityThreshold = stabilityThreshold;
}
public override TestMethodAttribute GetTestMethodAttribute(TestMethodAttribute testMethodAttribute)
{
if (testMethodAttribute is IterativeTestMethodAttribute) return testMethodAttribute;
return new IterativeTestMethodAttribute(this.stabilityThreshold);
}
}
```
The Test Method level extensibility workflow then kicks in when running all test methods in the class ensuring that each method is run 'n' number of times. A point to note from the code sample is that one can have a method level value for 'n' that overrides the class level value. This is possible because the `GetTestMethodAttribute` conditionally returns a new `IterativeTestMethodAttribute` only if the attribute is not already of that type. So if a method is already adorned with an `IterativeTestMethodAttribute` then the stabilityThreshold on the method take precedence over the class. Thus, one can choose how each individual method in the unit is executed.
From a test authors perspective, the test class would now be adorned with a `IterativeTestClassAttribute` instead.
```csharp
[IterativeTestClass(10)]
public class LongRunningScenarios()
{
[TestMethod]
public void TestConnection()
{
}
[IterativeTestMethod(5)]
public void LongRunningTest()
{
}
}
```
## Open questions
1. There can only be one extension that is in control of the execution flow in this model. Should this change to allow the execution flow through multiple extensions? How would that look like?
2. Would a similar model work for extensions that want to hook into Initialize/Cleanup functionality?
# RFC 003- Framework Extensibility for Custom Test Execution
## Summary
This document deals with how test runs can be customized using the MSTest V2 Framework extensibility.
## Motivation
The default workflow for running tests in MSTest V2 involves creating an instance of a TestClass and invoking a TestMethod in it. There are multiple instances where this workflow is required to be tweaked so specific tests are runnable. Some tests require to be run on a UI Thread, some others need to be parameterized. This requires that the Test Framework provide extensibility points so that test authors have the ability to run their tests differently.
## Detailed Design
The execution flow can broadly be extended at two levels:
1. Test Method level
2. Test Class level
The sections below details how one can customize execution at these two points.
### Test Method level
Customizing test method level execution is simple - Extend the `TestMethodAttribute`. The `TestMethodAttribute` has the following signature:
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestMethodAttribute : Attribute
{
/// <summary>
/// Executes a test method.
/// </summary>
/// <param name="testMethod">The test method to execute.</param>
/// <returns>An array of TestResult objects that represent the outcome(s) of the test.</returns>
/// <remarks>Extensions can override this method to customize running a test method.</remarks>
public virtual TestResult[] Execute(ITestMethod testMethod) { }
}
```
Extension writers would only need to override the `Execute` method to gain control on how a test is run. The `ITestMethod` instance allows one to get more context of the method under execution. The test method can be executed using `ITestMethod.Invoke()` or by just calling `base.Execute()` on the TestMethodAttribute to go back through the default flow of the Framework.
```csharp
/// <summary>
/// TestMethod for execution.
/// </summary>
public interface ITestMethod
{
/// <summary>
/// Gets the name of test method.
/// </summary>
string TestMethodName { get; }
/// <summary>
/// Gets the name of test class.
/// </summary>
string TestClassName { get; }
/// <summary>
/// Gets the return type of test method.
/// </summary>
Type ReturnType { get; }
/// <summary>
/// Gets the parameters of test method.
/// </summary>
ParameterInfo[] ParameterTypes { get; }
/// <summary>
/// Gets the methodInfo for test method.
/// </summary>
/// <remarks>
/// This is just to retrieve additional information about the method.
/// Do not directly invoke the method using MethodInfo. Use ITestMethod.Invoke instead.
/// </remarks>
MethodInfo MethodInfo { get; }
/// <summary>
/// Invokes the test method.
/// </summary>
/// <param name="arguments">
/// Arguments to pass to test method. (E.g. For data driven)
/// </param>
/// <returns>
/// Result of test method invocation.
/// </returns>
/// <remarks>
/// This call handles asynchronous test methods as well.
/// </remarks>
TestResult Invoke(object[] arguments);
/// <summary>
/// Get all attributes of the test method.
/// </summary>
/// <param name="inherit">
/// Whether attribute defined in parent class is valid.
/// </param>
/// <returns>
/// All attributes.
/// </returns>
Attribute[] GetAllAttributes(bool inherit);
/// <summary>
/// Get attribute of specific type.
/// </summary>
/// <typeparam name="AttributeType"> System.Attribute type. </typeparam>
/// <param name="inherit">
/// Whether attribute defined in parent class is valid.
/// </param>
/// <returns>
/// The attributes of the specified type.
/// </returns>
AttributeType[] GetAttributes<AttributeType>(bool inherit)
where AttributeType : Attribute;
}
```
From a test authors perspective, the test method would now be adorned with the type that extends `TestMethodAttribute` to light up the extended functionality.
Let us take a very simple example to apply this extensibility on - the task is to validate the stability of a test scenario, that is ensure that the test for that scenario passes always when run 'n' number of times.
We start by declaring an `IterativeTestMethodAttribute` that extends `TestMethodAttribute`. We then override `TestMethodAttribute.Execute()` to run the test 'n' number of times.
```csharp
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class IterativeTestMethodAttribute : TestMethodAttribute
{
private int stabilityThreshold;
public IterativeTestMethodAttribute(int stabilityThreshold)
{
this.stabilityThreshold = stabilityThreshold;
}
public override TestResult[] Execute(ITestMethod testMethod)
{
var results = new List<TestResult>();
for(int count = 0; count < this.stabilityThreshold; count++)
{
var currentResults = base.Execute(testMethod);
results.AddRange(currentResults);
}
return results.ToArray();
}
}
```
From a test authors perspective, the test method would now be adorned with a `IterativeTestMethodAttribute` instead.
```csharp
[TestClass]
public class LongRunningScenarios()
{
[IterativeTestMethod(5)]
public void LongRunningTest()
{
}
}
```
### Test Class level
Scaling up the test method level extensibility gets one to a position of customizing execution of all test methods under a unit, which in this case is a TestClass. One can do so by extending the `TestClassAttribute`.
```csharp
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class TestClassAttribute : Attribute
{
/// <summary>
/// Gets a test method attribute that enables running this test.
/// </summary>
/// <param name="testMethodAttribute">The test method attribute instance defined on this method.</param>
/// <returns>The <see cref="TestMethodAttribute"/> to be used to run this test.</returns>
/// <remarks>Extensions can override this method to customize how all methods in a class are run.</remarks>
public virtual TestMethodAttribute GetTestMethodAttribute(TestMethodAttribute testMethodAttribute) { }
}
```
Overriding `GetTestMethodAttribute()` allows extensions to provide a custom `TestMethodAttribute` that specifies how a specific method is run as detailed in the Test Method level extensibility section above.
From a test authors perspective, the test class would now be adorned with the type that extends `TestClassAttribute` to light up the extended functionality.
To explain this better, lets go back to the example of running a test method 'n' number of times to determine the stability of a scenario. The task now is scaled up to ensure all test methods in a unit are stable.
We start by declaring an `IterativeTestClassAttribute` that extends `TestClassAttribute`. We then extend `GetTestMethodAttribute()` to return an `IterativeTestMethodAttribute`.
```csharp
[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class IterativeTestClassAttribute : TestClassAttribute
{
private int stabilityThreshold;
public IterativeTestClassAttribute(int stabilityThreshold)
{
this.stabilityThreshold = stabilityThreshold;
}
public override TestMethodAttribute GetTestMethodAttribute(TestMethodAttribute testMethodAttribute)
{
if (testMethodAttribute is IterativeTestMethodAttribute)
return testMethodAttribute;
return new IterativeTestMethodAttribute(this.stabilityThreshold);
}
}
```
The Test Method level extensibility workflow then kicks in when running all test methods in the class ensuring that each method is run 'n' number of times. A point to note from the code sample is that one can have a method level value for 'n' that overrides the class level value. This is possible because the `GetTestMethodAttribute` conditionally returns a new `IterativeTestMethodAttribute` only if the attribute is not already of that type. So if a method is already adorned with an `IterativeTestMethodAttribute` then the stabilityThreshold on the method take precedence over the class. Thus, one can choose how each individual method in the unit is executed.
From a test authors perspective, the test class would now be adorned with a `IterativeTestClassAttribute` instead.
```csharp
[IterativeTestClass(10)]
public class LongRunningScenarios()
{
[TestMethod]
public void TestConnection()
{
}
[IterativeTestMethod(5)]
public void LongRunningTest()
{
}
}
```
## Open questions
1. There can only be one extension that is in control of the execution flow in this model. Should this change to allow the execution flow through multiple extensions? How would that look like?
2. Would a similar model work for extensions that want to hook into Initialize/Cleanup functionality?

18
RFCs/004-In-Assembly-Parallel-Execution.md
Просмотреть файл

@ -45,7 +45,7 @@ The value for the number of worker threads to spawn to execute tests can be set
An assembly/Class/Method can explicitly opt-out of parallelization using an attribute that will indicate that it may not be run in parallel with any other tests. The attribute does not take any arguments, and may be added at the method, class, or assembly level.
```csharp
[DoNotParallelize()]
[DoNotParallelize]
```
When used at the assembly level, all tests within the assembly will be executed serially.
When used at the Class level, all tests within the class will be executed serially after the parallel execution of all other tests is completed.
@ -72,29 +72,31 @@ In-assembly parallel execution can be conditioned using the following means:
2. as configuration properties set via a .runsettings file [[see here for more]](https://github.com/Microsoft/vstest-docs/blob/master/docs/configure.md).
3. by passing runsettings arguments via the command line [[see here for more]](https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md).
(3) overrides (2) which in turn overrides (1). The ```[DoNotParallelize()]``` annotation may be applied only to source code, and hence remains unaffected by these rules - thus, even if in-assembly parallel execution in conditioned via (2) or (3), specific program elements can still opt-out safely.
(3) overrides (2) which in turn overrides (1). The ```[DoNotParallelize]``` annotation may be applied only to source code, and hence remains unaffected by these rules - thus, even if in-assembly parallel execution in conditioned via (2) or (3), specific program elements can still opt-out safely.
### Example
Consider an assembly UTA1.dll that has a 2 test classes TC1 and TC2 as follows:
```
```csharp
[assembly: Parallelize(Workers = 3, Scope = ExecutionScope.ClassLevel)]
...
//...
[TestClass]
public class TC1
{
...
//...
}
[TestClass]
[DoNotParallelize()] // this test class is opting out
[DoNotParallelize] // this test class is opting out
public class TC2
{
...
//...
}
```
Furthermore, consider the following test.runsettings file:
```
```xml
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- MSTest adapter -->

85
RFCs/005-Framework-Extensibility-Custom-DataSource.md
Просмотреть файл

@ -17,75 +17,76 @@ Here is a solution for using custom data source in data driven tests.
The test framework should define an interface class `ITestDataSource` which can be extended to get data from custom data source.
```csharp
public interface ITestDataSource
{
/// <summary>
/// Gets the test data from custom data source.
/// </summary>
IEnumerable<object[]> GetData(MethodInfo methodInfo);
public interface ITestDataSource
{
/// <summary>
/// Gets the test data from custom data source.
/// </summary>
IEnumerable<object[]> GetData(MethodInfo methodInfo);
/// <summary>
/// Display name to be displayed for test corresponding to data row.
/// </summary>
string GetDisplayName(MethodInfo methodInfo, object[] data);
}
/// <summary>
/// Display name to be displayed for test corresponding to data row.
/// </summary>
string GetDisplayName(MethodInfo methodInfo, object[] data);
}
```
Here is how the test methods are decorated with concrete implementation of `ITestDataSource`:
```csharp
public class CustomTestDataSourceAttribute : Attribute, ITestDataSource
public class CustomTestDataSourceAttribute : Attribute, ITestDataSource
{
public IEnumerable<object[]> GetData(MethodInfo methodInfo)
{
public IEnumerable<object[]> GetData(MethodInfo methodInfo)
return new[]
{
return new[]
{ new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
}
public string GetDisplayName(MethodInfo methodInfo, object[] data)
{
if (data != null)
{
return string.Format(CultureInfo.CurrentCulture, "{0} ({1})", methodInfo.Name, string.Join(",", data));
}
public string GetDisplayName(MethodInfo methodInfo, object[] data)
{
if (data != null)
{
return string.Format(CultureInfo.CurrentCulture, "{0} ({1})", methodInfo.Name, string.Join(",", data));
}
return null;
}
}
return null;
}
}
```
```csharp
[TestMethod]
[CustomTestDataSource]
public void TestMethod1(int a, int b, int c)
{
Assert.AreEqual(1, a%3);
Assert.AreEqual(2, b%3);
Assert.AreEqual(0, c%3);
}
[TestMethod]
[CustomTestDataSource]
public void TestMethod1(int a, int b, int c)
{
Assert.AreEqual(1, a % 3);
Assert.AreEqual(2, b % 3);
Assert.AreEqual(0, c % 3);
}
```
In a similar way, multiple test methods can be decorated with same data source.
A test method can also be decorated with multiple data sources.
Users can customize the display name of tests in test results by overriding `GetDisplayName()` method.
```csharp
public override string GetDisplayName(MethodInfo methodInfo, object[] data)
{
return string.Format(CultureInfo.CurrentCulture, "MyFavMSTestV2Test ({0})", string.Join(",", data));
}
public override string GetDisplayName(MethodInfo methodInfo, object[] data)
{
return string.Format(CultureInfo.CurrentCulture, "MyFavMSTestV2Test ({0})", string.Join(",", data));
}
```
The display name of tests in the above example would appear as :
```csharp
```
MyFavMSTestV2Test (1,2,3)
MyFavMSTestV2Test (4,5,6)
```
### Discovery of `ITestDataSource` attributes
The MSTest v2 framework, on discovering a `TestMethod` probes additional attributes. On finding attributes inheriting from `ITestDataSource`, framework invokes `GetData()` to fetch test data and iteratively invokes test method with the test data as arguments.
The MSTest v2 framework, on discovering a `TestMethod`, probes additional attributes. On finding attributes inheriting from `ITestDataSource`, the framework invokes `GetData()` to fetch test data and iteratively invokes the test method with the test data as arguments.
### Benefits of using `ITestDataSource`
1. Users can extend `ITestDataSource` to support custom data sources.
2. Multiple tests can reuse the test data defined in same data source.
2. Multiple tests can reuse the test data defined in the same data source.
3. A test case can use multiple test data sources.

52
RFCs/006-DynamicData-Attribute.md
Просмотреть файл

@ -20,22 +20,24 @@ A static property or a static method having test data should be declared as belo
public class UnitTests
{
static IEnumerable<object[]> ReusableTestDataProperty
{
get
{
get
{
return new[]
{ new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
}
return new[]
{
new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
}
}
static IEnumerable<object[]> ReusableTestDataMethod()
{
return new[]
{ new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
{
new object[] {1, 2, 3},
new object[] {4, 5, 6}
};
}
// Property ReusableTestDataProperty can be used as data source for test data with data driven test case.
@ -43,9 +45,9 @@ public class UnitTests
[DynamicData("ReusableTestDataProperty")]
public void DynamicDataTestMethod1(int a, int b, int c)
{
Assert.AreEqual(1, a%3);
Assert.AreEqual(2, b%3);
Assert.AreEqual(0, c%3);
Assert.AreEqual(1, a % 3);
Assert.AreEqual(2, b % 3);
Assert.AreEqual(0, c % 3);
}
// Method ReusableTestDataMethod can be used as data source for test data with data driven test case.
@ -53,9 +55,9 @@ public class UnitTests
[DynamicData("ReusableTestDataMethod", DynamicDataSourceType.Method)]
public void DynamicDataTestMethod2(int a, int b, int c)
{
Assert.AreEqual(1, a%3);
Assert.AreEqual(2, b%3);
Assert.AreEqual(0, c%3);
Assert.AreEqual(1, a % 3);
Assert.AreEqual(2, b % 3);
Assert.AreEqual(0, c % 3);
}
}
```
@ -63,9 +65,9 @@ public class UnitTests
In case, the property or method exists in a class other than the test class, an additional `Type` argument should be passed to `DynamicData` constructor.
```csharp
[DynamicData("ReusableTestDataProperty", typeOf(UnitTests))]
[DynamicData("ReusableTestDataProperty", typeOf(UnitTests))]
[DynamicData("ReusableTestDataMethod", typeOf(UnitTests), DynamicDataSourceType.Method)]
[DynamicData("ReusableTestDataMethod", typeOf(UnitTests), DynamicDataSourceType.Method)]
```
Please note that Enum `DynamicDataSourceType` is used to specify whether test data source is a property or method.
Data source is considered as property by default.
@ -73,19 +75,19 @@ Data source is considered as property by default.
Optionally, to provide a custom name for each data driven test case, `DynamicDataDisplayName` can be used to reference a public static method declared as below:
```csharp
public static string GetCustomDynamicDataDisplayName(MethodInfo methodInfo, object[] data)
{
return string.Format("DynamicDataTestMethod {0} with {1} parameters", methodInfo.Name, data.Length);
}
public static string GetCustomDynamicDataDisplayName(MethodInfo methodInfo, object[] data)
{
return string.Format("DynamicDataTestMethod {0} with {1} parameters", methodInfo.Name, data.Length);
}
// Method GetCustomDynamicDataDisplayName can be used to provide a custom test name for test data with data driven test case.
[DynamicData("ReusableTestDataProperty", DynamicDataDisplayName = "GetCustomDynamicDataDisplayName")]
// Method GetCustomDynamicDataDisplayName can be used to provide a custom test name for test data with data driven test case.
[DynamicData("ReusableTestDataProperty", DynamicDataDisplayName = "GetCustomDynamicDataDisplayName")]
```
`DynamicDataDisplayNameDeclaringType` should be used in cases where the dynamic data display name method exists in a class other than the test class
```csharp
[DynamicData("ReusableTestDataMethod", DynamicDataDisplayName = "GetCustomDynamicDataDisplayName", DynamicDataDisplayNameDeclaringType = typeOf(UnitTests))]
[DynamicData("ReusableTestDataMethod", DynamicDataDisplayName = "GetCustomDynamicDataDisplayName", DynamicDataDisplayNameDeclaringType = typeOf(UnitTests))]
```
### Benefits of using DynamicData attribute

25
RFCs/007-DataSource-Attribute-VS-ITestDataSource.md
Просмотреть файл

@ -1,7 +1,7 @@
# RFC 007- DataSource Attribute Vs ITestDataSource
## Summary
This details the MSTest V2 framework attribute "DataSource" for data driven tests where test data can be present in an excel file, xml file, sql database or OleDb. You can refer msdn documentation here[https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.datasourceattribute.aspx] for more details.
This details the MSTest V2 framework attribute "DataSource" for data driven tests where test data can be present in an excel file, xml file, sql database or OleDb. You can refer msdn documentation [here](https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.datasourceattribute.aspx) for more details.
## Motivation
At present, there are two codeflows for data-driven tests, one for DataSource Attribute and another for DataRow & DynamicData Attributes. This aims to have one common codeflow for handling data-driven tests.
@ -43,23 +43,23 @@ namespace Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Int
```
There is no change in how DataSource Attribute will be consumed. Test methods can be decorated as they were decorated earlier like this:
```csharp
[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "MyFile.xml", "MyTable", DataAccessMethod.Sequential)]
public void MyTestMethod()
{
var v = testContext.DataRow[0];
Assert.AreEqual(v, "3");
}
[TestMethod]
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "MyFile.xml", "MyTable", DataAccessMethod.Sequential)]
public void MyTestMethod()
{
var v = testContext.DataRow[0];
Assert.AreEqual(v, "3");
}
```
The display name of tests in the above example would appear like they used to be as :
```csharp
```
MyTestMethod (Data Row 0)
MyTestMethod (Data Row 1)
```
### Behaviour Changes in DataSource Attributes
Presently, TestFrameworks's Execute() is called once for a data-driven TestMethod, which in-turn takes care of running test for all DataRows. This will be changed to calling TestFramework's Execute() for each DataRow. i.e. the logic of executing data-driven tests will be moved out from framework to adapter.
### Behaviour Changes in DataSource Attributes
Presently, TestFrameworks's `Execute()` is called once for a data-driven TestMethod, which in-turn takes care of running test for all DataRows. This will be changed to calling TestFramework's `Execute()` for each DataRow. i.e. the logic of executing data-driven tests will be moved out from framework to adapter.
### Differences between DataSource Attribute and ITestDataSource
| DataSource           | ITestDataSource                                       |
@ -68,8 +68,7 @@ Presently, TestFrameworks's Execute() is called once for a data-driven TestMetho
| TestMethod does not require to have parameters    | TestMethod is required to have parameters |
Note :
Test authors should not expect data to be set in TestContext for attributes inheriting from ITestDataSource. Going forward, data should only be consumed from Testmethod parameters for data-driven tests.
Test authors should not expect data to be set in TestContext for attributes inheriting from `ITestDataSource`. Going forward, data should only be consumed from Testmethod parameters for data-driven tests.
### Support Scenarios
Following scenarios will not supported in case of DataSource Attributes :

12
RFCs/008-TestCase-Timeout.md
Просмотреть файл

@ -7,12 +7,12 @@ User should be able to configure global test case timeout for all the test cases
Make test case timeout configurable via TestTimeout tag which is part of the adapter node in the runsettings.
Here is a sample runsettings: 
```
<Runsettings> 
 <MSTestV2> 
<TestTimeout>5000</TestTimeout>   
 </MSTestV2> 
</Runsettings> 
```xml
<Runsettings> 
<MSTestV2> 
<TestTimeout>5000</TestTimeout>   
</MSTestV2> 
</Runsettings> 
```
### Honoring the settings