Update standard spec for FullyQualifiedName (#133)
* Update standard spec for FullyQualifiedName * Be more precise about type encoding. * Incorporate spec feedback * Update RFC to reference properties instead of StdFQN. * More tweaking and wordsmithing. * Add clarification regarding uniqueness. * Update encoding spec for multi-dimensional arrays.
This commit is contained in:
Peter Waldschmidt
pvlakshm
Родитель
f6b2f092ce
Коммит
4a59c5e1eb
|
|
@ -0,0 +1,109 @@
|
|||
# 0017 Properties for TestCases in Managed Code
|
||||
|
||||
## Summary
|
||||
This document standardizes additional properties on TestCase to support test cases written in managed languages.
|
||||
|
||||
## Overview
|
||||
There are a broad variety of choices that current test adapter implementers have made when setting the `FullyQualifiedName` on TestCase objects created during test discovery. `FullyQualifiedName` typically identifies the type and method that implements a test case in managed code, but because there has been no standardization, VS cannot rely on the format of the `FullyQualifiedName`. We've considered standardizing the format of `FullyQualifiedName`, but this presents migration & compatibility costs that outweigh the benefits. Instead, we are adding two new properties to TestCase that represent the type (`ManagedType`) and method (`ManagedMethod`) for each test case. These additional properties are only applicable to unit tests written in managed languages (e.g. C# & VB, et al.).
|
||||
|
||||
## Specification
|
||||
|
||||
TestCases for managed code must include a string-valued property named `ManagedType` property and a string-valued property named `ManagedMethod`. The specification below outlines the requirements for the contents of these two properties.
|
||||
|
||||
**`ManagedType` Property**
|
||||
|
||||
The `ManagedType` test case property represents the fully specified type name in metadata format:
|
||||
|
||||
NamespaceA.NamespaceB.ClassName`1+InnerClass`2
|
||||
|
||||
* The type name must be fully qualified (in the CLR sense), including its namespace. Any generic classes must also include an arity value using backtick notation (`# where # is the number of type arguments that the class requires).
|
||||
* Nested classes are appended with a '+' and must also include an arity if generic.
|
||||
* There must be no whitespace included in the property value.
|
||||
|
||||
**`ManagedMethod` Property**
|
||||
|
||||
The `ManagedMethod` test case property is the fully specified method including the method name and a list of its parameter types inside parentheses separated by commas.
|
||||
|
||||
MethodName`2(ParamTypeA,ParamTypeB,…)
|
||||
|
||||
* If the method accepts no parameters, then the parentheses should be omitted.
|
||||
* If the method is generic, then an arity must be specified (in the same way as the type property).
|
||||
* There must be no whitespace included in the segment
|
||||
* The list of parameter types must be encoded using the type specification below.
|
||||
* Return types are not encoded.
|
||||
|
||||
### Parameter Type Encoding
|
||||
|
||||
Parameters are encoded as a comma-separated list of strings in parentheses at the end of the `ManagedMethod` property. Each parameter is encoded using the rules below.
|
||||
|
||||
* Basic Types - Types should be written using their namespace and type name with no extra whitespace. For example (`NamespaceA.NamespaceB.Class`). Native types should be written using their CLR type names (`System.Int32`,`System.String`).
|
||||
* Array Types - Arrays should be encoded as the element type followed by square brackets (`[]`). Multidimensional arrays are indicated with commas inside the square brackets. There should be one less comma than the number of dimensions (i.e. 2 dimensional array is encoded as `[,]`, 3 dimensional array as `[,,]` ).
|
||||
* Generic Types - Generic types should be encoded as the type name, followed by comma-separated type arguments in angle brackets (`<>`).
|
||||
* Generic Parameters - Parameters that are typed by a generic argument on the containing type are encoded with an exclamation point (`!`) followed by the ordinal index of the parameter in the generic argument list.
|
||||
* Generic Method Parameters - Parameters that are typed by a generic argument on the method are encoded with a double exclamation point (`!!`) followed by the ordinal index of the parameter in the method's generic argument list.
|
||||
* Pointer Types - Pointer types should be encoded as the type name, followed by an asterisk (`*`).
|
||||
* Dynamic Types - Dynamic types should be represented as System.Object.
|
||||
|
||||
#### Examples
|
||||
|
||||
```csharp
|
||||
Method(NamespaceA.NamespaceB.Class) // Custom Types
|
||||
Method(System.String,System.Int32) // Native Types
|
||||
Method(System.String[]) // Array Types
|
||||
Method(List<System.String>) // Generic Types
|
||||
Method(!0) // Generic Type Parameters
|
||||
Method(!!0) // Generic Method Parameters
|
||||
Method(List<!0>) // Generic Type with a Generic Type Parameter
|
||||
```
|
||||
|
||||
### Special Methods
|
||||
|
||||
The CLR has some features that are implemented by way of special methods. While these methods are unlikely to be used to represent tests, the list below indicates how they should be encoded, if necessary.
|
||||
|
||||
* Explicit Interface Implementation - methods that explicitly implement an interface should be prefixed with the interface typename. For example the method name for the explicit implementation of IEnumerable<T>.GetEnumerator would look like this; `System.Collections.Generic.IEnumerable<T>.GetEnumerator`. Note that this is only the `ManagedMethod` property. The `ManagedType` property would still include the namespace of the class on which this method is declared.
|
||||
* Constructors - constructors should be referenced using the method name `.ctor`
|
||||
* Operators - operators are a language specific feature, and are translated into methods using compiler-specific rules. Use the underlying compiler-generated method name to reference the operator. For instance, in C#, `operator+` would be represented by a method named `op_Addition`.
|
||||
* Finalizers - finalizers should be referenced using the name `Finalize`
|
||||
|
||||
### Nested Generic Type Parameter Numbering
|
||||
|
||||
If a generic type is nested in another generic type, then the generic arguments of the containing type are also reflected in the nested type, even if it's not reflected in the language syntax. This means that the numbering of generic parameters take into account the generic parameters of the containing type as if they are declared on the local type.
|
||||
|
||||
```csharp
|
||||
// type A has arity 1, and one generic argument T
|
||||
// ManagedType = "A`1"
|
||||
public class A<T>
|
||||
{
|
||||
// type B has arity 1, but two generic arguments T and X
|
||||
// ManagedType = "A`1+B`1"
|
||||
public class B<X> {
|
||||
|
||||
// ManagedMethod = "Method(!0, !1)"
|
||||
public void Method(T t, X x) {}
|
||||
|
||||
// ManagedMethod = "Method(!1)"
|
||||
public void Method(X x) {}
|
||||
|
||||
// ManagedMethod = "Method(!0, !1, !!0)"
|
||||
public void Method<U>(T t, X x, U u) {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Syntactic vs Semantic Test Location
|
||||
|
||||
We are defining syntactic location as the location in the code/syntax where the method is defined that implements the test. On the other hand, semantic location is the location in the type hierarchy at which point the method becomes recognized as a valid test.
|
||||
|
||||
There can be a situation where a test is declared on an abstract base type (syntactic), but the test is only discovered in a derived class (semantic). The question is which type should the `ManagedType` property point to? The base type, where the test code is located? Or the derived type, where the test is discovered? We will use the semantic location for the type because that is where the test adapter will find the test and it is where the test class would be instantiated. Also, there is a one to many relationship between the two. There can be many semantic test locations that each are implemented by the same base class method. If we were to standardize on using the syntactic location, we would risk having name collisions or fail to find the correct test case since there are multiple possible candidates.
|
||||
|
||||
## Managed Tests that are Not Type/Method Based
|
||||
|
||||
It is possible to create unit tests in managed code where tests are not necessarily based on types & methods. While this is somewhat uncommon, we need to be able to handle this situation gracefully. If a test adapter cannot provide a mapping between a testcase and a managed type and method, then it should not provide those properties. When `ManagedType` and `ManagedMethod` properties are not provided, the end-user may lose access to some features in the Test Explorer such as fast test execution by name, or source linking.
|
||||
|
||||
## Tests Returning Multiple Results
|
||||
|
||||
There are some situations where the number of test cases cannot be determined at discovery time. In these cases, when the tests are executed multiple results are returned for a single discovered test case.
|
||||
|
||||
## Uniqueness
|
||||
|
||||
The combination of `ManagedType` and `ManagedMethod` will be unique to a particular method within an assembly, but are not unique to a test case. This is due to the fact that when tests are data-driven, there can be many test cases that are executed by the same method. The test case ID is expected to be unique for every test case within an assembly. The test case ID should also be deterministic with respect to the method and its arguments. In other words, the test ID should not change given a particular method and a set of argument values (if any).
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
# 0017 Guidelines for setting the FullyQualifiedName on TestCase
|
||||
|
||||
## Summary
|
||||
This note clarifies the guidelines that test adapters should consider when setting the FullyQualifiedName on test cases before returning them from the discovery process.
|
||||
|
||||
## Overview
|
||||
There are a broad variety of choices that current Test Adapter implementers have made when setting the FullyQualifiedName on TestCase objects created during test discovery. This note lays out some basic guidelines to make the usage of FullyQualifiedName more useful across adapters. These guidelines are focused on languages that support namespaces, classes and methods for structuring test cases (e.g. C# & VB). Other languages may not fit these guidelines.
|
||||
|
||||
## Usage of FullyQualifiedName
|
||||
The Test Platform uses FullyQualifiedName as a string moniker to name tests during discovery. Visual Studio may filter tests by FullyQualifiedName in order to more quickly filter and run specific tests. The guidelines below clarify the usage of FullyQualifiedName so that these optimized test execution scenarios can be realized.
|
||||
|
||||
## Guidelines
|
||||
1. **FullyQualifiedName should be unique across your test set.** Internally, TestCase objects are tracked by a unique ID (GUID) which can either be set by the Test Adapter implementor, or automatically generated otherwise. The FullyQualifiedName should be unique if possible, but it's not a requirement.
|
||||
1. **FullyQualifiedName should be deterministic** FullyQualifiedName should have the same value for the same test across different discovery sessions. This means that including random data, time-dependent data or environment specific data in the FullyQualifiedName should be avoided.
|
||||
1. **FullyQualifiedName is expected to be a reference to the method that implements the test** The Test Platform is expecting the FullyQualifiedName to refer to a particular test case. If there are multiple tests that are instantiated from a test method (data-driven tests for instance), then the FullyQualifiedName should distinguish these using parentheses to represent data parameters and angle brackets to represent type parameters. See below.
|
||||
|
||||
## Recommended Form
|
||||
FullyQualifiedName is expected to be a reference to the TestCase method that implements the test (or tests). The FullyQualifiedName should be unique to a particular TestCase instance, if possible.
|
||||
|
||||
The recommended forms of FullyQualifiedName are described by the following examples:
|
||||
|
||||
Namespace.Namespace.Class.Method
|
||||
Namespace.Namespace.Class.Method()
|
||||
Namespace.Namespace.Class<Int32>.Method()
|
||||
Namespace.Namespace.Class<Int32>.Method<String>()
|
||||
Namespace.Namespace.Class+InnerClass.Method
|
||||
Namespace.Namespace.Class.Method(x: 1, msg: "string")
|
||||
Namespace.Namespace.Class.Method(1, "string")
|
||||
Namespace.Namespace.Class(init: "Value").Method(param: 23)
|
||||
Namespace.Namespace.Class<String>("Value").Method(23)
|
||||
|
||||
## Normalized Form
|
||||
When referencing test cases by FullyQualifiedName, the test platform recognizes a normalized form of the FullyQualifiedName. This normalized form is the FullyQualifiedName with segments in parentheses and segments in angle brackets removed. The normalized form will not be unique among TestCases.
|
||||
|
||||
| Fully Qualified Name | Normalized Fully Qualified Name |
|
||||
|---|---|
|
||||
| `Namespace.Namespace.Class.Method()` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class<Int32>.Method()` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class<Int32>.Method<String>()` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class+InnerClass.Method` | `Namespace.Namespace.Class+InnerClass.Method` |
|
||||
| `Namespace.Namespace.Class.Method(x: 1, msg: "string")` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class.Method(1, "string")` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class(init: "Value").Method(param: 23)` | `Namespace.Namespace.Class.Method` |
|
||||
| `Namespace.Namespace.Class<String>("Value").Method(23)` | `Namespace.Namespace.Class.Method` |
|
||||
|
||||
|
||||
The normalized form allows the test platform to reference groups of tests in a more general form without having to interpret the logic that the test adapter uses when constructing the fully qualified name.
|
||||
|
||||
NOTE: Normalization is currently only used for languages that support source-based test discovery. At the time of this writing, that includes MSTest, NUnit and xUnit tests written in C# or VB.
|
||||
Загрузка…
Ссылка в новой задаче