249 строки
12 KiB
Markdown
249 строки
12 KiB
Markdown
# Implementation Details
|
|
|
|
"How This Whole Crazy Thing Works: An extended apology"
|
|
by Steve Dower
|
|
|
|
|
|
## Intent
|
|
|
|
We are building an extension for Visual Studio that integrates into many (most)
|
|
of its features. We would like to automatically test this extension.
|
|
|
|
In general, automated tests for extensions are most efficient when the platform
|
|
is simulated ("mocked out"). Over time, we aim to mock out as much of VS as we
|
|
can, but integration testing can only reasonably take place within a running VS
|
|
instance.
|
|
|
|
This package (VSTestHost) allows automated tests to be launched from VS or
|
|
MSTest that run in the context of a Visual Studio instance. This enables
|
|
developers to validate the product in its running context.
|
|
|
|
Making all of this work is messy, and this document is meant to help whoever has
|
|
to touch it next.
|
|
|
|
|
|
## Requirements
|
|
|
|
* Write and run tests using the normal VS unit testing framework
|
|
* Execute tests in a selectable VS instance (SKU, version, hive, etc.)
|
|
* Debug tests by selecting "Debug selected tests" within VS
|
|
* Resilient to VS crashes during test runs
|
|
|
|
|
|
## Architecture
|
|
|
|
There are three executable processes involved when running these tests. Within
|
|
VSTestHost, they are referred to as the TESTER, the TESTEE, and the EXECUTION
|
|
ENGINE. Another concept is the TEST ADAPTER, which is a .NET class.
|
|
|
|
The TESTER is the instance of Visual Studio that the developer is using to
|
|
launch tests. If the developer is debugging tests, the TESTER is the VS instance
|
|
that will attach to the other instance.
|
|
|
|
> When running tests from the command line, there is no TESTER.
|
|
|
|
The TESTEE is the instance of Visual Studio where the test will actually run.
|
|
The testsettings file selected by the TESTER will determine which version, SKU,
|
|
and hive of Visual Studio will be started as the TESTEE.
|
|
|
|
> The TESTEE is launched and terminated by the TEST ADAPTER from the EXECUTION
|
|
ENGINE
|
|
|
|
The EXECUTION ENGINE is a separate process that allows unit tests to run with
|
|
different process-wide settings to what the TESTER was launched with. When tests
|
|
are launched from the TESTER, it will start the EXECUTION ENGINE with a request
|
|
to start running the selected tests.
|
|
- The EXECUTION ENGINE is not aware of the TESTER (which may not exist)
|
|
|
|
The TEST ADAPTER is a .NET class that is loaded in the EXECUTION ENGINE and
|
|
controls the test execution sequence. The TEST ADAPTER is initialized for a run
|
|
and is notified when the run is paused, resumed, stopped, or aborted. The TEST
|
|
ADAPTER is passed each test in turn to be executed and the result passed back
|
|
via a result sink.
|
|
|
|
VSTestHost includes two TEST ADAPTERs to support the IPC required to run tests
|
|
in the TESTEE. The TEST ADAPTER loaded in the EXECUTION ENGINE is responsible
|
|
for marshalling calls (including error handling) to the TESTEE's TEST ADAPTER,
|
|
which is responsible for executing the test.
|
|
|
|
> Neither TEST ADAPTER is the "real" unit test adapter, so executing a test
|
|
looks more like instantiating another ITestAdapter and invoking its Run().
|
|
|
|
|
|
## Test Settings
|
|
|
|
VSTestHost depends on the use of a testsettings file or TestProperty attributes
|
|
to specify the SKU and hive of VS to use as the TESTEE. Setting the version is
|
|
possible but not recommended - by default, the same version will be used for
|
|
the TESTEE as the TESTER, which is most stable. The available settings are as
|
|
follows:
|
|
|
|
| Setting | Description |
|
|
| --- | --- |
|
|
| VSApplication | The registry key name, like "VisualStudio" or "WDExpress" |
|
|
| VSExecutable | The executable name, like "devenv" or "wdexpress" |
|
|
| VSVersion | The version number, like "12.0" or "14.0" |
|
|
| VSHive | The hive name, like "Exp" or "Default" |
|
|
| VSLaunchTimeoutInSeconds [opt] | The number of seconds to wait for launch |
|
|
| VSDebugMixedMode | True to use mixed-mode debugging for tests |
|
|
| VSReuseInstance [opt] | False to disable the reuse of the same VS instance for distinct tests |
|
|
| ScreenCapture [opt] | Relative path to capture screenshots to |
|
|
| ScreenCaptureInterval [opt] | Milliseconds between screenshots |
|
|
|
|
Note that identical screenshots are not saved.
|
|
|
|
## Run Test Sequence
|
|
|
|
Because a TESTEE instance has no way of knowing whether it is actually a TESTEE
|
|
or just an instance of VS with VSTestHost installed, all instances will open an
|
|
IPC server and wait a short period of time for incoming connections. This server
|
|
channel is uniquely named for the process and will not collide with other VS
|
|
instances. The TESTEE's TEST ADAPTER is made available over this IPC channel.
|
|
|
|
The typical execution sequence looks like this:
|
|
|
|
1. Developer clicks "Run selected test" in the TESTER
|
|
2. TESTER launches EXECUTION ENGINE
|
|
3. EXECUTION ENGINE loads the TEST ADAPTER
|
|
4. EXECUTION ENGINE calls Initialize() on the TEST ADAPTER
|
|
5. EXECUTION ENGINE's TEST ADAPTER reads the configuration and launches the TESTEE
|
|
6. TESTEE opens an IPC channel that is unique to the process and publishes its own TEST ADAPTER
|
|
7. EXECUTION ENGINE's TEST ADAPTER connects to the TESTEE's TEST ADAPTER and returns from Initialize().
|
|
8. EXECUTION ENGINE calls its TEST ADAPTER's Run() method for each test.
|
|
9. EXECUTION ENGINE's TEST ADAPTER calls TESTEE's TEST ADAPTER's Run() method
|
|
10. TESTEE's TEST ADAPTER instantiates/caches the real unit test adapter
|
|
11. TESTEE's TEST ADAPTER invokes the unit test adapter's Run() method
|
|
12. Repeat 8-11 for each test.
|
|
13. EXECUTION ENGINE calls its TEST ADAPTER's PreTestRunFinished() and Cleanup() methods
|
|
14. EXECUTION ENGINE's TEST ADAPTER exits the TESTEE.
|
|
15. Test run is complete.
|
|
|
|
|
|
## Debug Test Sequence
|
|
|
|
Debugging is more complicated, because the TESTER needs to perform debug/attach
|
|
but only the EXECUTION ENGINE knows the TESTEE's process ID. However, the
|
|
EXECUTION ENGINE does not know the TESTER's process ID, and so cannot send a
|
|
message directly to the TESTER. Instead, we use a global IPC server that is only
|
|
active for a period of time after the TESTER begins debugging.
|
|
|
|
Reliably detecting when a debugging session is for a unit test requires extra
|
|
dependencies for the package loaded in the TESTER, which may prevent the
|
|
assembly from loading or being useful in as many VS configurations as we would
|
|
like. With our current architecture, we have a single DLL that is installed into
|
|
the GAC and is loaded in all instances of VS, whether they will be the TESTER,
|
|
the TESTEE, or are not taking part in unit tests at all. An alternate
|
|
implementation would require three assemblies with two VSPackages and at least
|
|
one vsixmanifest for the TESTER's package, as well as multiple dependencies
|
|
between these assemblies and other dependencies that lead to a complicated and
|
|
error-prone deployment process.
|
|
|
|
Rather than dealing with this, we will open the debug IPC channel each time any
|
|
VS instance starts debugging. If the channel is already open, the new instance
|
|
will connect to it and signal it to terminate, so that the most recent debugging
|
|
session is the one listening. This results in a potential race if the user
|
|
starts multiple debugging sessions where at least one is supposed to attach to
|
|
a TESTEE, but since this race is with a human, it is considered unlikely. If the
|
|
EXECUTION ENGINE needs to attach but is unable to find an IPC server, it will
|
|
abort the run and no tests are executed. The workaround is to restart debugging.
|
|
There will also be more IPC channels opened than necessary, since we will open
|
|
one for every debugging session unless we know the instance is a TESTEE. These
|
|
are not much more expensive than creating a named pipe, so it is considered a
|
|
worthwhile cost.
|
|
|
|
In short, debugging works fine unless you try to break it (for example, by
|
|
starting to debug a test then quickly starting a separate debugging session in
|
|
another VS instance).
|
|
|
|
The typical debug execution sequence looks like this:
|
|
(Steps prefixed with an asterisk are different from the Run Test sequence.)
|
|
|
|
1. *Developer clicks "Debug selected test" in the TESTER
|
|
2. *TESTER launches EXECUTION ENGINE under its managed debugger.
|
|
3. *TESTER opens the debug IPC server. If another process has opened the server already,
|
|
then TESTER tells it to close.
|
|
4. EXECUTION ENGINE loads the TEST ADAPTER
|
|
5. EXECUTION ENGINE calls Initialize() on the TEST ADAPTER
|
|
6. EXECUTION ENGINE's TEST ADAPTER reads the configuration and launches the TESTEE
|
|
7. TESTEE opens an IPC channel that is unique to the process and publishes its own TEST ADAPTER
|
|
8. *EXECUTION ENGINE's TEST ADAPTER connects to the TESTEE's TEST ADAPTER
|
|
9. *EXECUTION ENGINE's TEST ADAPTER connects to the debug IPC server and tells it to attach to TESTEE.
|
|
10. *EXECUTION ENGINE's TEST ADAPTER returns from Initialize().
|
|
11. *TESTER attaches its debugger to TESTEE.
|
|
12. EXECUTION ENGINE calls its TEST ADAPTER's Run() method for each test.
|
|
13. EXECUTION ENGINE's TEST ADAPTER calls TESTEE's TEST ADAPTER's Run() method
|
|
14. TESTEE's TEST ADAPTER instantiates/caches the real unit test adapter
|
|
15. TESTEE's TEST ADAPTER invokes the unit test adapter's Run() method
|
|
16. Repeat 12-15 for each test.
|
|
17. EXECUTION ENGINE calls its TEST ADAPTER's PreTestRunFinished() and Cleanup() methods
|
|
18. EXECUTION ENGINE's TEST ADAPTER exits the TESTEE.
|
|
19. Test run is complete.
|
|
|
|
|
|
## VSTestHost Classes
|
|
|
|
This is an overview of the classes within VSTestHost and how they are used. Some
|
|
classes are used in multiple places with slightly different purposes, and
|
|
because not every VS SKU or version can support being a TESTER, some parts are
|
|
#if'd out to allow separate builds for those. Because of the reference to
|
|
Microsoft.VisualStudio.Shell.##.0, each VS version requires its own build of
|
|
VSTestHost.
|
|
|
|
### `VSTestHostPackage`
|
|
|
|
This package is auto-loaded in every instance of VS.
|
|
|
|
For TESTEEs (bearing in mind that we always initially assume this), package
|
|
initialization creates the IPC channel and publishes its test adapter. This
|
|
channel is not closed until the package is disposed at process exit.
|
|
|
|
For TESTERs (again, this is always assumed initially), the debugging state is
|
|
hooked so the debug IPC channel can be activated/deactivated as necessary.
|
|
Channel creation is handled by the static members of TesterDebugAttacher. When
|
|
the timeout expires, another TESTER manually aborts the wait, or an EXECUTION
|
|
ENGINE signals to attach, the channel is closed.
|
|
|
|
### `TesterDebugAttacher`
|
|
|
|
This class is published over the debug IPC channel to allow the EXECUTION ENGINE
|
|
to instruct the debugger in the TESTER to attach to the TESTEE. Other TESTERs
|
|
may also use it to abort listeners so they can take over the channel.
|
|
|
|
The class has static methods for use by the TESTER to simplify listening for
|
|
commands.
|
|
|
|
### `TesterTestAdapter`
|
|
|
|
This class is loaded in the EXECUTION ENGINE to run tests that have a
|
|
[HostType("VSTestHost")] attribute. It is largely responsible for connecting to
|
|
an instance of TesteeTestAdapter and marshalling calls to the TESTEE.
|
|
|
|
### `TesteeTestAdapter`
|
|
|
|
This class is published from the TESTEE on the unique IPC channel created by
|
|
VSTestHostPackage. It implements the logic for running arbitrary tests within VS
|
|
by loading the intended test adapter.
|
|
|
|
### `VisualStudio`
|
|
|
|
This class encapsulates the logic for starting VS and getting its DTE object
|
|
from the EXECUTION ENGINE. Currently, DTE is only used by the EXECUTION ENGINE
|
|
to safely exit VS at the end of a test run, but in the future could be used for
|
|
tests that need to drive VS from outside the TESTEE process.
|
|
|
|
### `VSTestContext`
|
|
|
|
This class is used by tests running within the TESTEE to easily access the VS
|
|
instance's DTE and global ServiceProvider objects. Unit test projects should
|
|
include the following reference for VSTestHost:
|
|
|
|
```xml
|
|
<Reference Include="Microsoft.VisualStudioTools.VSTestHost.$(VSTarget)" />
|
|
```
|
|
|
|
Test projects that do not refer to `VSTestContext` do not require this reference.
|
|
|
|
`VSTestContext` is the only public and supported class in the
|
|
**Microsoft.VisualStudioTools.VSTestHost** assembly. All other public classes are
|
|
infrastructure.
|