[Xamarin.Android.NUnitLite] Improve test name formatting on Jenkins (#917)

We've been running on-device NUnit tests for quite some time, and
[Jenkins is nice enough to provide a test report][xa668].

[xa668]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/668/testReport/

There's just one small annoyance: the Package name for on-device tests
is `(root)`:

	# All Tests
	+-----------------------------+-----------+-------+-------+-------+-------+
	| Package                     | Duration  |  Fail |  Skip |  Pass | Total |
	+-----------------------------+-----------+-------+-------+-------+-------+
	| (root)                      | 14 min    |     0 |   814 | 20260 | 21074 |
	| Java.Interop.DynamicTests   | 15 sec    |     0 |     0 |    14 |    14 |
	| ...

The reason the for the `(root)` package -- which constitutes all
on-device tests -- is due to three interractions:

 1. [junit-plugin renders `(root)`][junit-root] when the
    `//testcase/@classname` or `//testcase/@name` attribute does
    *not* contain a `.` (period).

 2. [xunit-plugin's XSLT to convert NUnit2 XML to JUnit XML][xunit-xslt]
    uses `test-suite//results//test-case[1]/@name` to determine the
    assembly name (!).

    (Half of this XSLT is inexplicable to me. Why is line 39 checking
    to see if `test-suite//results//test-case[1]//@name` ends with
    `)` (close parenthesis)?)

 3. `Xamarin.Android.NUnitLite`'s NUnit2 XML output never contains any
    `.`s within `//test-case/@name` values.

    ```xml
    <!-- fragment from TestResult-Mono.Android_Tests.xml -->
    <test-case
        name="ApplicationContextIsApp"
        executed="True"
        result="Success"
        success="True"
        time="0.003"
        asserts="2"
    />
    ```

[junit-root]: https://github.com/jenkinsci/junit-plugin/blob/2c1655c/src/main/java/hudson/tasks/junit/CaseResult.java#L332-L336
[xunit-xslt]: https://github.com/jenkinsci/xunit-plugin/blob/f122ddd/src/main/resources/org/jenkinsci/plugins/xunit/types/nunit-1.0-to-junit-2.xsl

Compare to e.g. the NUnit2 XML output that NUnit 3.7's
`nunit3-console.exe` + `format=nunit2` generates -- as produced by
`make run-nunit-tests`:

```xml
<!-- fragment from TestResult-Java.Interop-Tests.xml -->
<test-case
    name="Java.InteropTests.InvokeVirtualFromConstructorTests.ActivationConstructor"
    executed="True"
    result="Success"
    success="True"
    time="0.139"
    asserts="2"
/>
```

the result of passing `Xamrin.Android.NUnitLite` NUnit2 XML through
xunit-plugin's XSLT is:

```xml
<!-- Fragment from: `xsltproc nunit-1.0-to-junit-2.xsl TestResult-Mono.Android_Tests.xml` -->
<testcase classname="ApplicationTest" name="ApplicationContextIsApp" time="0.003"/>
```

Compare to NUnit 3.7 output:

```xml
<!-- Fragment from: `xsltproc nunit-1.0-to-junit-2.xsl TestResult-Java.Interop-Tests.xml.txt -->
<testcase classname="Java.InteropTests.InvokeVirtualFromConstructorTests" name="ActivationConstructor" time="0.139"/
```

To get rid of the `(root)` package, I see only one simple way to do
so: Update `Xamarin.Android.NUnitLite`s `NUnit2XmlOutputWriter` so
that the `//test-case/@name` value *always* contains a `.`.
Specifically, instead of using `ITestResult.Name`, use
`ITestResult.FullName`.

*However*, some of our existing tests are in "funny" namespaces. For
example, `Mono.Android-Tests.csproj` contains an
`Android.AppTests.ApplicationTest` fixture. Jenkins Test output also
doesn't "care" about assembly or process boundaries; all "packages",
regardless of where they're from, are merged.

Try to improve on that by not just using `ITestResult.FullName` for
the `//test-case/@name` value, but *also* use the value of the
currently executing Android Package, as per
`Android.App.Application.PackageName`, as an "assembly-qualified name"
-like construct.

This will hopefully result in a more useful table:

	# All Tests
	+---------------------------------------+-----------+-------+-------+-------+-------+
	| Package                               | Duration  |  Fail |  Skip |  Pass | Total |
	+---------------------------------------+-----------+-------+-------+-------+-------+
	| Java.Interop.DynamicTests             | 15    sec |     0 |     0 |    14 |    14 |
	| Mono.Android_Tests, Android.AppTests  |  0.18 sec |     0 |     0 |     4 |     4 |
	| ...

Alternatively, maybe this is a sign we should "re-namespace" our
existing NUnit tests to start with the assembly name...
This commit is contained in:
Jonathan Pryor 2017-10-06 16:49:12 -04:00 коммит произвёл Dean Ellis
Родитель a296784fcc
Коммит 23b2642ed0
2 изменённых файлов: 2 добавлений и 2 удалений

Просмотреть файл

@ -351,7 +351,7 @@ namespace Xamarin.Android.NUnitLite
if (suite != null)
return suite;
else {
var ts = new TestSuite ("");
var ts = new TestSuite (global::Android.App.Application.Context.PackageName);
Console.Error.WriteLine (ts.FullName);
foreach (var i in AssemblyLevel.Cast<NUnitTest> ())
ts.Add (i);

Просмотреть файл

@ -216,7 +216,7 @@ namespace NUnitLite.Runner
else
{
xmlWriter.WriteStartElement("test-case");
xmlWriter.WriteAttributeString("name", result.Name);
xmlWriter.WriteAttributeString("name", global::Android.App.Application.Context.PackageName + ", " + result.FullName);
}
if (test.Properties.ContainsKey(PropertyNames.Description))