When determining the color for a collection of tests, first check if the
execution result is identical for all tests in the collection, in which case
just use the corresponding color for a single test.
xharness needs a solution in order to ask for a nuget restore, so make sure to
provide the path to the solution.
This fixes a build issue where the install source tests would fail to build
due to picking up the system's nunit.framework.dll because the nuget one
wasn't found/restored:
MonoPathManglerTest.cs(8,3): error CS0619: 'TestFixtureAttribute' is obsolete: 'The NUnit framework shipped with Mono is deprecated and will be removed in a future release. It was based on NUnit 2.4 which is long outdated. Please move to the NUnit NuGet package or some other form of acquiring NUnit.'
XamarinSourcesPathManglerTest.cs(8,3): error CS0619: 'TestFixtureAttribute' is obsolete: 'The NUnit framework shipped with Mono is deprecated and will be removed in a future release. It was based on NUnit 2.4 which is long outdated. Please move to the NUnit NuGet package or some other form of acquiring NUnit.'
OpenTKManglerTest.cs(8,3): error CS0619: 'TestFixtureAttribute' is obsolete: 'The NUnit framework shipped with Mono is deprecated and will be removed in a future release. It was based on NUnit 2.4 which is long outdated. Please move to the NUnit NuGet package or some other form of acquiring NUnit.'
PathManclerFactoryTests.cs(8,3): error CS0619: 'TestFixtureAttribute' is obsolete: 'The NUnit framework shipped with Mono is deprecated and will be removed in a future release. It was based on NUnit 2.4 which is long outdated. Please move to the NUnit NuGet package or some other form of acquiring NUnit.'
OpenTKManglerTest.cs(29,4): error CS0616: 'TestCase' is not an attribute class
OpenTKManglerTest.cs(30,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(29,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(30,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(31,4): error CS0616: 'TestCase' is not an attribute class
XamarinSourcesPathManglerTest.cs(33,4): error CS0616: 'TestCase' is not an attribute class
XamarinSourcesPathManglerTest.cs(35,4): error CS0616: 'TestCase' is not an attribute class
XamarinSourcesPathManglerTest.cs(37,4): error CS0616: 'TestCase' is not an attribute class
OpenTKManglerTest.cs(36,4): error CS0616: 'TestCase' is not an attribute class
OpenTKManglerTest.cs(37,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(37,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(38,4): error CS0616: 'TestCase' is not an attribute class
MonoPathManglerTest.cs(39,4): error CS0616: 'TestCase' is not an attribute class
XamarinSourcesPathManglerTest.cs(47,4): error CS0616: 'TestCase' is not an attribute class
* [xharness] Run monotouch-test both with and without the static registrar in the simulator.
This way we catch (some) problems with the static registrar without having to run on device.
* [xharness] Add comments to explain extra test configurations.
* [xharness] Fix how projects are cloned.
Fix how projects are cloned so that xharness doesn't end up confused when
multiple test tasks share the same build task.
Previously xharness would iterate over all test tasks and modifying their
build tasks, now it will create the build tasks correctly in the first place.
In particular the mmp tests build a lot of project files, which means they can
find issues whenever something in the MSBuild logic changes, so run them.
They would have caught https://github.com/xamarin/maccore/issues/612 for instance.
* Not every old annotations have been migrated (work in progress, to be completed in another PR);
* Sanitation of the data files (e.g. removal of dupes and fixed, by Apple, entries) is done, but not automated (also a work in progress)
Even then this is immediately useful, i.e. better merged before 15.6 gets branched.
* [Debugger] Allow to step into Xamarin code.
The tool now detects the different sources and mangles the path to
ensure that the correct paths are installed in the following location:
/Library/Frameworks/Xamarin.iOS/Versions/Current/src
/Library/Frameworks/Xamarin.Mac/Versions/Current/src
The tool will detect if the path map command was used in the mdb and pub files, will point to the correct source code and will copy it to the installation dir.
Tests have been added that will be ran both by wrench and jenkins ensuring that changes in the manglers do not change it behaviour.
Make the collection of logs (the `Logs` class) more capable by making it
disposable (which will dispose all contained logs) and give it a Directory
property to state where the logs should be stored. This makes it possible to
simplify a few repeated path calculations. It also allows us to easily dispose
the entire collection of logs when done with a test, as opposed to dispoing
the logs one by one.
Make LogFile more capable:
* Add support for writing byte arrays.
* Add support for logging after disposal: this will still write to the file,
and not keep any files open after finished writing. This fixes a problem
where writing to a log after it was disposed would crash xharness (which is
not all that uncommon, given the async nature of how xharness runs tests).
This makes it possible to get rid of LogStream, and use LogFile instead.
* [msbuild] Pack all iOS MSBuild Task assemblies into a single assembly
* Fixed the build
* Renamed ProcessArgumentBuilder to CommandLineArgumentBuilder
This is needed to prevent symbol conflicts with Xamarin.MacDev's
ProcessArgumentBuilder (which is functionally different from
Xamarin.MacDev.Tasks.Core's class of the same name).
* Fixed ILRepack logic for filtering dll's to repack
* Fixed building of Xamarin.iOS.Tasks.Tests now that X.iOS.Tasks.dll contains all symbols
* Updated Makefile now that only 1 iOS Task assembly needs to be distributed
* ILRepack Xamarin.Mac.Tasks as well
* Fixed up *.targets to specify The One Assembly To Rule Them All
* [xharness] Build MSBuild tests with MSBuild.
* Touch the ilrepack stamp file *after* invoking ILRepack, not before.
* Same for Xamarin.Mac.Tasks
The mmp regression tests are already executed in parallel (using `make -j8` in
the Makefile), and they're quite CPU intensitive, which means that if xharness
runs other tests in addition to the these tests, then those other tests may
fail randomly.
Dispose logs (so that the corresponding files are closed) when we're done with
them, and also don't open a file log by default (usually we just want a
filename to pass to somebody else), but instead open the file if needed.
This should decrease the number of open file descriptors in xharness, which
sometimes become a problem when running many tests.
* [tests] Add support for passing arguments to XM unit tests from the command line.
* [xharness] Get xml results for mac unit tests and parse it to show failures inline in the html report.
Usually an exit code of neither 0 nor 1 means a test run crashed (in
particular if the exit code is 130+), but this may not always be true, so
report the exit code as well so that a human can evaluate properly.
Change [Intermediate]OutputPath of Xamarin.Mac test projects to include the
project flavor, so that they don't build into (and run from) the same location
(which leads to random and incorrect behavior for at least one of the
flavors).
Fixes https://github.com/xamarin/maccore/issues/584
* [xharness] Add support for executing a command periodically.
This will be used to run 'rsync' on bots to upload the html report somewhere
while the tests are running.
* [tests] Disable all the wrench test targets, and instead run the jenkins target (only) on wrench.
* [xharness] Write xharness log to stdout as well on wrench.
Wrench has a 30-min stdout timeout: if nothing is printed in 30 minutes, then
the step fails. Printing the harness log to stdout makes us not hit this
timeout.
* [xharness] Timestamp a few more logs.
* [xharness] Disable the @MonkeyWrench calls, since we're not uploading directly to wrench anymore.
* [tests][generator] Port bindas1048error to NUnit.
* [tests][generator] Port bindas1049error to NUnit.
* [tests][generator] Port bindas1050modelerror to NUnit.
* [tests][generator] Port bindas1050protocolerror to NUnit.
* [tests][generator] Port bug42855 to NUnit.
* [tests][generator] Port bug57070 to NUnit.
* [tests][generator] Port bug52570classinternal to NUnit.
* [tests][generator] Port bug52570methodinternal to NUnit.
* [tests][generator] Port bug52570allowstaticmembers to NUnit.
* [tests][generator] Port protocol-duplicate-abstract-error to NUnit.
* [tests][generator] Port protocol-duplicate-method-diff-length to NUnit.
* [tests][generator] Port protocol-duplicate-method-diff-out to NUnit.
* [tests][generator] Port protocol-duplicate-method-diff-type to NUnit.
* [tests][generator] Port protocol-duplicate-method-diff-return to NUnit.
* [tests][generator] Port warnaserror to NUnit.
* [tests][generator] Port nowarn to NUnit.
* [tests][generator] Add support for inspecting/asserting the generated content.
* [tests][generator] Port some Xamarin.Mac tests to NUnit.
Ported:
* bmac_smoke
* bmac-with-hyphen-in-name
* property-redefination-mac
* NSApplicationPublicEnsureMethods
* protocol-duplicate-abstract
* [tests][generator] Point bgen to our local installation.
* [tests][generator] Port the bug31788 test to a unit test.
* [generator] Make the 'bgen' helper target more complete.
* [tests][generator] Port non-custom iOS tests to unit tests.
* [tests][generator] Add new test.
* [tests][generator] Port the forum54078 test to a unit test.
* [tests][generator] Port the desk63279 test to a unit test.
* [tests][generator] Port the desk79124 test to a unit test.
* [tests][generator] Port the multiple-api-definitions tests to unit tests.
* [generator] Use mono code style.
* [tests][generator] Port the bug29493 test to a unit test.
* [tests][generator] Port the classNameCollision test to a unit test.
* [tests][generator] Port the bi1036 test to a unit test.
* [tests][generator] Port the bug37527 test to a unit test.
Also fix BI1112 and BI1113 to show up as errors in the console output (since
they're exceptions they're already treated as errors and would cause bgen to
fail).
* [tests][generator] Port the bug27986 test to a unit test.
* [tests][generator] Port the bug35176 test to a unit test.
* [tests][generator] Port the bi1046 test to a unit test.
* [tests][generator] Port the virtualwrap test to a unit test.
* [tests][generator] Port the bug42742 test to a unit test.
* [tests][generator] Port the noasyncinternalwrapper test to a unit test.
* [tests][generator] Port the noasyncwarningcs0219 test to a unit test.
* [tests][generator] Port the bug53076 test to a unit test.
* [tests][generator] Port the bug53076withmodel test to a unit test.
* [tests][generator] Port the fieldenumtests test to a unit test.
* [tests][generator] Port the smartenumwithframework test to a unit test.
* [tests][generator] Port the forcedtype test to a unit test.
* [tests][generator] Port the bug46292 test to a unit test.
* [tests][generator] Build tests with MSBuild.
There's no need to use xbuild for these tests.
* [tests][generator] Remove dead code.
* [xharness] Don't run the makefile-based generator tests anymore.
Since there aren't any makefile-based generator tests anymore, they've all
been ported to NUnit tests.
* [tests][generator] Make the bug39614 test do what it was supposed to do: make sure a namespace isn't required (but recommended).
Detect MT1111 from mlaunch (which means mlaunch won't wait for the app to
exit), and instead use test run completion to determine test completion.
The main drawback is that if the test app crashes, it won't be detected (the
test run will time out, and reported as such), but it's still an improvement
over the current behavior (the tests may complete successfully, and still be
reported as timed out).
This also requires bumping maccore to get an updated mlaunch (one that reports
MT1111).
* [macos] Fix guiunit error on clean builds by depending on correct copy
- From a clean build making a BCL test would error due to the non-mobile guiunit not being built
- This was because the Makefile-mac.inc target was incorrect
- This was because xharness assumed that non variation based targets were always Modern
- However, BCL tests are Full, not Modern
* Code review change
* Swap to var to reduce diff
This will add Objective Sharpie as an optional dependency, only enforced if
actually trying to run the xtro tests.
The wrench/jenkins tests will show up as green for now (since there are known
failures in the xtro tests that have to be addressed first).
* [xharness] Don't list unusable devices.
* [xharness] Show the list of candidate devices in the html report.
* [xharness] Prioritize devices depending on the interface speed.
* [xharness] Simplify code a bit.
Logs are TextWriters by themselves, so no need to get a StreamWriter to pass
to API that takes TextWriter, when we can just pass the log instance itself.
This makes it possible to timestamp external process output (because
Log.Timestamp is honored instead of bypassed).
* [xharness] Timestamp install logs.
So that we get exact numbers of how long it takes to install on watch (and if
the watch installation stalls, or just times out because it takes too long).
We construct every test case independent of whether a simulator or device is
available, which means we don't have to wait for simulators/devices to load
before creating test cases. Enumerating the available simulators/devices will
still block until loading is complete, but that only happens when we want to
run a test (after building it).
This makes the initial load much faster, since listing devices is becoming
slower now.
There seems to be two bugs:
1. Comparing identical styles returns false (although this may just be how javascript works).
2. Given the following:
ar_obj.style.display = 'hide';
new_obj.style.display = 'hide';
ar_obj.style = new_obj.style;
// now ar_obj.style.display == 'block'. HUH!?!
I can't fathom how this is expected, but Chrome shows the same behavior. Yay javascript.
Fix this by not trying to assign identical styles, by comparing the css text
instead of the style objects.
Rolf Bjarne Kvinge
Jeffrey Stedfast
Sebastien Pouliot
Manuel de la Pena
Chris Hamons
Sebastien Pouliot