26a3394623
[XHarness] Clean the bcl-tests directory. ( #6165 )
...
The change is a pre step to fix https://github.com/xamarin/xamarin-macios/issues/6162
We are removing the not longer child directory bcl-test/BCLTests since
there is not longer any reason for it to exist. The changes make sure
that the tests work as expected in the new directory.
Next step, move all the test apps to their own directory.
2019-05-31 13:05:22 +02:00
d8b7580849
[XHarness] Re-enable tvOS tests. ( #6174 )
...
After mono fixed issues https://github.com/mono/mono/issues/14497 and
https://github.com/mono/mono/issues/14496 re-enable the ignored tests.
Fixes: https://github.com/xamarin/maccore/issues/1610
Fixes: https://github.com/xamarin/maccore/issues/1611
2019-05-30 19:14:45 +02:00
b5958e89c4
Merge remote-tracking branch 'origin/master' into binary-artifacts
2019-05-29 08:27:24 +02:00
0346e71213
[xharness] Remove mono tests built locally in favor of binary archive binaries ( #6141 )
...
* [XHarness] Remove the old style mscorlib tests.
Remove the old style test and replace it with the xunit equivalent which
has more tests and is provided by the mono package.
* Only skip the mscorlib tests on watchOS devices with 32b. Run them anywhere else.
2019-05-28 23:25:01 -07:00
b8b79b03cb
Merge remote-tracking branch 'upstream/master' into binary-artifacts
2019-05-28 21:43:52 +02:00
f1c7d1e3fd
[Harness] Pass extra arguments to configure the linker for certain tests ( #6143 )
...
Some tests need a special linker configuration. Add the extra arguments
so that tests do pass on device.
Fixes: https://github.com/xamarin/maccore/issues/1608
2019-05-28 07:04:42 -07:00
2892c1b03a
[XHarness] Ignore threadpool tests that make tests deadlock. ( #6142 )
...
Ignore the threadpool tests that block our testing applications.
The following mono issue was created: https://github.com/mono/mono/issues/14666
Fixes: https://github.com/xamarin/maccore/issues/1651
2019-05-28 01:16:15 -07:00
8627c4311a
Merge remote-tracking branch 'upstream/master' into binary-artifacts
2019-05-24 20:27:09 +02:00
9fa35569d1
[XHarness] Re-enable corlbi tests on Mac Full and Modern ( #6121 )
...
Tests are readded from the test dll from mono (xunit)
Fixes: https://github.com/xamarin/maccore/issues/1203
2019-05-24 01:29:30 -07:00
25b5d36ddd
[tests] Don't make the mscorlib tests exit upon suspend. ( #6115 )
...
It means that on watchOS they'll exit if the screen turns off.
I have no idea why we set it up like this in the first place, it's been like
this since we created the mscorlib tests ([1]).
[1]: 76e15036e8
2019-05-24 00:37:25 -07:00
42b4b383b7
Fix BCLTests projects to use xunit.execution.dotnet.dll from mono
2019-05-24 01:36:37 +02:00
776d40550c
Merge remote-tracking branch 'upstream/master' into binary-artifacts
2019-05-23 20:19:27 +02:00
8e57754ded
[XHarness] Improve xml parsing. Fixes #6072 ( #6099 )
...
The entire Xml parsing code has been changed to make the following
improvements:
1. Simplify the logic.
2. Ensure that the tests are not added more than once in the human text
log.
3. Do not use the APIs that load the entire tests into memory. XmlReader
is used to read the file and parse the results.
Fixes: https://github.com/xamarin/xamarin-macios/issues/6072
Fixes: https://github.com/xamarin/maccore/issues/1632
2019-05-23 01:04:37 -07:00
48b3b82410
Merge remote-tracking branch 'upstream/master' into binary-artifacts
...
# Conflicts:
# builds/Makefile
# external/mono
2019-05-22 20:07:18 +02:00
fb16b9b733
Remove unnecessary test projects
2019-05-22 18:02:37 +02:00
5ba7fc4088
BCLTests: use Xunit.NetCore.Extensions from the mono archive
...
Also remove other xunit assembly since we get the from nuget PackageReference already.
2019-05-22 17:48:55 +02:00
6228e49554
[tests] Link NUnit/XUnit watchOS extensions.
...
Makes them much smaller.
2019-05-22 08:13:31 +02:00
2782ad3482
[tests] Fix reporting back to xharness for watchOS NUnit/XUnit tests.
...
* Choose the first hostname for the HttpTextWriter if there are multiple hosts.
* Open the HttpTextWriter before writing to it.
* Don't overwrite the http writer with another writer immediately after creating it.
* Close the HttpTextWriter when done writing.
* Wait for the HttpTextWriter to complete the final http request before exiting.
2019-05-22 08:13:25 +02:00
4e9b69e870
[xharness] Add ARM64_32 support.
2019-05-22 08:06:44 +02:00
247a6bf551
Switch to mono archive and remove submodule
2019-05-21 22:16:02 +02:00
835daf1f60
Merge remote-tracking branch 'origin/master' into arm64_32-v3
2019-05-21 07:53:23 +02:00
b8824524b2
[XHarness] Remove the old style bcl tests. ( #6083 )
2019-05-20 16:58:45 -07:00
a9372793a7
Merge remote-tracking branch 'origin/master' into arm64_32-v3
2019-05-20 18:45:19 +02:00
5b4b398ae6
[tests] Link NUnit/XUnit watchOS extensions.
...
Makes them much smaller.
2019-05-20 18:00:18 +02:00
21b8690818
[tests] Fix reporting back to xharness for watchOS NUnit/XUnit tests.
...
* Choose the first hostname for the HttpTextWriter if there are multiple hosts.
* Open the HttpTextWriter before writing to it.
* Don't overwrite the http writer with another writer immediately after creating it.
* Close the HttpTextWriter when done writing.
* Wait for the HttpTextWriter to complete the final http request before exiting.
2019-05-20 18:00:18 +02:00
94fe39b118
[XHarness] Do use the test dlls from the mono downloads. ( #6055 )
...
This updates the project generation. We cannot yet fully remove the submodule because:
* We are missing the xunit dlls which should be added in the SDK.
* We have not yet remove all the old style tests. Would make the PR huge, better to deal with it in a diff PR.
* The xunit CoreLib tests have issues loading all the tests, needs some extra work and again, the PR is already large.
Fixes : xamarin/maccore#1199
Fixes : xamarin/maccore#1204
Fixes : xamarin/maccore#1209
Fixes : xamarin/maccore#1510
2019-05-19 14:55:14 -07:00
47beab425d
Merge remote-tracking branch 'origin/master' into arm64_32-v3
2019-05-16 15:26:15 -07:00
762146a95c
[XHarness] Ignore IO flacky tests in devices too. ( #6057 )
2019-05-16 10:15:11 -07:00
81d6225eb7
[XHarness] Fix some issues in the BCL test generator. ( #5938 )
...
We had issues in the code that adds a type found in an assembly to
ensure that it was not removed by the linker. This resulted in some
assemblies having 0 tests.
Added the needed ignore for the corlib tests and system ones.
2019-05-11 12:03:38 +02:00
2f8100d962
[xharness] Add ARM64_32 support.
2019-05-10 11:28:22 +02:00
6c3674658a
Download the Mono SDK archive for macOS as well. ( #5953 )
...
* Download the Mono SDK archive for macOS as well.
* [tests] Don't forget the tests.
2019-04-26 16:44:29 +02:00
8ed5e3926a
bcl-test-importer: Add three more unit test assemblies
2019-04-17 14:58:49 +02:00
dbbc954447
Reenable tests that were fixed
2019-04-17 11:30:41 +02:00
1cdf2ac3a8
Revert "[XHarness] Update ignore for some failing tests. ( #5887 )"
...
This reverts commit a3ad78c937
2019-04-11 20:31:48 +02:00
db86c8cc88
Merge remote-tracking branch 'upstream/master' into mono-2019-02
2019-04-11 20:28:57 +02:00
a3ad78c937
[XHarness] Update ignore for some failing tests. ( #5887 )
...
The mono bump added some new tests that fail on device + release
configurations. This is a known issue by mono and they'll fix it in the
next version but will not be backported to 2018-10.
2019-04-11 17:20:55 +02:00
3a398ee5dd
[System.Net.Http] Set LEGACY_HTTPCLIENT for test project
...
Otherwise a bunch of tests that assume the new behavior fail.
2019-04-09 12:11:07 +02:00
1664cf6426
Merge branch 'master' into mono-2019-02
2019-04-05 15:53:37 -05:00
2326e87bd5
[XHarness] Ignore tests that fail on device. ( #5858 )
2019-04-05 15:42:01 -05:00
dd11ba5d6c
[XHarness] Fix issues building the today extensions for the BCL tests apps. ( #5854 )
2019-04-05 14:07:39 +02:00
067cbf0a28
Merge remote-tracking branch 'upstream/master' into mono-2019-02
...
# Conflicts:
# tools/mmp/driver.cs
2019-04-05 11:19:11 +02:00
3a0dfe42a7
[XHarness] Ignore a failing test on devices. ( #5848 )
2019-04-04 18:34:22 +02:00
56bb4f9505
Reenable more BCLTests
2019-03-28 11:16:47 +01:00
b1b557a1b9
Reenable a few BCLTests
2019-03-27 13:33:23 +01:00
421e673227
Merge remote-tracking branch 'upstream/master' into mono-2019-02
2019-03-27 01:33:49 +01:00
45f3e649b9
Ignore tests for devices ( #5807 )
...
* [XHarness] Ignore some tests that fail on device.
Added the ignore, which can be later removed on the new mono 2019-02
since the issues do not happen there.
Fixes: https://github.com/xamarin/maccore/issues/1495
* [XHarness] Ignore corlib tets failing on devices.
Added the required ignores to get the devices green and workaround mono
issue https://github.com/mono/mono/issues/13641
2019-03-26 12:25:23 +01:00
0e249818d2
Merge remote-tracking branch 'upstream/master' into mono-2019-02
2019-03-22 10:32:58 +01:00
1615b28ed1
[XHarness] Reduce the number of test applications for the BCL tests. ( #5768 )
...
This commit improves the state of the BCL testing in the following ways:
1. Improve the device tets running. Less apps, faster results.
2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues.
We now support the ignore files per assembly name to simplify the
tracking of the ignored tests. All
2019-03-20 12:10:18 +01:00
8c96fb2600
Fix missing file in System-mac.csproj.template
2019-03-13 19:54:04 +01:00
0bcbb67a45
Merge remote-tracking branch 'upstream/master' into mono-2019-02
2019-03-13 18:39:15 +01:00
22832369df
[XHarness] Fix crash in the bcl tests. ( #5762 )
2019-03-13 17:28:16 +01:00
60812c0ba6
Use LangVersion=latest in mscorlib test csproj
...
Some tests require ref structs now which needs latest C#.
2019-03-13 00:34:55 +01:00
3fc1e32adb
[XHarness] Reenable the NUnit System.Data tests on mac os x. ( #5749 )
...
Fixes https://github.com/xamarin/maccore/issues/1192
Fixes https://github.com/xamarin/maccore/issues/1193
2019-03-11 18:50:18 +01:00
849355eaf2
[XHarness] Don't run BitcodeNotSupported BCL tests on watchOS. ( #5750 )
2019-03-11 12:08:48 +01:00
5bfbefab14
[XHarness] Reneable System.Security nunit tests on mac os x. ( #5751 )
...
Fixes https://github.com/xamarin/maccore/issues/1197
2019-03-11 12:08:10 +01:00
7ef507b579
[XHarness] Fix the hostname parsing on devices. ( #5752 )
...
Following this
221f3e4a37/2478649/device-tests/jenkins-results/tests/index.html
2019-03-11 12:04:32 +01:00
9eb75dda71
[XHarness] Reenable the System.Core tests on mac os x. ( #5743 )
...
Fixes https://github.com/xamarin/maccore/issues/1191
2019-03-08 17:02:15 +01:00
99e727c90b
Merge pull request #4904 from xamarin/mono-2018-10
...
Bump to mono:2018-10
2019-03-08 11:06:50 +01:00
112502c081
[Xharness] Reenable the SystemXml tests on mac os x. ( #5735 )
...
Fixes https://github.com/xamarin/maccore/issues/1201
Fixes https://github.com/xamarin/maccore/issues/1202
2019-03-07 17:19:47 +01:00
bc3b5a2d43
[Tests] Fix the tests.sln to include the new test projects. ( #5720 )
...
Update the sun and ensure that the GUID of the projects are always the same to make the .sln file happy.
2019-03-07 13:14:53 +01:00
cad438dcfa
[XHarness] Reenable the System.Configuration tests on mac os x. ( #5702 )
...
Reenable the tests and add support to ignoring certain tests depending
on the version of the mac os x that is under test. The
System.Configuration tests make no sense in Modern.
Fixes https://github.com/xamarin/maccore/issues/1189
Fixes https://github.com/xamarin/maccore/issues/1190
2019-03-07 13:09:56 +01:00
92ebe5b64a
[XHarness] Reenable SystemServiceModelTests on mac os x. ( #5710 )
...
Fixes https://github.com/xamarin/maccore/issues/1198
2019-03-04 18:17:56 +01:00
568d19944d
[XHarness] Reneable SystemIdentityModelTests on mac os x. ( #5708 )
...
Fixes https://github.com/xamarin/maccore/issues/1194
2019-03-04 17:42:07 +01:00
10912633e3
[Xharness] Reenable the xunit based System.Data tests on mac os x. ( #5709 )
...
Fixes https://github.com/xamarin/maccore/issues/1208
2019-03-04 17:41:37 +01:00
98d9fc2cf3
[XHarness] Reenable MonoCSharpTests on mac os x. ( #5696 )
...
Fixes https://github.com/xamarin/maccore/issues/1186
2019-03-04 10:56:22 +01:00
de82764456
[XHarness] Reneable the System.Transaction tests on Mac OS X. ( #5686 )
...
Reenable the tests and ensure that the failing ones on Modern are
ignored. The bcl tests on mac os x can now make a diff between the
version (Moder/Full) to ignore files so that tests that work in a
version are not ignored in the other one.
Fixes https://github.com/xamarin/maccore/issues/1200
2019-03-01 10:29:21 +01:00
97b5aef47e
Merge branch 'master' into mono-2018-10
2019-02-28 21:18:49 +01:00
c2967c0d50
[XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes #5383 ( #5574 )
...
* [XHarness] Ensure we do a nuget restore on bcl imported tests. Fixes #5383
We need to ensure that a nuget restore is done, otherwise we might have
build issues with missing libs.
Fixes https://github.com/xamarin/xamarin-macios/issues/5383
2019-02-28 18:27:01 +01:00
bbbf15a72b
Merge branch 'master' into mono-2018-10
...
# Conflicts:
# tests/bcl-test/System.Core/System.Core.csproj.template
# tests/bcl-test/System.Net.Http/System.Net.Http-mac.csproj.template
# tests/bcl-test/System.ServiceModel.Web/System.ServiceModel.Web.csproj.template
# tests/bcl-test/System.Xml.Linq/System.Xml.Linq-mac.csproj.template
# tests/bcl-test/System/System.csproj.template
2019-02-28 13:06:00 +01:00
297aebef60
[Tests] Fix the make rule that generate the test projects. Fixes #5638 ( #5666 )
...
Fixed the rule to generate the tests. This branch adds missing test dlls
that the make rule check, or it will give an error stating that tests
are missing.
This feature should be added to xharness and an issue was created for
that: https://github.com/xamarin/maccore/issues/1447
Fixes https://github.com/xamarin/xamarin-macios/issues/5638
2019-02-28 11:52:04 +01:00
9c35ba28d1
[XHarness] Allow all cultures in the bcl tests. #5640 ( #5667 )
...
Some of the bcl tests need certain cultures. Add them to the templates
so that tests do not fail due to a missing culture.
Fixes https://github.com/xamarin/xamarin-macios/issues/5640
2019-02-28 11:48:35 +01:00
8972d43416
[XHarness] Remove not longer needed .csproj on the bcl tests. ( #5676 )
...
Followin PRs:
* Mac removal https://github.com/xamarin/xamarin-macios/pull/5671
* iOS/TvOS/WatchOS removal https://github.com/xamarin/xamarin-macios/pull/5668
We do not longer needs theses .csprojs since the tests are not longer
ran.
2019-02-28 11:45:19 +01:00
1e9de96789
Merge branch 'master' into mono-2018-10
2019-02-24 10:20:44 +01:00
7cfbaf019a
[XHarness] Ensure that the correct values are shown on the test reports. ( #5649 )
...
Fixes #5639
The results reported in the xharness run do not add up. Total tests
should include the number of skipped tests. This change ensures that the
tests results correctly add up.
Example nunit fixed tests: [NUnit] Mono SystemIOCompressionTests (total should
be 24, passed 4, ignored 20).
Example xunit fixed tests: [xUnit] Mono SystemDataXunit (total should be
1061, passed 1054, ignored 7).
Fixes https://github.com/xamarin/xamarin-macios/issues/5639
2019-02-22 17:12:07 +01:00
3f512606fb
Remove no longer needed ignore files from BCLTests
...
The issues don't show up in 2018-10 anymore.
2019-02-20 14:30:04 +01:00
8b21371437
Merge branch 'master' into mono-2018-10
2019-01-30 13:43:03 +01:00
40c998e264
[XHarness] Add XUnit category filtering. ( #5455 )
...
Provide the ability to filter per categories in the xunit runner and add the default categories for Mac, iOS, TvOS and WatchOS.
2019-01-28 17:47:36 +01:00
6c12862d3e
[XHarness] Remove not needed ignore file. ( #5456 )
...
Since we support filtering via categories, we do not longer need the
ignore file in these tests.
2019-01-24 13:11:11 +01:00
debcee412c
[XHarness] Clean the System.Net.Http ignore file. ( #5459 )
...
Since we support categories, there is a test that does not longer needs
to be ignored in ios and tvos but needs to be in watchos (we do not yet
have the correct test assembly for the platform).
2019-01-24 13:10:40 +01:00
5165f1cdde
[Xharness] Remove not longer needed System.Service.Model.Web ignore file. ( #5460 )
...
Since we support categories, the file is not longer needed since the
failing tests are part of an ignored category.
2019-01-23 12:28:54 +01:00
99bda10aa4
[XHarness] Clean the System.Data ignore. ( #5458 )
...
Since we support filtering there are a number of tests that were ignore
that are part of a ignored category, therefore, they are not needed in
the file.
2019-01-23 10:17:46 +01:00
02f65cc119
[XHarness] Remove not needed ignore file for System.Transactions. ( #5457 )
...
Since we support filtering via categories, we do not longer need the
ignore file in these tests.
2019-01-22 23:24:44 +01:00
a71f421a67
[XHarness] Reneable Sytem.Security tests on iOS, TvOS and WatchOS. ( #5421 )
...
Reneable the tests and ignore those failing ones.
Fixes https://github.com/xamarin/maccore/issues/1139
2019-01-22 23:18:45 +01:00
a97dddf6db
[XHarness] Remove not needed dll from the test templates. ( #5454 )
2019-01-22 17:49:03 +01:00
d9dcc9ae21
[XHarness] Add support for filtering categories in NUnit. ( #5446 )
2019-01-22 14:59:36 +01:00
bec4e8604a
Merge branch 'master' of github.com:xamarin/xamarin-macios into mono-2018-10
2019-01-18 14:09:29 +01:00
879629bc9d
[XHarness] Reenable the System tests on iOS, WatchOS and TVOs. ( #5425 )
...
All the failing tests have been ignored. There are a large number of
tests ignored on the watchOS platform beacuase atm the iOS test dll is
used.
Fixes https://github.com/xamarin/maccore/issues/1135
2019-01-17 22:58:31 +01:00
cf1a034e96
[XHarness] Reenable the NUnit Corlib tests from the iOS Mono SDK. ( #5316 )
...
Reenable the tests on iOS/TvOS/WatchOS. Fixes xamarin/maccore#1153
2019-01-17 22:55:06 +01:00
8b68cdc62f
[XHarness] Report issues with the TestFixtureSetup ( #5428 )
...
There are a number of tests that use a TestFixtureSetup method that
fails. While the xml results report these errors, if a user runs xharness locally,
he will not get the errors reported. This issue meant that we had
different errors reported in jenkis and locally.
2019-01-17 22:54:06 +01:00
fea85dc569
[XHarness] Reenable SystemTransactionsTests on iOS, TvOS and WatchOs. ( #5411 )
...
Fixes https://github.com/xamarin/maccore/issues/1134
2019-01-16 19:12:20 +01:00
34ca0496fc
[XHarness] Reneable the SystemNetHttpTests on iOS, TvOS and WatchOS. ( #5405 )
...
Failing iOS tests are ignored WatchOS is running the iOS tests until we have the tests in the SDK
for the watch runtime.
Fixes https://github.com/xamarin/maccore/issues/1144
Fixes https://github.com/xamarin/maccore/issues/1145
Mono issue: https://github.com/mono/mono/issues/12430
2019-01-16 12:16:22 +01:00
c455a1d7b9
[XHarness] Reneable Mono.Security tests on WatchOS. ( #5412 )
...
Fixes https://github.com/xamarin/maccore/issues/1142
2019-01-16 12:13:34 +01:00
293ade0a30
[XHarness] Reneable the SystemSecurity tests on iOS, TvOS and WatchOS ( #5410 )
...
Some of the test classes are abstract, and we needed to change the
generator to ignore that fact. Before the change, we would have 0 test
cases.
Failing tests are ignored.
Fixes https://github.com/xamarin/maccore/issues/1128
Mono issue https://github.com/mono/mono/issues/12446
2019-01-16 12:11:53 +01:00
e53bfac5c0
[XHarness] Reenable System.Core tests on ios/tvos/watchos ( #5397 )
...
Fixes https://github.com/xamarin/maccore/issues/1143
Mono issue: https://github.com/mono/mono/issues/12419
2019-01-14 17:16:41 +01:00
102339ad47
[XHarness] Reenable the System.ServiceModel tests from the bcl. ( #5384 )
...
Fixes https://github.com/xamarin/maccore/issues/1138
A new mono issue was added: https://github.com/mono/mono/issues/12417
2019-01-14 16:58:36 +01:00
929e4623c4
Merge branch 'master' into mono-2018-10
...
# Conflicts:
# external/mono
2019-01-14 15:26:56 +01:00
adc9ddccf3
[XHarness] Reenable the System.Data tests from the bcl. ( #5379 )
2019-01-11 16:56:30 +01:00
25f306461c
Fix building and running of BCL tests
...
BCL tests are now reading test assets from embedded resources instead of from disk.
2019-01-11 14:30:47 +01:00
6a1ecaabb0
[XHarness] Reenable the System.IO.Compression.Filesystem tests and ignore those failing ones. ( #5374 )
2019-01-11 12:22:48 +01:00
f2e5ffba6d
Remove SyncTestResources.targets
...
It's no longer needed now that the BCL tests read test assets from embedded resources.
2019-01-10 22:14:58 +01:00
Manuel de la Pena
Rolf Bjarne Kvinge
Alexander Köplinger
Sebastien Pouliot
Bernhard Urban