We've recently started using more and more ARM64 bots, so these tests are
failing in those cases.
Fixes:
MonoTouchFixtures.Security.SecureTransportTest
[FAIL] SslSupportedCiphers : 3224092716
Expected: True
But was: False
at MonoTouchFixtures.Security.SecureTransportTest.SslSupportedCiphers () [0x000de] in <f08bd76ea2204d929c96e3bd36ae955a>:0
Xamarin.Mac.Tests.EveryFrameworkSmokeTests
[FAIL] ExpectedLibrariesAreLoaded : QTKitLibrary (/System/Library/Frameworks/QTKit.framework/QTKit) failed to load but this was not expected
at Xamarin.Mac.Tests.EveryFrameworkSmokeTests.ExpectedLibrariesAreLoaded () [0x000e6] in <f08bd76ea2204d929c96e3bd36ae955a>:0
Also adjust naming and descriptions to make it clear we don't use a
specific minor version of each major OS version (so we're running on
"macOS 11", not necessarily "macOS 11.5").
---------
Co-authored-by: Alex Soto <alex@soto.dev>
Co-authored-by: Alex Soto <alex@alexsoto.me>
* Try again if we get a DuplicateItem error in MonoTouchFixtures.Security.KeyChainTest.Add_Certificate, hopefully fixes this random test failure:
MonoTouchFixtures.Security.KeyChainTest
[FAIL] Add_Certificate : Add_Certificate
Expected: Success
But was: DuplicateItem
at MonoTouchFixtures.Security.KeyChainTest.Add_Certificate() in /Users/builder/azdo/_work/2/s/xamarin-macios/tests/monotouch-test/Security/KeyChainTest.cs:line 49
Ref: https://github.com/xamarin/maccore/issues/2394
* Use a unique account name in MonoTouchFixtures.Security.RecordTest.Protocol_17579, hopefully fixes this random test failure:
MonoTouchFixtures.Security.RecordTest
[FAIL] Protocol_17579 : Add
Expected: Success
But was: DuplicateItem
at MonoTouchFixtures.Security.RecordTest.Protocol(SecProtocol protocol) in /Users/builder/azdo/_work/2/s/xamarin-macios/tests/monotouch-test/Security/RecordTest.cs:line 204
at MonoTouchFixtures.Security.RecordTest.Protocol_17579() in /Users/builder/azdo/_work/2/s/xamarin-macios/tests/monotouch-test/Security/RecordTest.cs:line 232
Ref: https://github.com/xamarin/maccore/issues/2692
The “Thawte SGC CA” certificate we're using for testing is now blocked by
Apple, evaluating trust results in this error:
> “Thawte SGC CA” certificate is blocked
So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed).
Then adjust the tests accordingly.
Finally delete all unused certificates from the code.
Fixes these test failures:
MonoTouchFixtures.Security.TrustTest
[FAIL] Trust_FullChain : GetTrustResult-2
Expected: RecoverableTrustFailure
But was: FatalTrustFailure
at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389
at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321
[FAIL] Trust2_FullChain : GetTrustResult-2
Expected: RecoverableTrustFailure
But was: FatalTrustFailure
at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389
at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449
* Move all the RunAsync logic to the TestRuntime class, instead of having some
in TestRuntime and some in AppDelegate.
* Create a unified Task-based implementation for all platforms, optionally showing
a UI on the platforms where we can do that.
* Remove all the overloads that took a DateTime timeout, and instead only use a
TimeSpan timeout. This simplified some of the code.
* The new Task-based implementation will capture any exceptions (and rethrow most
of them) from the tasks we're waiting for, so no need to do that in each RunAsync
caller. This simplifies the testing code a lot for some tests.
* Add a new TryRunAsync method that will return (instead of rethrowing) any exceptions.
This simplifies some of the testing code (which verifies the correct exception,
or ignores the test in case of some exceptions).
* The new Task-based implementation will bubble up any NUnit exceptions, which
means that the tasks we're waiting for can call NUnit's Assert and the right thing
happens (in particular Assert.Ignore will actually ignore the test).
Change all null checking expressions to use 'is null' and 'is not null'
instead of '== null' and '!= null'.
This was mostly done with sed, so code can probably be improved in many
other ways with manual inspection, but that will come over time.
Also add code to the autoformat script to automatically fix these issues in the future.
* Remove the SslCipherSuite enum from .NET, it's complicated to implement
correctly on macOS for both x64 and arm64, and it's also obsolete, so just
remove it.
* Change the type for NSUrlSessionTaskTransactionMetrics.NegotiatedTlsCipherSuite
to be TlsCipherSuite instead of SslCipherSuite for .NET (this is in fact the
correct value according to the headers).
Fixes https://github.com/xamarin/xamarin-macios/issues/11498.
* Subclass NativeObject to reuse object lifetime code.
* Enable nullability and fix code accordingly.
* Use 'is' and 'is not' instead of '==' and '!=' for object identity.
* Use the null-safe NativeObjectExtensions.GetHandle extension method to get
the handle instead of checking for null (avoids some code duplication).
* Use 'nameof (parameter)' instead of string constants.
* Call 'GetCheckedHandle ()' (which will throw an ObjectDisposedException if
Handle == IntPtr.Zero) instead of manually checking for IntPtr.Zero and
throwing ObjectDisposedException.
* Use the 'Runtime.GetNSObject<T> (IntPtr, bool)' overload to specify handle
ownership, to avoid having to call CFObject.CFRelease manually later.
* Use Array.Empty<T> instead of creating an empty array manually.
* Remove the (IntPtr) constructor for .NET.
* Bump maccore.
New commits in xamarin/maccore:
* xamarin/maccore@9acbbed1f6 [mlaunch] Add support for Xcode 13 beta 1. (#2452)
* xamarin/maccore@e48f75c0b6 [Xamarin.Hosting] Fix the --stdout arg not being forwarded to DeviceLaunchConfig (#2435)
* xamarin/maccore@109c695b1b [Xamarin.Hosting] Fix help string for launchdev argument (#2429)
Diff: cddbd1915d..9acbbed1f6
* [xtro] Fix generation of .pch files
* [xtro] Fix deprecated check to handle (anonymous) declarations and enable latest C# syntax in project
* [xtro] Fix _sanity_ checks
* [xtro] Update todo for beta 1
* [Siminstaller] Force siminstaller to use the xcode 12.5 url
Related issue: https://github.com/xamarin/xamarin-macios/issues/11881
* Fix introspection failures (due to [breaking] changes)
* [tests][intro] Fix hang for tvOS
Creating an instance of `NSMetadataQuery` hangs the simulator.
Even after (xharness) timeout the simulator is not in a good state
to run further tests and every new (tvOS) test will also hang...
* [tests][intro] Same hang for watchOS
except that further test execution does not seem affected (like tvOS)
```
CoreSimulator 772.1 - Device: Apple Watch Series 3 - 38mm (watchOS 8.0) - created by XHarness (42262867-E060-40C0-803E-6DA676AF50CC) - Runtime: watchOS 8.0 (19R5266p) - DeviceType: Apple Watch Series 3 - 38mm
Thread 0 Crashed:: tid_103 Dispatch queue: com.apple.main-thread
0 com.apple.Foundation 0x00007fff21470bd0 -[NSMetadataQuery dealloc] + 432
1 libobjc.A.dylib 0x00007fff200d11f7 objc_object::sidetable_release(bool, bool) + 177
2 com.apple.Foundation 0x00007fff21470a03 -[NSMetadataQuery init] + 64
3 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107efc139 xamarin_dyn_objc_msgSend + 217 (trampolines-x86_64-objc_msgSend.s:15)
4 ??? 0x000000010c76d4f6 0 + 4504081654
5 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107cffc85 mono_jit_runtime_invoke + 1621 (mini-runtime.c:3197)
6 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107e177d8 do_runtime_invoke + 54 (object.c:3052) [inlined]
7 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107e177d8 mono_runtime_invoke_checked + 136 (object.c:3220)
8 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107e1e3c5 mono_runtime_try_invoke_array + 2101 (object.c:5601)
9 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107daf977 ves_icall_InternalInvoke + 871 (icall.c:3927)
10 com.xamarin.introspection_watch.watchkitapp.watchkitextension 0x0000000107dc0167 ves_icall_InternalInvoke_raw + 103 (icall-def.h:667)
11 ??? 0x000000010a232799 0 + 4465043353
12 ??? 0x000000010c76e08b 0 + 4504084619
```
* [tests][monotouch-test] Fix failures with xcode 13 beta 1
* [tests][mmptest] Use a FAT framework that's build with x86_64 and arm64
Co-authored-by: Alex Soto <alex@alexsoto.me>
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@microsoft.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
* [tests] Add a .NET/macOS version of monotouch-test.
I made a macOS version of monotouch-test instead of a .NET version of xammac tests,
so that one day we might have only one test suite for all our API tests.
* Add a project file for .NET/macOS
* Fix some code to handle the fact that we're called 'monotouchtest' on macOS (but
only on .NET).
* Ignore exception marshalling tests, those aren't working yet.
* [xharness] Add support for .NET/macOS and add a macOS version of monotouch-test to our tests
* [dotnet-linker] Skip libSystem.Net.Security.Native and libSystem.Native when collecting native methods to preserve for .NET/macOS.
This fixes at least some variations of this failure:
* Protocol_17579: Add
Expected: Success
But was: DuplicateItem
In this test we try to delete any existing keychain items, but it seems that
may fail if the keychain item in question was created by a different app, in
which case we wouldn't have permission to delete it (but adding a new one
would still fail with a duplicate item error). Cleaning up any added keychain
items by the same test (and process) that added them avoids the permissions
problem.
Also clean up in other tests as well, to avoid filling up our keychain with
test stuff.
* [monotouch-test] Rework big parts of KeyChainTest. Fixes #xamarin/maccore@2365.
'GenericPassword' keychain items are unique by their Service+Account
properties [1]. This means that changing the Label property will not create a
different 'GenericPassword', which has a few consequences:
* It's possible to filter (and try to delete) using the Label property.
* It's possible to try to delete a 'GenericPassword' item, and have that
deletion attempt fail with 'no item found', and then subsequently trying to
add the same item will fail with a DuplicateItem, because the deletion
was filtered using the Label property.
The change I've made is to:
* Make the Label property much more descriptive, and unique per process. This
makes it easier to figure out where things come from in the Keychain Access
app.
* Make the Service property unique per process. This way these tests are
parallel safe and they won't stomp on eachother.
* Keep the Account property the same (a constant value), so that it's easy to
filter to just these items in the Keychain Access app.
* Remove the Label property from all queries, it doesn't matter anyway. The
Label property is still set when adding items to the keychain.
Finally try to clean up after ourselves as good as possible. This way we don't
fill the keychain with test stuff. This involves removing certificates and
passwords we add to the keychain at the end of tests.
Fixes https://github.com/xamarin/maccore/issues/2365.
[1]: https://stackoverflow.com/a/11672200/183422
* Adjust query for RecordTest.AuthenticationType as well to not include fields that don't determine uniqueness.
Also make the InternetPassword items we add to the keychain more descriptive and labelled helpfully.
* [monotouch-test] Use a more descriptive label for items added to the keychain in KeyTest.
* [monotouch-test] Use a more descriptive label for some items in CertificateTest.
Also make generated keys non-permanent so that they're not added to the keychain.
* [monotouch-test] Clean up a few items added to the keychain by CertificateTest.
* Fix system version checks to work properly on Mac Catalyst (which uses the macOS
version as its system version).
* Add the framework-specific defines to the build for monotouch-test.csproj (using
the generated response files), this way we can use them in the tests.
* Sprinkle conditionals in numerous places - I tried using either framework-specific
or XAMCORE_3_0 whenever that made since (instead of using Mac Catalyst as a condition).
* Updated a few tests to use non-deprecated API (because the deprecated API often
doesn't exist in Mac Catalyst).
* Also a few minor API fixes to make any corresponding tests compile.
* Bump to Xcode 12.2 beta 1.
* [tests] Fix xtro and intro
* Fix wrong availability
* Fix monotouch tests
* Disable watchOS tests and bump iOS version to the right one
watchOS tests have been disabled because Xcode 12.2 Beta is
broken, you cannot create a watch app and deploy, Xcode just
crashes when you try to list simulators, If you try to use
our tools at the moment of deploying we get:
> error HE0046: Failed to install the app 'app' on the device 'watchOS 7.1 (18R5552f) - Apple Watch Series 6 - 44mm': Unable To Install ???app???
* Fix macOS check and bump both versions to satisfy commit distance check, luckily we have infinite numbers!!
Co-authored-by: Alex Soto <alex@alexsoto.me>
* [xcode12] Bump to Xcode 12 Beta 4
* [Tests] Disable Trust_FullChain and Trust2_FullChain tests
These tests started showing a different error than the one we expect
I'll let our security expert chime in
We've modified our NUnitLite to special-case the native types, so that they
can easily be compared with the standard types, but that doesn't work anymore
when switching to the official NUnit[Lite], so change all asserts to
explicitly convert whenever needed.
* [tests] Use Assert.Throws instead of the ExpectedException attribute.
The ExpectedExceptionAttribute doesn't exist in newer versions of NUnit[Lite] anymore.
* [monotouch-test] Rework some CaptiveNetwork tests.
They need permissions, and entitlements, and something else I couldn't figure
out, so just assert they're returning null instead.
* [Network] Improve bindings for NWProtocolMetadata.
It turns out the NWProtocolMetadata can contain metadata for different
protocols (Ip/Tls/Tcp). This is important; if someone tries to get a value for
one protocol and the metadata is for another protocol, then they invoke the
wrath of superior beings who will smite that poor someone with uninitialized
memory.
At that point there's not much left but to pray.
I don't like to depend on divine intervention, so I've modified the API here
to check if the metadata's protocol is the required type for the native API
we're calling, and if the check fails, we throw a nice and dependable managed
exception.
This is a functional breaking change; but if there are any lost souls who
likes to pray, they can always re-implement the P/Invokes themselves and skip
the sanity checks.
In addition I've renamed a few properties whose name didn't clearly specify
which protocol type they operate on.
Ref: Apple feedback FB6155967.
Ref: https://trello.com/c/1TW0BSKJ/145-fb6155967-nwipcreatemetadata-returns-uninitialized-metadata-in-ios-13
* Fix casing in exception message.
* [tests] Adjust the SecProtocolMetadataTest according to new knowledge about the Network API.
* Fix alternative property name in obsolete attribute.
Alex Soto
Rolf Bjarne Kvinge
Sebastien Pouliot
Manuel de la Pena