Includes support for `UTType`-based `[Field]` in the generator as an
hundred (or so) of them were added. Unit test shows the field-based
properties are working as expected.
There are two method not bound due to a dependency on AVAudioSession that
does not seem to be exposed on Mac OS X.
Related issue: https://github.com/xamarin/maccore/issues/2257
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
We were using the managed name, e.g. `NSUrl`, instead of the native name,
e.g. `NSURL`, when dealing with categories.
To fix this we must resolve the type and this caused issues as other
assemblies (e.g. OpenTK) were not already loaded/cached and some type
could not be resolved (and this throw exceptions)
The runner now loads all assemblies before starting to visit them.
The fix solved a known issue (iOS-NetworkExtension.ignore), some API
that were already bound (common-Foundation.ignore) and also caught an
additional API where we missed a `[NullAllowed]` on a return value
* [arkit] Remove fields (from beta2) to fix introspection
* [tests][introspection] AVMutableMediaSelection is as bad as it's non mutable parent
* [tools] Update IsFrameworkBroken (remove CoreAudioTypes and MediaPlayer)
* [tests][monotouch-test] MKPinAnnotationView seems fixed in beta 2
* [tests][xtro] Update ARKit todo (with previous fix)
A number of APIs added and deprecated in the same release. We will see
that is that about.
The status property move to be a instance property in CLLocationManager,
we expose those and add a deprecation warning. That needs to be ignored in xtro due to issue https://github.com/xamarin/xamarin-macios/issues/9026
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Add crash reports to a logs collection we care about, instead of to a logs
collection that's promptly forgotten.
This makes sure crash reports actually show up in the html report.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [linker] Fix infinite queue found with nullability PR (#8337)
and removed previous workaround
Replace previous attempt https://github.com/xamarin/xamarin-macios/pull/8336
* [linker] Remove code to deal with ExportedTypes (#8632)
This is now supported by upstream mono/linker
* [linker] Remove internal [NullablePublicOnly] attribute from apps (#8568)
I've only seen it with .net5 so far but it's better handled in master
and flow back into the branch
* [linker] Update custom attributes that can be removed (#8535)
Some are no longer part of the SDK (or converted into new ones
at build time), others were new (and missing).
A full list of attributes and their usage frequency in what we ship can
be seen in https://gist.github.com/spouliot/ca03c6da7d4d75670ca77749350eb8a2
Also update tests: no need to check for removals of stuff that does not
exists anymore.
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
* [AppKit] NSTextView allows passing nil to PasteAsPlainText and PasteAsRichText.
This is documented in Apple's documentation, their headers, and even proved
experimentally.
* Update xtro.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
instead use `Buffer.MemoryCopy`.
Currently only used from `CGDataProvider`. Added unit tests for the
public/indirect, usage of the API (we had none).
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@microsoft.com>
The test has started failign more commonly, but only in a number of old
OS versions and not all the time. The function should return a new
Pasteboard, but it is true that we are not releasing the old ones,
meaning that the pasteboard is left after the app is done as stated by
the apple documenation.
The test has been updated to:
1. Release the pasteboard.
2. Be inconclusive if the pasteboard service could not create a new one.
fixes: https://github.com/xamarin/xamarin-macios/issues/8787
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Note: affected tools are not included in the code shipping inside the SDK
Usage of deprecated cryptographic algorithms is not approved anymore,
even if not used for cryptographic purpose.
MD5 was used to create immutable GUID since it's output is 128bits which
is just what a GUID wants as it's input (16 bytes). The same can be
achieved (even if a bit slower) with a newer/longer hash function
https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1128148
As described in the issue, when the credentials for the base auth are set
to be empty the request reaches a timeout.
The inner issue is that when we have empty credentials the query that
is performed to the credentials db of the system throws an exception.
This exception is not raised to our code, but simply prints out in stderr.
When this situation occurs we get in an infinite loop in which we keep
sending the same credentials, we get another challenge, but the OS does
not do the right thing, we believe that we have to provide the creds
again (which are the same) and we get another challenge.. this goes on
an on until we reach the timeout.
Fortunately we know the number of failed challenges and the failure
response. With this information we can deduce if we already sent the
credentials, and if we did, do not set them again. In this case, the OS
sees no credentials in out delegate, uses the default handler and
correctly returns a 401.
fixes: https://github.com/xamarin/xamarin-macios/issues/8342
Fixes: https://github.com/xamarin/xamarin-macios/issues/8344
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
* Added nullability attributes to AVFoundation
* Revert AVPlayerItem FromAsset nullability attributes changes to keep existing tests passing and allow backwards compatibility
* Documents common-AVFoundation.ignore entry related to Foundation.NSNumber[] AVFoundation.AVVideoCompositionInstruction::get_RequiredSourceTrackIDs()
!extra-null-allowed! 'Foundation.NSNumber[] AVFoundation.AVVideoCompositionInstruction::get_RequiredSourceTrackIDs()' has a extraneous [NullAllowed] on return type
Co-authored-by: Cosmin Stirbu <stirbucosmin@gmail.com>
PR https://github.com/xamarin/xamarin-macios/pull/8184 removed the
inheritance with TextWriter, therefore the `as` will return null and we
will not generate the Html report. In this particular case, we do not
need an ILog, we just use it to get a path to the correct location,
therefore, we can create the file using the full path and pass it to the
xslt.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8364
VSMac has failing tests when they query the NSScrees.Screens property
which the following swift code shows that it can be executed in a diff
thread:
```swift
import Cocoa
import AppKit
DispatchQueue.global(qos: .background).async {
print("This is run on the background queue")
print(Thread.current)
var screens = NSScreen.screens
print (screens.count)
}
```
Fixes: https://github.com/xamarin/xamarin-macios/issues/8329
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Goals
* Reflect Apple nullability annotations in our bindings using C#8
* No warnings when building bindings
Non-Goals
* Update (add or fix) `[NullAllowed]` to match Apple headers (next phase)
* Make the generator or internal code fully nullable aware (`nowarn` is used)
Notes
* Apple's own annotations are not 100% accurate :(
* Where known issue exists we have _fixed_ our attributes to match reality :)
* We also do additional null-checks internally that might seems not required (better safe than sorry).
Enabling this will ensure that future bindings (and Xcode updates that
change nullability information) are spotted right away.
The binding fixes are **not** complete, i.e. what was done was mostly
to debug the xtro rule and find corner cases. The backlog will be
_ignored_ so the builds won't fail.
Threading is hard, part 2. We could have a situation in which two tasks
that are instantiated async use the same id. The id is later used to
build the logs directory etc.. When they share the id we end up in a
situation in which the logs overlap.
fixes: https://github.com/xamarin/xamarin-macios/issues/8146
The testing framework dlls come from mono and their paths are harcoded
in the templates. Move the hardcoded paths out to the assembly locator
which know were to find the paths.
This is needed to later allow the cmd to pass the location of the paths.
The assets were being used from the bcl-test directory. Move them as
part of the template. Renamed a file to make things a little easier when
adding the assets (a '.' in the middle of the name made this
complicated).
The template should be selfcotained.
- Code that will later be moved to the `dotnet/xharness` repo is first moved to an isolated project before it will be extracted to the new repo and NuGetified
- New project for the shared code and new test project for tests accompanying the moved code are created
- Only Factories are left behind that are used only by XHarness
The Generator and the Factory classes are a xamarin-macios thing.
Initially, they were separated because the code that generated the bcl
tests was not inside xharness. That is not longer the case. We can merge
both classes, generalize the namespace and be more prepared to move out
of the xamarin-macios repo.
Threading is hard, even with async. We reached a situation in which the
src code of the test applications from the template was being generated
more than once. Ideally, we could have a nice solution with
AsyncLazy<bool> and use the lazy async to copy the code, but it was
moved to netcore 5. The fix is a little uglier but valid, lock and
enure we only generate it once. Please note that you CANNOT use a lock
statement inside a async method.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8240
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
* [Harness] Remove the BCLTestProjectInfo class.
The class is not really needed. Move to a tuple, make the definition of
the group test project nicer since the dictionary initiallization makes
it cleaner.
Looks like we have cases in which we get empty nodes when listing
devices, which makes the parsing of the nodes fail and crash.
Move the code to a diff method, try to parse the data, if there is an
exception (missing data) return null. If null is returned, do not add
it.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8160
Looks like mono is not investing much time at the moment fixing
https://github.com/mono/mono/issues/18560
The above error happens always on 32b device tests. Added a new known
failure for the monitoring to use in their task so that they do not have
to goole the issue and they know that is knonw already.
Added a test in the xml parser that verifies that the parsing of the xml
is correct and that the correct number of failures is reported in the
human readable form.
fixes: https://github.com/xamarin/xamarin-macios/issues/8214
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
Ignore a failing test on devices. Related mono issue: https://github.com/mono/mono/issues/19308
Mono has to fix the issue, so we ignore in the meantime.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Refactor result parsing and test it. Make sure that all the possible
corner cases are taken into account. The changes are as follow:
1. Move all the logic to a new class, which will take care of the reustl
parsing.
2. Simplify the parsing method by splitting the noise code to other
functions. There is no need to parse several files in the same method.
3. Add tests that make sure that the correct result is returned when:
a. The app cannot launch.
b. The app timesout.
c. The app crashes.
The _nice_ thing about `NSURLProtocol` is that it does not force the use
of the network - but we inflicted this upon us on tests.
So this updates the test to avoid the network, which should make it
more reliable and not so much influenced by the URL used or the OS
versions where it's executed.
This does not cover as much callback as before. However this it does
test that the NSUrlProtocol is registered correctly (what we care) and
the rest is largely only related the the OS (which is not the target
of our unit tests).
Hopefully fix https://github.com/xamarin/maccore/issues/1956
xcrun can print stuff to stderr even if it succeeds and prints what it found
to stdout, which means we have to separate stdout and stderr when capturing
xcrun's output.
This also means that we need a few changes to our RunCommand API:
* Add support for capturing stdout and stderr separately by providing
different callbacks/StringBuilder instances.
* Use overloads instead of default arguments, because with default arguments
we end up with ambiguous matches in a lot of places with the new overloads.
Fixes https://github.com/xamarin/xamarin-macios/issues/8147.
Backport of #8150.
This is not a complete solution, because it doesn't work when running device
tests, because then we don't build locally. On the other hand, that bug has
never been a problem for device bots, so hopefully that won't change in the
future.
Ref: https://github.com/xamarin/maccore/issues/2177
* [mtouch/mmp] Only use stdout from xcrun as the path of the file that was found. Fixes#8147.
xcrun can print stuff to stderr even if it succeeds and prints what it found
to stdout, which means we have to separate stdout and stderr when capturing
xcrun's output.
This also means that we need a few changes to our RunCommand API:
* Add support for capturing stdout and stderr separately by providing
different callbacks/StringBuilder instances.
* Use overloads instead of default arguments, because with default arguments
we end up with ambiguous matches in a lot of places with the new overloads.
Fixes https://github.com/xamarin/xamarin-macios/issues/8147.
* [tests] Fix mmptest build.
* [mtouch/mmp] Fix process log output.
First step to try and clean up the psring of results and ensure that we
are doing the correct thing. We have several issue there. We want to
move all the complicated logic out of AppRunner and test the different
outputs and the full result matrix.
This commit first allows to use an instance class for the parsing, later
we will move things out. Step by step
XHarness TestAssemblyDefinition is very particular on the way we deal
with the mono test assemblies, in the general case, it only contains the
assembly name, but in the xamarin-macios it calculates the name of the
assembly using the platform.
Create an interface so that the command line was its own implementaiton
and move as much as possible to use the AssemblyLocator class. Later we
can add two simpler implementations for the more general cases.
Move the AssemblyLocator out of the Templates namespace since it makes
more sense.
* Unify target framework code between mtouch and mmp.
* Simplify the code in mmp: have three possible valid target frameworks for
most of code, and add special code to handle setting any other valid target
frameworks to redirect to one of those three valid target frameworks (and
warn if given any of those valid, but not "main", target frameworks). Any
other code can then depend on the target framework having exactly one of
those specific values, which means we can make IsUnified* variables
convenience properties instead.
* Unify a bit more of the argument parsing code between mtouch and mmp, since
that made a few other things easier.
* Add TargetFramework.IsValidFramework to have one validation implementation.
* Move the implementation of TargetFramework.MonoFrameworkDirectory to mmp
itself, it's not really related to the target framework.
* Remove Driver.IsUnified and IsClassic from mmp, they're not used anymore.
* Formally deprecate --xamarin-[full|system]-framework in mmp, they've really been deprecated for many years.
* Remove LinkerOptions.TargetFramework, it's not used anymore.
* Get rid of mmp's userTargetFramework fried, it's duplicated with the
targetFramework field.
* Add a few tests, and tweak others a bit.
Breaking changes:
* Both mtouch and mmp require --target-framework now. The only direct
consumers should be the MSBuild tasks, which already pass --target-framework
all the time. This simplifies code, and removes assumptions.
Initially the location of the test dlls of the downloaded artefact and the compiled mono were different. That was no longer the case and therefore there was a lot of code that could be removed since the wasDownloaded variable was set to be always true.
We remove the not needed code that will calculate a diff path if mono was compiled and use the AssemblyLocator class to point to the correct root path were test dlls will be found.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
At this point the BCLTestProjectGenerator class just represents a configuration class that knows about the project that are particular to xamarin-macios. All the logic that creates the projects has been moved to the XamariniOSTemplate which creates the project using the information passed by the config class.
At this stage, we have the ability to generate test projects by simply passing the info classes, which can be built either by a class (like the generator one) or a command line. We have fully decoupled the project generation from Xharness.
Tests have been added or moved to the correct location.
The name of the tests is picked up in a more inner node than
expected. Fwd the test name to make sure that noe all
crashes/timeouts/launches have the same name.
Add extra information when we have a timeout. We want to know the
name and the bot used to ensure that it is easier to identify patterns.
Timeout tests now have:
test title as: App Timeout {AppName} {Variation} on bot {device_name}
message: AppName} {Variation} Test run timed out after {timeout.TotalMinutes} minute(s) on bot {device_name}.
Build issues now have:
test title: App Build {projectTask.TestName} {projectTask.Variation}
That way we know the exact app that failed to build and the exact app that timeout and the device used to run it.
Move the project templates to be a resource in the dll. That way we can
move all the code outside of xharness and used it outside. This is not a
fill refactor but a first step to decouple the bcl test generation from
the xamarin-macios project.
XHarness should not see a difference, everythign works and simply makes
the dlls larger due to the new resources.
The code is going to take into account that the mono team will want to
use and unmanaged template with no dependencies on Xamarin.iOS and
Xamarin.Mac.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Move the project templates to be a resource in the dll. That way we can
move all the code outside of xharness and used it outside. This is not a
fill refactor but a first step to decouple the bcl test generation from
the xamarin-macios project.
XHarness should not see a difference, everythign works and simply makes
the dlls larger due to the new resources.
The code is going to take into account that the mono team will want to
use and unmanaged template with no dependencies on Xamarin.iOS and
Xamarin.Mac.
Add tests for the SimDevice and Simulators classes that ensure that the
process is correctly called and the xml with the simulators parsed and
returns the correct number of sims.
Update the generation of the xml to include a more descriptive title
than "AppCrash". After this commit the name of the testcase and the test
follow this pattern:
- "App Crash {ApplicationName} {Variation}"
This will make it easier to filter since the an example failure will be:
"App Crash Monotouch (Debug)"
Note, the '()' in the example are part of the variation and that is
build by xharness and passed to the runner.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8077
Moved the implementations under Hardare. Add the following to simplify
the testing of the AppRunner:
1. Add interfaces for the listing/loading of devices and simulators.
2. Add a special class for mlaunch args. That way we are sure that
arguments are correctly passed and we do not have typos.
3. Added tests for the new classes and for the device and devices
classes.
4. Added tests for the management of the TCC db.
Tests for the SimDevice and Simulators will come in a following PR. The
only reason for it is that I realize that the commit was getting to
large.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Moved the implementations under Hardware. Add the following to simplify
the testing of the AppRunner:
1. Add interfaces for the listing/loading of devices and simulators.
2. Add a special class for mlaunch args. That way we are sure that
arguments are correctly passed and we do not have typos.
3. Added tests for the new classes and for the device and devices
classes.
4. Added tests for the management of the TCC db.
Tests for the SimDevice and Simulators will come in a following PR. The
only reason for it is that I realize that the commit was getting to
large.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [Harness] Refactor process management to be testable.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
* Apply suggestions from code review
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
* Move out utils into a separate namespace
* Move Cache to the test project
* Fix namespaces after merge
* Remove unneeded code
* Move Target files
* Sort csproj
* Refactor Targets
* Rename Cache to TempDirectory
* Fix using
* Move ProjectFileExtensions
* Remove dead code
* Move Extensions
* Add empty StringUtilsTests
* Add StringUtils tests
* Revert refactorings
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Add better formatarguments test
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This also meant adding support for running .NET tests in xharness. Some
refactoring was done to extract common code to shared members, in order to
avoid duplicating a lot of code.
* [mtouch][mmp] Share error.cs and make it safer against FormatException
Beside the obvious code sharing [1] this makes the error reporting code
safer against `FormatException`.
Even with code review this happened a few times in the past - we get
a string that can't format correctly in some error handling code. This
means the `FormatException` must be fixed before we can see the
underlying root issue (delaying bug fixes and support).
Recent changes made for localization makes it harder to review the
original strings (and even harder for translated strings) for
correctness. This means we can end up with something like
```
String.Format ("{0} {1}", "foo");
``
and easily miss it (since they split between different files) if there
are several changes inside a PR.
Finally sprinkle some unit tests :)
[1] we should move everything to use `ProductException` but that will be
for another PR...
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Provide tests that will ensure that the Tcp listener and File listener
will work correctly. Tests ensure that both listeners receive the data
and write it in the ILog
The tcp test is an interesting one because it redirects the call of the
method to a diff callback that will write the data in a stream to check
the final result.
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
Using this option it's possible to test for the presence of a type
reference in both pre-linked and post-linked assemblies.
This makes it possible to detect if
* a 3rd party assemblies are using some specific type you would like to avoid;
* a type reference has been removed during the build (e.g. linker)
Notes:
* Custom attributes are encoded differently and not included in the assembly type references metadata.
* Assembly that define a type `X` do not have a reference (but the definition) of the type (and won't be reported).
If either the pre or post-linked warnings are not useful then it's possible
to add `-nowarn:150x` to exclude the results.
E.g.
* `-nowarn:1502` would not report references in pre-linked assemblies;
* `-nowarn:1503` would not report references in post-linked assemblies;
Finally `-warnaserror:150x` can be used to stop a build that would not
satisfy either the pre or post-linked condition.
* `-warnaserror:1502` would not report references in pre-linked assemblies;
* `-warnaserror:1503` would not report references in post-linked assemblies;
_side note_ this was first done on `d16-6` to ease backports as `master`
has multiple changes for localization. A similar PR will be done for
`master` once merged.
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
so if we optimize (to remove references) the tests will be ignored.
Doing this based on define was wrong since we have different (and more)
configurations executed on devices,
`Default` property was using a nil-handle which is incorrect since
* we don't allow that (this is generally a bad sign)
* it does not map to `OS_LOG_DEFAULT`
Since `Default` was assigned in the type (static) constructor then
the whole type became unusable :(
Header `log.h` shows that the right definition requires us to load a
field and use it.
```
define OS_LOG_DEFAULT OS_OBJECT_GLOBAL_OBJECT(os_log_t, _os_log_default)
```
While `NULL` can actually be used for disabled (not exposed) by this
contradicting (nullability-wise) macro
```
define OS_LOG_DISABLED ((os_log_t _Nonnull)NULL)
```
Also adds unit tests. A more general tests for `.cctor` will be added
to introspection tests in a separate PR.
Fixes https://github.com/xamarin/xamarin-macios/issues/7959
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
Some crashes are reported as tcp connection issues because they happen
before the app had the chance to write anything in the log. In that
case, we checked for the presence of the file, and if not present we
decided it was a tcp issue when it is not the case.
In this commit, we check for the tcp erorr message in the main_log so
that we are certain that the issue was with the connection and not
anyother.
Testingis simple, ran tests without the phone being part of the same
network and test with a branch that has a crash. For example the one in
https://github.com/xamarin/xamarin-macios/pull/8009 hash 83240612e8
* XAMARIN_MAC: Used once, replace with MONOMAC.
* UNIFIED: Use once elsewhere, replace with __UNIFIED__.
* OBJECT_REF_TRACKING: not used anywhere.
* XAMARIN_MODERN: not used anywhere.
* NET_2_0: not used anywhere.
Also update the README a bit.
* [msbuild] Provide the correct value for the operating system for tvOS and watchOS to a few tasks. Fixes#6200. (#7226)
The problem with #6200 was that we'd pass -mios-version-min=x.y to the metal
tool even for tvOS apps. This fixes it so that now pass -mtvos-version-min.
Fixes https://github.com/xamarin/xamarin-macios/issues/6200.
* [msbuild] Add support for Metal in the simulator. Fixes#7392. (#7983)
We support different outputs, lets add the avility for the caller to
decide which one to use. We default to NUnit V3 due to or dependency to
it in VSTS.
We do not longer compile the bcl tests, but we consume them as part or
mono. The old BCL targets and classes are not needed. As you can see,
the lists were empty.
Add tests to make sure that we can be safe when we make changes to that
part of the code.
Some methods are not tested but due to dependencies. Will work un
separating things a little to keep moving fwd.
This will spot cases like https://github.com/xamarin/xamarin-macios/issues/7959
where a type (static) constructor can fail at runtime, leading to
`TypeLoadException`
Ideally it's covered by it's own tests but it's better covered twice
than never :)
On iOS 64bits [1] simulator we hit some failures [2], later, if the
`.cctor` is executed. It's not a big deal to avoid those types since
we it will be executed on devices later.
[1] API not present on 32bits
[2] Fixing the following triggers similar failures for `DCDevice`
```
ApiClassPtrTest.VerifyClassPtr: class_ptr and RegisterAttribute are different: NFCIso15693CustomCommandConfiguration
Expected: 0
But was: 140735471513712
ApiSelectorTest.StaticMethods: 7 errors found in 2788 static selector validated:
CoreNFC.NFCIso15693ReaderSession : readingAvailable
CoreNFC.NFCNdefMessage : ndefMessageWithData:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithString:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithURL:
CoreNFC.NFCNdefPayload : wellKnownTypeTextPayloadWithString:locale:
CoreNFC.NFCNdefReaderSession : readingAvailable
CoreNFC.NFCReaderSession : readingAvailable
Expected: 0
But was: 7
```
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
This will spot cases like https://github.com/xamarin/xamarin-macios/issues/7959
where a type (static) constructor can fail at runtime, leading to
`TypeLoadException`
Ideally it's covered by it's own tests but it's better covered twice
than never :)
On iOS 64bits [1] simulator we hit some failures [2], later, if the
`.cctor` is executed. It's not a big deal to avoid those types since
we it will be executed on devices later.
[1] API not present on 32bits
[2] Fixing the following triggers similar failures for `DCDevice`
```
ApiClassPtrTest.VerifyClassPtr: class_ptr and RegisterAttribute are different: NFCIso15693CustomCommandConfiguration
Expected: 0
But was: 140735471513712
ApiSelectorTest.StaticMethods: 7 errors found in 2788 static selector validated:
CoreNFC.NFCIso15693ReaderSession : readingAvailable
CoreNFC.NFCNdefMessage : ndefMessageWithData:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithString:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithURL:
CoreNFC.NFCNdefPayload : wellKnownTypeTextPayloadWithString:locale:
CoreNFC.NFCNdefReaderSession : readingAvailable
CoreNFC.NFCReaderSession : readingAvailable
Expected: 0
But was: 7
```
We initially added a cmd to do the bcl test import generation that has
never been used.
We are moving the bcl test importer code to xharness and creating a
NUnit test project to run the tests that we already added. Unit tests
pass.
It will be nice to be able to have a smaller set of tests to be ran by
xharness. In this commit we add a set of labels to be able to divide the
test in the following groups:
* Xamarin - all tests BUT monotouch.
* Monotouch - just monotouch tests.
* Old BCL - The BCL tests that are based on NUnit
* New BCL - The BCL tests that are based on xUnit.
* mscorlib - Just the test that exercise mscorlib (and the different
groups, mscorlib 1, mscorlib 2 etc..)
This will allow to parallelize the execution of the full test suit in
different agents in VSTS.
`Default` property was using a nil-handle which is incorrect since
* we don't allow that (this is generally a bad sign)
* it does not map to `OS_LOG_DEFAULT`
Since `Default` was assigned in the type (static) constructor then
the whole type became unusable :(
Header `log.h` shows that the right definition requires us to load a
field and use it.
```
define OS_LOG_DEFAULT OS_OBJECT_GLOBAL_OBJECT(os_log_t, _os_log_default)
```
While `NULL` can actually be used for disabled (not exposed) by this
contradicting (nullability-wise) macro
```
define OS_LOG_DISABLED ((os_log_t _Nonnull)NULL)
```
Also adds unit tests. A more general tests for `.cctor` will be added
to introspection tests in a separate PR.
Fixes https://github.com/xamarin/xamarin-macios/issues/7959
Using this option it's possible to test for the presence of a type
reference in both pre-linked and post-linked assemblies.
This makes it possible to detect if
* a 3rd party assemblies are using some specific type you would like to avoid;
* a type reference has been removed during the build (e.g. linker)
Notes:
* Custom attributes are encoded differently and not included in the assembly type references metadata.
* Assembly that define a type `X` do not have a reference (but the definition) of the type (and won't be reported).
If either the pre or post-linked warnings are not useful then it's possible
to add `-nowarn:150x` to exclude the results.
E.g.
* `-nowarn:1502` would not report references in pre-linked assemblies;
* `-nowarn:1503` would not report references in post-linked assemblies;
Finally `-warnaserror:150x` can be used to stop a build that would not
satisfy either the pre or post-linked condition.
* `-warnaserror:1502` would not report references in pre-linked assemblies;
* `-warnaserror:1503` would not report references in post-linked assemblies;
_side note_ same as https://github.com/xamarin/xamarin-macios/pull/7925
except that this one uses the localized mtouch/mmp errors only in master (so far)
We can end up with a format that has `{x}` inside it and no argument. If
this is logged (which depends on verbosity level) then we end up with an
uncaught exceptions and either `MT0000` or `MM0000` errors.
This happens in #7904 due to other, unrelated, issues (a variable is not
expanded) but this can happen in other circumstances, e.g. a file could
be named `{x}.cs`.
The easy fix is to use the right `Console.WriteLine` overload if there
are no arguments provided. This is always good since it avoid an non
required call to `String.Format`.
Fixes https://github.com/xamarin/xamarin-macios/issues/7904
Xml should be:
```
<failure>
<message>Foo</message>
<stack-trace>Bar</stack-trace>
</failure>
```
But we generate:
```
<failure>
<message>Foo
<stack-trace>Bar</stack-trace>
</message>
</failure>
```
Makes the parsing of the failures impossible.
In order to simplify the monitoring job add the device name to the
following failures:
* Installation
* Launch
* Tcp Connection
All the above are most of the time due to a misconfigured device. The
device name is useful information for the monitoring person to be able
to reach IT and address the issue.
The parsing code set the crashed variable, but it was not set as an
output variable, that meant that the value was not used. Further in the
code we use the variable to decide if we had a crash or not. Most of the
time, there is no probel since if we have a real crash, we will get the
crash reports and to the right thing, but in the case of tcp connection
issues we do not have them, and therefore the crash xml result is not
created.
Bonus: the local logs variable was hidding a variable in another scope,
fixed that.
xharness does not only allow to run tests in CI, but also helps to run
specific project. That feature is used for the mtouch tests. In that
scenario, we do not have a build task and therefore we will not have the
build logs.
fixes: https://github.com/xamarin/maccore/issues/2154
If we are in VSTS and we want to have correct reportings we need to
generate a test failure when the applicaiton times out. If not, we have
a missmatch between the results from xharness (we have failures) and
VSTS (success).
On devices that cannot reach the host via TCP we do not have a log, this
means that in the if statement needs to have a case for it.
The main problem is that when the device cannot connect to the host, we
do not get a log OR a crash reason from the crash logs. It makes sense
not to have a crash reason, because the app did not crash. In these
sitations, we have to create a xml crash report (since we really do not
know if we can parse the file) that will tell vsts that there was an
issue. Adding the main log will let the monitoring person see the
results of the test run.
Because the PublishTest task is taking a regular expression, it is
importing the test results more than one, which gives wrong stats. Add a
prefix to better filter those files we are interested in (thos with
attachments).
As a bonus, refactored the xml failure code for less copy pasting and to
have a single place where we had to add the prefix.
VSTS does not provide a good way to report an app installation issue,
but we can fake a failure in the test when the installation does not
happen.
In the case of the installation failure a xml test result is generated
that will expose all the required information and will attach all the
needed logs (install logs). An example of the generated result can be
seen here: https://gist.github.com/mandel-macaque/2274bcd8785eebd636b98142e228afa9
Ignore Xamarin.MMP.Tests.WarningTests.MM0135 on macOS 10.15+, because this
test requires Xcode 9.4, which doesn't work on macOS 10.15+.
Fixes https://github.com/xamarin/maccore/issues/2035.
The NUnit3 xml requires an id for the tests, but xunit does not provide
one. Add an extension object to the xslt that will return a very lame id
so that the test uploader does not complain.
fixes: https://github.com/xamarin/xamarin-macios/issues/7888
Sebastien Pouliot
Manuel de la Pena
monojenkins
Rolf Bjarne Kvinge
Alex Soto
Přemek Vysoký
Chris Hamons
Whitney Schmidt