The testing framework dlls come from mono and their paths are harcoded
in the templates. Move the hardcoded paths out to the assembly locator
which know were to find the paths.
This is needed to later allow the cmd to pass the location of the paths.
The assets were being used from the bcl-test directory. Move them as
part of the template. Renamed a file to make things a little easier when
adding the assets (a '.' in the middle of the name made this
complicated).
The template should be selfcotained.
- Code that will later be moved to the `dotnet/xharness` repo is first moved to an isolated project before it will be extracted to the new repo and NuGetified
- New project for the shared code and new test project for tests accompanying the moved code are created
- Only Factories are left behind that are used only by XHarness
The Generator and the Factory classes are a xamarin-macios thing.
Initially, they were separated because the code that generated the bcl
tests was not inside xharness. That is not longer the case. We can merge
both classes, generalize the namespace and be more prepared to move out
of the xamarin-macios repo.
Threading is hard, even with async. We reached a situation in which the
src code of the test applications from the template was being generated
more than once. Ideally, we could have a nice solution with
AsyncLazy<bool> and use the lazy async to copy the code, but it was
moved to netcore 5. The fix is a little uglier but valid, lock and
enure we only generate it once. Please note that you CANNOT use a lock
statement inside a async method.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8240
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
* [Harness] Remove the BCLTestProjectInfo class.
The class is not really needed. Move to a tuple, make the definition of
the group test project nicer since the dictionary initiallization makes
it cleaner.
Looks like we have cases in which we get empty nodes when listing
devices, which makes the parsing of the nodes fail and crash.
Move the code to a diff method, try to parse the data, if there is an
exception (missing data) return null. If null is returned, do not add
it.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8160
Looks like mono is not investing much time at the moment fixing
https://github.com/mono/mono/issues/18560
The above error happens always on 32b device tests. Added a new known
failure for the monitoring to use in their task so that they do not have
to goole the issue and they know that is knonw already.
Added a test in the xml parser that verifies that the parsing of the xml
is correct and that the correct number of failures is reported in the
human readable form.
fixes: https://github.com/xamarin/xamarin-macios/issues/8214
Remove two classes that are not really needed and a method. We moved to
named tuples to make things simpler. Unfortunally C# does not support:
```csharp
using MyNamedTuple = (string Name, double value, bool answer);
```
But we have to live with this. Makes the namespace simpler and removes
confusion with the already present project classes that are more widely
used.
Ignore a failing test on devices. Related mono issue: https://github.com/mono/mono/issues/19308
Mono has to fix the issue, so we ignore in the meantime.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Refactor result parsing and test it. Make sure that all the possible
corner cases are taken into account. The changes are as follow:
1. Move all the logic to a new class, which will take care of the reustl
parsing.
2. Simplify the parsing method by splitting the noise code to other
functions. There is no need to parse several files in the same method.
3. Add tests that make sure that the correct result is returned when:
a. The app cannot launch.
b. The app timesout.
c. The app crashes.
The _nice_ thing about `NSURLProtocol` is that it does not force the use
of the network - but we inflicted this upon us on tests.
So this updates the test to avoid the network, which should make it
more reliable and not so much influenced by the URL used or the OS
versions where it's executed.
This does not cover as much callback as before. However this it does
test that the NSUrlProtocol is registered correctly (what we care) and
the rest is largely only related the the OS (which is not the target
of our unit tests).
Hopefully fix https://github.com/xamarin/maccore/issues/1956
xcrun can print stuff to stderr even if it succeeds and prints what it found
to stdout, which means we have to separate stdout and stderr when capturing
xcrun's output.
This also means that we need a few changes to our RunCommand API:
* Add support for capturing stdout and stderr separately by providing
different callbacks/StringBuilder instances.
* Use overloads instead of default arguments, because with default arguments
we end up with ambiguous matches in a lot of places with the new overloads.
Fixes https://github.com/xamarin/xamarin-macios/issues/8147.
Backport of #8150.
This is not a complete solution, because it doesn't work when running device
tests, because then we don't build locally. On the other hand, that bug has
never been a problem for device bots, so hopefully that won't change in the
future.
Ref: https://github.com/xamarin/maccore/issues/2177
* [mtouch/mmp] Only use stdout from xcrun as the path of the file that was found. Fixes#8147.
xcrun can print stuff to stderr even if it succeeds and prints what it found
to stdout, which means we have to separate stdout and stderr when capturing
xcrun's output.
This also means that we need a few changes to our RunCommand API:
* Add support for capturing stdout and stderr separately by providing
different callbacks/StringBuilder instances.
* Use overloads instead of default arguments, because with default arguments
we end up with ambiguous matches in a lot of places with the new overloads.
Fixes https://github.com/xamarin/xamarin-macios/issues/8147.
* [tests] Fix mmptest build.
* [mtouch/mmp] Fix process log output.
First step to try and clean up the psring of results and ensure that we
are doing the correct thing. We have several issue there. We want to
move all the complicated logic out of AppRunner and test the different
outputs and the full result matrix.
This commit first allows to use an instance class for the parsing, later
we will move things out. Step by step
XHarness TestAssemblyDefinition is very particular on the way we deal
with the mono test assemblies, in the general case, it only contains the
assembly name, but in the xamarin-macios it calculates the name of the
assembly using the platform.
Create an interface so that the command line was its own implementaiton
and move as much as possible to use the AssemblyLocator class. Later we
can add two simpler implementations for the more general cases.
Move the AssemblyLocator out of the Templates namespace since it makes
more sense.
* Unify target framework code between mtouch and mmp.
* Simplify the code in mmp: have three possible valid target frameworks for
most of code, and add special code to handle setting any other valid target
frameworks to redirect to one of those three valid target frameworks (and
warn if given any of those valid, but not "main", target frameworks). Any
other code can then depend on the target framework having exactly one of
those specific values, which means we can make IsUnified* variables
convenience properties instead.
* Unify a bit more of the argument parsing code between mtouch and mmp, since
that made a few other things easier.
* Add TargetFramework.IsValidFramework to have one validation implementation.
* Move the implementation of TargetFramework.MonoFrameworkDirectory to mmp
itself, it's not really related to the target framework.
* Remove Driver.IsUnified and IsClassic from mmp, they're not used anymore.
* Formally deprecate --xamarin-[full|system]-framework in mmp, they've really been deprecated for many years.
* Remove LinkerOptions.TargetFramework, it's not used anymore.
* Get rid of mmp's userTargetFramework fried, it's duplicated with the
targetFramework field.
* Add a few tests, and tweak others a bit.
Breaking changes:
* Both mtouch and mmp require --target-framework now. The only direct
consumers should be the MSBuild tasks, which already pass --target-framework
all the time. This simplifies code, and removes assumptions.
Initially the location of the test dlls of the downloaded artefact and the compiled mono were different. That was no longer the case and therefore there was a lot of code that could be removed since the wasDownloaded variable was set to be always true.
We remove the not needed code that will calculate a diff path if mono was compiled and use the AssemblyLocator class to point to the correct root path were test dlls will be found.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
At this point the BCLTestProjectGenerator class just represents a configuration class that knows about the project that are particular to xamarin-macios. All the logic that creates the projects has been moved to the XamariniOSTemplate which creates the project using the information passed by the config class.
At this stage, we have the ability to generate test projects by simply passing the info classes, which can be built either by a class (like the generator one) or a command line. We have fully decoupled the project generation from Xharness.
Tests have been added or moved to the correct location.
The name of the tests is picked up in a more inner node than
expected. Fwd the test name to make sure that noe all
crashes/timeouts/launches have the same name.
Add extra information when we have a timeout. We want to know the
name and the bot used to ensure that it is easier to identify patterns.
Timeout tests now have:
test title as: App Timeout {AppName} {Variation} on bot {device_name}
message: AppName} {Variation} Test run timed out after {timeout.TotalMinutes} minute(s) on bot {device_name}.
Build issues now have:
test title: App Build {projectTask.TestName} {projectTask.Variation}
That way we know the exact app that failed to build and the exact app that timeout and the device used to run it.
Move the project templates to be a resource in the dll. That way we can
move all the code outside of xharness and used it outside. This is not a
fill refactor but a first step to decouple the bcl test generation from
the xamarin-macios project.
XHarness should not see a difference, everythign works and simply makes
the dlls larger due to the new resources.
The code is going to take into account that the mono team will want to
use and unmanaged template with no dependencies on Xamarin.iOS and
Xamarin.Mac.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Move the project templates to be a resource in the dll. That way we can
move all the code outside of xharness and used it outside. This is not a
fill refactor but a first step to decouple the bcl test generation from
the xamarin-macios project.
XHarness should not see a difference, everythign works and simply makes
the dlls larger due to the new resources.
The code is going to take into account that the mono team will want to
use and unmanaged template with no dependencies on Xamarin.iOS and
Xamarin.Mac.
Add tests for the SimDevice and Simulators classes that ensure that the
process is correctly called and the xml with the simulators parsed and
returns the correct number of sims.
Update the generation of the xml to include a more descriptive title
than "AppCrash". After this commit the name of the testcase and the test
follow this pattern:
- "App Crash {ApplicationName} {Variation}"
This will make it easier to filter since the an example failure will be:
"App Crash Monotouch (Debug)"
Note, the '()' in the example are part of the variation and that is
build by xharness and passed to the runner.
Fixes: https://github.com/xamarin/xamarin-macios/issues/8077
Moved the implementations under Hardare. Add the following to simplify
the testing of the AppRunner:
1. Add interfaces for the listing/loading of devices and simulators.
2. Add a special class for mlaunch args. That way we are sure that
arguments are correctly passed and we do not have typos.
3. Added tests for the new classes and for the device and devices
classes.
4. Added tests for the management of the TCC db.
Tests for the SimDevice and Simulators will come in a following PR. The
only reason for it is that I realize that the commit was getting to
large.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Moved the implementations under Hardware. Add the following to simplify
the testing of the AppRunner:
1. Add interfaces for the listing/loading of devices and simulators.
2. Add a special class for mlaunch args. That way we are sure that
arguments are correctly passed and we do not have typos.
3. Added tests for the new classes and for the device and devices
classes.
4. Added tests for the management of the TCC db.
Tests for the SimDevice and Simulators will come in a following PR. The
only reason for it is that I realize that the commit was getting to
large.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* [Harness] Refactor process management to be testable.
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
* Apply suggestions from code review
Co-Authored-By: Přemek Vysoký <premek.vysoky@microsoft.com>
* Move out utils into a separate namespace
* Move Cache to the test project
* Fix namespaces after merge
* Remove unneeded code
* Move Target files
* Sort csproj
* Refactor Targets
* Rename Cache to TempDirectory
* Fix using
* Move ProjectFileExtensions
* Remove dead code
* Move Extensions
* Add empty StringUtilsTests
* Add StringUtils tests
* Revert refactorings
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Add better formatarguments test
* Update tests/xharness/Utilities/StringUtils.cs
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Premek Vysoky <prvysoky@microsoft.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This also meant adding support for running .NET tests in xharness. Some
refactoring was done to extract common code to shared members, in order to
avoid duplicating a lot of code.
* [mtouch][mmp] Share error.cs and make it safer against FormatException
Beside the obvious code sharing [1] this makes the error reporting code
safer against `FormatException`.
Even with code review this happened a few times in the past - we get
a string that can't format correctly in some error handling code. This
means the `FormatException` must be fixed before we can see the
underlying root issue (delaying bug fixes and support).
Recent changes made for localization makes it harder to review the
original strings (and even harder for translated strings) for
correctness. This means we can end up with something like
```
String.Format ("{0} {1}", "foo");
``
and easily miss it (since they split between different files) if there
are several changes inside a PR.
Finally sprinkle some unit tests :)
[1] we should move everything to use `ProductException` but that will be
for another PR...
Move all the extension methods to a class. After this refactor, we will
be able to DI the manager in the other classes and assert that the
processes are called with the correct parameters without the need of
launching them.
Also added tests for the manager. We create a dummy console app that
will be executed by the tests. The console app has a number of
parameters that will be used to ensure that the new process behaves as
we want:
- Use the passed exit code.
- Create child proecesses if needed.
- Sleep to force a timeout.
- Writer messages to stdout and stderr.
Our tests call the dummy app and ensures that the results match the
behaviour expected by the dummy app.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
Provide tests that will ensure that the Tcp listener and File listener
will work correctly. Tests ensure that both listeners receive the data
and write it in the ILog
The tcp test is an interesting one because it redirects the call of the
method to a diff callback that will write the data in a stream to check
the final result.
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
Using this option it's possible to test for the presence of a type
reference in both pre-linked and post-linked assemblies.
This makes it possible to detect if
* a 3rd party assemblies are using some specific type you would like to avoid;
* a type reference has been removed during the build (e.g. linker)
Notes:
* Custom attributes are encoded differently and not included in the assembly type references metadata.
* Assembly that define a type `X` do not have a reference (but the definition) of the type (and won't be reported).
If either the pre or post-linked warnings are not useful then it's possible
to add `-nowarn:150x` to exclude the results.
E.g.
* `-nowarn:1502` would not report references in pre-linked assemblies;
* `-nowarn:1503` would not report references in post-linked assemblies;
Finally `-warnaserror:150x` can be used to stop a build that would not
satisfy either the pre or post-linked condition.
* `-warnaserror:1502` would not report references in pre-linked assemblies;
* `-warnaserror:1503` would not report references in post-linked assemblies;
_side note_ this was first done on `d16-6` to ease backports as `master`
has multiple changes for localization. A similar PR will be done for
`master` once merged.
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
Co-authored-by: Přemek Vysoký <premek.vysoky@microsoft.com>
so if we optimize (to remove references) the tests will be ignored.
Doing this based on define was wrong since we have different (and more)
configurations executed on devices,
`Default` property was using a nil-handle which is incorrect since
* we don't allow that (this is generally a bad sign)
* it does not map to `OS_LOG_DEFAULT`
Since `Default` was assigned in the type (static) constructor then
the whole type became unusable :(
Header `log.h` shows that the right definition requires us to load a
field and use it.
```
define OS_LOG_DEFAULT OS_OBJECT_GLOBAL_OBJECT(os_log_t, _os_log_default)
```
While `NULL` can actually be used for disabled (not exposed) by this
contradicting (nullability-wise) macro
```
define OS_LOG_DISABLED ((os_log_t _Nonnull)NULL)
```
Also adds unit tests. A more general tests for `.cctor` will be added
to introspection tests in a separate PR.
Fixes https://github.com/xamarin/xamarin-macios/issues/7959
Move to use interfaces, that will let us later add tests that will
verify that all the correct logging is performed. As an example, added a
test for XmlResultParser that ensures that the failures are correctly
generated. The test uses Moq to pass the different paths to be used and
later be able to verify the wirtten xml.
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
Some crashes are reported as tcp connection issues because they happen
before the app had the chance to write anything in the log. In that
case, we checked for the presence of the file, and if not present we
decided it was a tcp issue when it is not the case.
In this commit, we check for the tcp erorr message in the main_log so
that we are certain that the issue was with the connection and not
anyother.
Testingis simple, ran tests without the phone being part of the same
network and test with a branch that has a crash. For example the one in
https://github.com/xamarin/xamarin-macios/pull/8009 hash 83240612e8
* XAMARIN_MAC: Used once, replace with MONOMAC.
* UNIFIED: Use once elsewhere, replace with __UNIFIED__.
* OBJECT_REF_TRACKING: not used anywhere.
* XAMARIN_MODERN: not used anywhere.
* NET_2_0: not used anywhere.
Also update the README a bit.
* [msbuild] Provide the correct value for the operating system for tvOS and watchOS to a few tasks. Fixes#6200. (#7226)
The problem with #6200 was that we'd pass -mios-version-min=x.y to the metal
tool even for tvOS apps. This fixes it so that now pass -mtvos-version-min.
Fixes https://github.com/xamarin/xamarin-macios/issues/6200.
* [msbuild] Add support for Metal in the simulator. Fixes#7392. (#7983)
We support different outputs, lets add the avility for the caller to
decide which one to use. We default to NUnit V3 due to or dependency to
it in VSTS.
We do not longer compile the bcl tests, but we consume them as part or
mono. The old BCL targets and classes are not needed. As you can see,
the lists were empty.
Add tests to make sure that we can be safe when we make changes to that
part of the code.
Some methods are not tested but due to dependencies. Will work un
separating things a little to keep moving fwd.
This will spot cases like https://github.com/xamarin/xamarin-macios/issues/7959
where a type (static) constructor can fail at runtime, leading to
`TypeLoadException`
Ideally it's covered by it's own tests but it's better covered twice
than never :)
On iOS 64bits [1] simulator we hit some failures [2], later, if the
`.cctor` is executed. It's not a big deal to avoid those types since
we it will be executed on devices later.
[1] API not present on 32bits
[2] Fixing the following triggers similar failures for `DCDevice`
```
ApiClassPtrTest.VerifyClassPtr: class_ptr and RegisterAttribute are different: NFCIso15693CustomCommandConfiguration
Expected: 0
But was: 140735471513712
ApiSelectorTest.StaticMethods: 7 errors found in 2788 static selector validated:
CoreNFC.NFCIso15693ReaderSession : readingAvailable
CoreNFC.NFCNdefMessage : ndefMessageWithData:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithString:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithURL:
CoreNFC.NFCNdefPayload : wellKnownTypeTextPayloadWithString:locale:
CoreNFC.NFCNdefReaderSession : readingAvailable
CoreNFC.NFCReaderSession : readingAvailable
Expected: 0
But was: 7
```
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
This will spot cases like https://github.com/xamarin/xamarin-macios/issues/7959
where a type (static) constructor can fail at runtime, leading to
`TypeLoadException`
Ideally it's covered by it's own tests but it's better covered twice
than never :)
On iOS 64bits [1] simulator we hit some failures [2], later, if the
`.cctor` is executed. It's not a big deal to avoid those types since
we it will be executed on devices later.
[1] API not present on 32bits
[2] Fixing the following triggers similar failures for `DCDevice`
```
ApiClassPtrTest.VerifyClassPtr: class_ptr and RegisterAttribute are different: NFCIso15693CustomCommandConfiguration
Expected: 0
But was: 140735471513712
ApiSelectorTest.StaticMethods: 7 errors found in 2788 static selector validated:
CoreNFC.NFCIso15693ReaderSession : readingAvailable
CoreNFC.NFCNdefMessage : ndefMessageWithData:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithString:
CoreNFC.NFCNdefPayload : wellKnownTypeURIPayloadWithURL:
CoreNFC.NFCNdefPayload : wellKnownTypeTextPayloadWithString:locale:
CoreNFC.NFCNdefReaderSession : readingAvailable
CoreNFC.NFCReaderSession : readingAvailable
Expected: 0
But was: 7
```
We initially added a cmd to do the bcl test import generation that has
never been used.
We are moving the bcl test importer code to xharness and creating a
NUnit test project to run the tests that we already added. Unit tests
pass.
It will be nice to be able to have a smaller set of tests to be ran by
xharness. In this commit we add a set of labels to be able to divide the
test in the following groups:
* Xamarin - all tests BUT monotouch.
* Monotouch - just monotouch tests.
* Old BCL - The BCL tests that are based on NUnit
* New BCL - The BCL tests that are based on xUnit.
* mscorlib - Just the test that exercise mscorlib (and the different
groups, mscorlib 1, mscorlib 2 etc..)
This will allow to parallelize the execution of the full test suit in
different agents in VSTS.
`Default` property was using a nil-handle which is incorrect since
* we don't allow that (this is generally a bad sign)
* it does not map to `OS_LOG_DEFAULT`
Since `Default` was assigned in the type (static) constructor then
the whole type became unusable :(
Header `log.h` shows that the right definition requires us to load a
field and use it.
```
define OS_LOG_DEFAULT OS_OBJECT_GLOBAL_OBJECT(os_log_t, _os_log_default)
```
While `NULL` can actually be used for disabled (not exposed) by this
contradicting (nullability-wise) macro
```
define OS_LOG_DISABLED ((os_log_t _Nonnull)NULL)
```
Also adds unit tests. A more general tests for `.cctor` will be added
to introspection tests in a separate PR.
Fixes https://github.com/xamarin/xamarin-macios/issues/7959
Using this option it's possible to test for the presence of a type
reference in both pre-linked and post-linked assemblies.
This makes it possible to detect if
* a 3rd party assemblies are using some specific type you would like to avoid;
* a type reference has been removed during the build (e.g. linker)
Notes:
* Custom attributes are encoded differently and not included in the assembly type references metadata.
* Assembly that define a type `X` do not have a reference (but the definition) of the type (and won't be reported).
If either the pre or post-linked warnings are not useful then it's possible
to add `-nowarn:150x` to exclude the results.
E.g.
* `-nowarn:1502` would not report references in pre-linked assemblies;
* `-nowarn:1503` would not report references in post-linked assemblies;
Finally `-warnaserror:150x` can be used to stop a build that would not
satisfy either the pre or post-linked condition.
* `-warnaserror:1502` would not report references in pre-linked assemblies;
* `-warnaserror:1503` would not report references in post-linked assemblies;
_side note_ same as https://github.com/xamarin/xamarin-macios/pull/7925
except that this one uses the localized mtouch/mmp errors only in master (so far)
We can end up with a format that has `{x}` inside it and no argument. If
this is logged (which depends on verbosity level) then we end up with an
uncaught exceptions and either `MT0000` or `MM0000` errors.
This happens in #7904 due to other, unrelated, issues (a variable is not
expanded) but this can happen in other circumstances, e.g. a file could
be named `{x}.cs`.
The easy fix is to use the right `Console.WriteLine` overload if there
are no arguments provided. This is always good since it avoid an non
required call to `String.Format`.
Fixes https://github.com/xamarin/xamarin-macios/issues/7904
Xml should be:
```
<failure>
<message>Foo</message>
<stack-trace>Bar</stack-trace>
</failure>
```
But we generate:
```
<failure>
<message>Foo
<stack-trace>Bar</stack-trace>
</message>
</failure>
```
Makes the parsing of the failures impossible.
In order to simplify the monitoring job add the device name to the
following failures:
* Installation
* Launch
* Tcp Connection
All the above are most of the time due to a misconfigured device. The
device name is useful information for the monitoring person to be able
to reach IT and address the issue.
The parsing code set the crashed variable, but it was not set as an
output variable, that meant that the value was not used. Further in the
code we use the variable to decide if we had a crash or not. Most of the
time, there is no probel since if we have a real crash, we will get the
crash reports and to the right thing, but in the case of tcp connection
issues we do not have them, and therefore the crash xml result is not
created.
Bonus: the local logs variable was hidding a variable in another scope,
fixed that.
xharness does not only allow to run tests in CI, but also helps to run
specific project. That feature is used for the mtouch tests. In that
scenario, we do not have a build task and therefore we will not have the
build logs.
fixes: https://github.com/xamarin/maccore/issues/2154
If we are in VSTS and we want to have correct reportings we need to
generate a test failure when the applicaiton times out. If not, we have
a missmatch between the results from xharness (we have failures) and
VSTS (success).
On devices that cannot reach the host via TCP we do not have a log, this
means that in the if statement needs to have a case for it.
The main problem is that when the device cannot connect to the host, we
do not get a log OR a crash reason from the crash logs. It makes sense
not to have a crash reason, because the app did not crash. In these
sitations, we have to create a xml crash report (since we really do not
know if we can parse the file) that will tell vsts that there was an
issue. Adding the main log will let the monitoring person see the
results of the test run.
Because the PublishTest task is taking a regular expression, it is
importing the test results more than one, which gives wrong stats. Add a
prefix to better filter those files we are interested in (thos with
attachments).
As a bonus, refactored the xml failure code for less copy pasting and to
have a single place where we had to add the prefix.
VSTS does not provide a good way to report an app installation issue,
but we can fake a failure in the test when the installation does not
happen.
In the case of the installation failure a xml test result is generated
that will expose all the required information and will attach all the
needed logs (install logs). An example of the generated result can be
seen here: https://gist.github.com/mandel-macaque/2274bcd8785eebd636b98142e228afa9
Ignore Xamarin.MMP.Tests.WarningTests.MM0135 on macOS 10.15+, because this
test requires Xcode 9.4, which doesn't work on macOS 10.15+.
Fixes https://github.com/xamarin/maccore/issues/2035.
The NUnit3 xml requires an id for the tests, but xunit does not provide
one. Add an extension object to the xslt that will return a very lame id
so that the test uploader does not complain.
fixes: https://github.com/xamarin/xamarin-macios/issues/7888
Don't remove the entire script, because I believe there's code out there that
checks for the existence of the smcs script to determine whether Xamarin.iOS
is installed or not.
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
* [xcode11.4] Add xcode 11.4 b1 initial support
* [xtro] re-enable PDFKit
* Disable watchOS and fix xtro
Unfortunately watchOS simulator hangs when we try to deploy to it
and it keeps our tests timing out. Disabling for now until we
can investigate more.
Disables PDFKit on xtro in macOS
* [jenkins] Switch to use the catalina bot group (#7819)
* Bump maccore to get fix for launching the simulator for watch apps.
New commits in xamarin/maccore:
* xamarin/maccore@546270c8f9 [Xamarin.Hosting] Fix the name of the notification we get when the simulator has launched. (#2145)
Diff: 55957e908d..546270c8f9
* [tests] Diable watch due to time out, enable 10,15,4 in intro, fix min version
* Bump macios-binaries to get updated binary mlaunch as well.
New commits in xamarin/macios-binaries:
* xamarin/macios-binaries@f8c6e63 Bump mlaunch to xamarin/maccore@546270c8f9
Diff: eb6980e8b6..f8c6e63228
* [msbuild] Reflect ibtool changes in our tests
Looks like Apple reverted some changes introduces in Xcode 11
in ibtool, for more context see xamarin/xamarin-macios#6970
* [mtouch] Workaround strange behavior of realpath.
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
As per the documentation of the VSTS test uploader (https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#attachments-support)
the tests support the addition of attachments. We want to be able to
access the logs of the different tests runs, this is achieved the
following way:
1. Move TouchUnit to use NUnit V3 which allows to add attachments.
2. Once the tests are ran, add the attachments to the following nodes:
1. The very first test-suite. This will allow to have the logs for
succesul tests.
2. Add logs to failing tests. Reduces the number of clicks to be done
to access to the logs when a test case fails.
3. Modify the assembly name of the test-suit to match the name of the
application. This ensures two things.
1. We have a consistent name for the file column in VSTS, that can be
used to see recurrent failing tests.
2. The name is more readable, since if not, it will contain the UUID
of the device.
Logs are not added to succesful tests because it will have the following
problems:
* Larger data storage usage.
* Longer upload time. The addtion of the logs per tests (succesful or
failed) was tested and resulted in an upload time LONGER than 6 hours
for all TouchUnit, NUnit blc tests and xUnit bcl tests.
In order for this to be useful, the task in the pipeline SHOULD NOT
merge test runs. We should have a test run PER application so that we do
not mix the logs.
Co-Authored-By: Rolf Bjarne Kvinge <rolf@xamarin.com>
One of the reason why we lost the TestReport was because the xml added
from the new code did not use the same name for the log as the expected
one (https://github.com/xamarin/xamarin-macios/pull/7835/files#diff-03773861bcef485bf343300f31b60b0eR374).
If we are going to be using the description as a way to decide what is
going to be done with it, lets use static vars so that we do not have
logs added with the wrong name.
Add code to write the TestReport when we have xunit results. Also
refactored code for the NUnit format to not use LINQ, it will use less
memory and in the future we can move to async (not now since it might
raise other problems).
fixes: https://github.com/xamarin/xamarin-macios/issues/7826
Only one to start... it's been discussed before but we generally
found other ways to do them. Let's continue to pick the best place
but we now have more options :)
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
Use the xml parsing helper methods to decide if the xml can be parsed
and if we will be able to generate the report. That way we avoid an
exception that makes the CI noise.
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
We ping the tcp listener to know that we have a tcp connection, that is
written in the xml logs, which means that parsing will not work. Ignore
the ping, parse xml, and make sure that the xml that will be consume by
vsts is valid.
This PR also fixes https://github.com/xamarin/maccore/issues/827 which does not longer happen.
When building extensions, we first store all the mtouch arguments in a file
when msbuild builds the extension, and then when msbuild builds the main
project, we load those arguments again and actually build the extension at the
same time as we build the main app.
As such, it's important to make sure that when we reload the extension
arguments we end up with the exact same build configuration as the first time.
Unfortunately that was not the case regarding the interpreter: we
automatically set the 'EnableRepl' value in Main according to whether the
interpreter was enabled or not, but Main is not called after re-loading the
arguments when building extensions.
Fix this by moving the logic that automatically sets 'EnableRepl' to somewhere
that is executed when re-loading arguments when building extensions.
Fixes https://github.com/xamarin/xamarin-macios/issues/7780.
Most false positives are (so far) because `nuget restore` fails - both
due to network issues and a mono issue [1].
This PR allows to filter/remove some projects from a solution before
nuget is invoked. This can reduce the number of packages to download
which, beside reducing false positives, is good performance wise.
It also allows some project (e.g. Xappy.iOS) to build even if some other
projects (e.g. Xappy.UWP) in the solution can't be built.
[1] https://github.com/mono/mono/issues/15418
* [sample-tester] Collect app size and build duration statistics.
* Always publish performance data.
* [mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_OPTIONS.
* [sample-tester] Make mmp/mtouch show timing information, and get the diagnostic msbuild log.
* Collect task and target info.
* Add more perf data and a baseline test.
* Redo the base test a bit.
* More tweaks.
* More tweaks.
* Huh?
* Debug stuff.
* diagnostics.
* Better diagnostics.
* cleanup
* tmp
* Use a separate stage to push data.
* Next attempt.
* Unique artifact names.
* Make the right name unique.
* More progress
* Improvements
* Don't need new maccore.
* cleanup
* Remove debug spew.
* Realign stuff.
* duh
* More path fixes.
* zippity zip.
* Show publishing errors.
* dependencies
* Not fake results anymore.
* Tweaks
* Merge xml files
* [sampletester] Bump nuget timeout to 5 minutes.
The TodoREST.iOS test seems to use a lot of packages, and it can apparently
take a while to restore them, so give a few more minutes.
* Improvements.
* Fix glob.
* [sampletester] Fix fetching of sample repository and hash.
* [sampletester] Centralize verbosity for mtouch and mmp.
When building extensions, we first store all the mtouch arguments in a file
when msbuild builds the extension, and then when msbuild builds the main
project, we load those arguments again and actually build the extension at the
same time as we build the main app.
As such, it's important to make sure that when we reload the extension
arguments we end up with the exact same build configuration as the first time.
Unfortunately that was not the case regarding the interpreter: we
automatically set the 'EnableRepl' value in Main according to whether the
interpreter was enabled or not, but Main is not called after re-loading the
arguments when building extensions.
Fix this by moving the logic that automatically sets 'EnableRepl' to somewhere
that is executed when re-loading arguments when building extensions.
Fixes https://github.com/xamarin/xamarin-macios/issues/7780.
* Acquire permission for the microphone (the microphone permission dialog is
not a blocking dialog, which is why things have worked so far).
* Set the last_modified field in the TCC database to now instead of 1970.
Apparently some permissions time out (kTCCServiceMediaLibrary), which means
that the last_modified field is important to get right.
Most false positives are (so far) because `nuget restore` fails - both
due to network issues and a mono issue [1].
This PR allows to filter/remove some projects from a solution before
nuget is invoked. This can reduce the number of packages to download
which, beside reducing false positives, is good performance wise.
It also allows some project (e.g. Xappy.iOS) to build even if some other
projects (e.g. Xappy.UWP) in the solution can't be built.
[1] https://github.com/mono/mono/issues/15418
Looks like we have issues with the internet sharing in the VSTS bots,
this means that now that we always try to parse the XML on CI, we get an
exception, catch it and do not show the results.
The workaround simply tries to read the xml, if possible, we will parse
it, else deal with the text only log. The fix will show the results, but
it is a workaround for a configuration issue in the CI.
Unify the harness properties to just look at InCI and remove all the
other ones. There is no real need to have differences between jenkins
and VSTS and Wrench is gone.
We used to test only on Jenkins, and if the build version had jenkins on
it, rather than doing so, just check if we are in the CI by looking if
BUILD_REVISION is present in the env.
This reverts commit 80825d6b6d.
Tests breaks under macOS 10.15.
Also some of the changes I requested on the original PR were not
made and several methods are missing validation. There might be
other stuff too (only found this out while looking at the test
failure).
* [registrar] Fixes NSString trampoline code generation in static registrar
Fixesxamarin/xamarin-macios#7733
This was introduced as a side effect of commit 8425129, we
used to generate `id foo` instead of the full block signature
in the trampoline code used by the static registrar, this is
the reason we never caught this condition before.
Added registrar test.
* Move tests to the appropiate test file
Co-authored-by: Alex Soto <alex@alexsoto.me>
* [registrar] Fixes NSString trampoline code generation in static registrar
Fixesxamarin/xamarin-macios#7733
This was introduced as a side effect of commit 8425129, we
used to generate `id foo` instead of the full block signature
in the trampoline code used by the static registrar, this is
the reason we never caught this condition before.
Added registrar test.
* Move tests to the appropiate test file
* Bump Xamarin.MacDev.
New commits in xamarin/Xamarin.MacDev:
* xamarin/Xamarin.MacDev@210c664 Adds net451 to Xamarin.MacDev.csproj
* xamarin/Xamarin.MacDev@64db365 [winios] Changes provisioning profiles default path
* xamarin/Xamarin.MacDev@d34430a Switch to short-form projects and build for both net461 and netstandard2.0. (#68)
Diff: 0f578f51e6..210c664e56
* [msbuild] Update to latest Mono.Cecil.
The older version doesn't support netstandard2.0.
No code changes were required.
* [msbuild] Remove unused usings.
* [msbuild] Make ILMerge work when building for netstandard2.0.
Also unify/deduplicate the ILMerge logic between Xamarin.iOS and Xamarin.Mac.
* [msbuild] Build for netstandard2.0 in addition to net461.
* [msbuild] Use custom project configurations to support running the tests for both netstandard2.0 and net461.
Use custom project configurations to support running the tests for when the
tasks assembly is built for netstandard2.0 and net461.
* [tests] Make command-line based 'make test-ios-tasks' run tests for both netstandard2.0 and net461.
* [xharness] Add test configuration to run iOS MSBuild tests using either netstandard2.0 or net461.
* [msbuild] Make the netstandard2.0-buils task assemblies the default.
* [msbuild] ILRepack lib assemblies, not ref assemblies.
Ask MSBuild to copy lib assemblies to the output folder when building for
netstandard2.0, this way we can easily find the actual implementation
libraries to pass to ILRepack.
* [msbuild] Merge System.Text.Encodings.Web.dll as well.
* [xharness] Fix build of MSBuild tests for iOS.
* [xharness] Fix two compiler warnings.
Fixes these warnings:
Jenkins.cs(538,37): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
Jenkins.cs(877,14): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
* [xharness] Simplify async code a bit.
* [msbuild] Convert to short-form csproj.
* [msbuild] Make asserts more useful.
* [msbuild] Make tests ignore the actual location of the test assembly.
* [msbuild] Short-style projects default to deterministic builds, which is not compatible with wildcard versions.
* [msbuild] Adjust test.
* Update .gitignore.
* Bump NUnit.ConsoleRunner version.
* [msbuild] Fix indentation.
* [msbuild] Simplify csproj.
- https://github.com/xamarin/xamarin-macios/issues/5738
- There are a number of managed exceptions Apple can throw at you during
debugging, such as expanding a NSColor in the wrong colorspace
- Throwing a managed exception is a nicer debugging experience, and
during debug we don't care about any performance penality.
This change allows to ignore the use of cookies and cookie containers in
the NSUrlSessionHandler. There are two different cookie containers to
consider:
1. The native NSHttoCookieStorage.
2. The managed CookieContainer.
If the native one is set to null, the native code will not use a cookie
storage, which is used as a flag to ignore the managed one.
There is an interesting situation, we allow different types of sessions.
From the cookie storage point of view, Default and Background sessions
are the same, but Ephemeral is not, since we only want to store in ram
the cookies and do not share them.
This supposes a problem because Apple does not provide any API that will
allow to determine the session type use in the configuration. The
workaround has been to hide the direct native call for the configuration
and add an enum value that can later be accessed in the
NSUrlSessionHandler. Of course things cannot be that easy. When a
session is created with the configuration, it creates a copy, and the
internal session configuration does not longer have the flag, therefore,
we need to store the session type in the handler.
Fixes: https://github.com/xamarin/xamarin-macios/issues/7659
Co-authored-by: Chris Hamons <chris.hamons@xamarin.com>
Make csc to bee more strict when compiling the projects and mix some
small errors we had in the bindings.
Fixes: https://github.com/xamarin/xamarin-macios/issues/5398
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@gmail.com>
The order of the check matters, we only skip with a sim && not on
xcode11.
The other whay around will do the wrong short circuit. Since if we are
not on Xcode11 we will always skip, including device, which is not
correct.
Fixes: https://github.com/xamarin/xamarin-macios/issues/6243.
The 32bits **debug** binaries are now too big for Apple's native linker
to process, which gives us (non useful) build errors on the bots.
This will still run the release builds configuration of the tests since
they are smaller and still within the limits of the tooling.
* [Makefile] Make csc strict and fix some small errors.
Make csc to bee more strict when compiling the projects and mix some
small errors we had in the bindings.
Fixes: https://github.com/xamarin/xamarin-macios/issues/5398
There are two important things to look at this:
1. That we do set the cookies that are present in the CookieContainer in
the request. That is, we need to set Cookie headers for all of them.
2. That if we receive a Set-Cookie from the server, the CookieContainer
gets correctly updated else we will have issues since we do not respect
the data sent from the server side.
Tests show both that we are setting the Cookie header and that we honour
the Set-Cookie header.
Fixes: https://github.com/xamarin/xamarin-macios/issues/5665
The tests in xunit are ran in parallel, the default number of threads is
to large which is making the iOS watchdog kill the process before we are
done with the tests.
The default value is either 0 (use as many as possible) of the
Enviroment.ProcessorsCount value, in our case we divide that by two to
make sure we do not use too much CPU.
Fixes: https://github.com/xamarin/maccore/issues/2083
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The tests in xunit are ran in parallel, the default number of threads is
to large which is making the iOS watchdog kill the process before we are
done with the tests.
The default value is either 0 (use as many as possible) of the
Enviroment.ProcessorsCount value, in our case we divide that by two to
make sure we do not use too much CPU.
Fixes: https://github.com/xamarin/maccore/issues/2083
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-Authored-By: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
**Problem**
32 bit tests could only be executed with 64 bit tests. E.g: `run-ios-32-tests` didn't work on its own, only `run-ios-tests,run-ios-32-tests` worked
**Solution**
- Change the logic so that 32 bit tests are only looking for `IncludeiOS32` and 64 bit tests only `IncludeiOS64`
- To keep `run-ios-tests` and `skip-ios-tests` working, hack `SetEnabled` so `IncludeiOS` also sets `IncludeiOS32` and `IncludeiOS64`.
Otherwise the way the code is currently organised we'd have to use `skip-ios-32-tests` and `skip-ios-64-tests`.
Co-authored-by: Vincent Dondain <vidondai@microsoft.com>
Although we provide the correct property to set the proxy configuration,
we do not add a strong dict that will make it nicer for users. This
fixes that API.
Fixes: https://github.com/xamarin/xamarin-macios/issues/5811
* [sample-tester] Collect app size and build duration statistics.
* Always publish performance data.
* [mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_OPTIONS.
* [sample-tester] Make mmp/mtouch show timing information, and get the diagnostic msbuild log.
* Collect task and target info.
* Add more perf data and a baseline test.
* Redo the base test a bit.
* More tweaks.
* More tweaks.
* Huh?
* Debug stuff.
* diagnostics.
* Better diagnostics.
* cleanup
* tmp
* Use a separate stage to push data.
* Next attempt.
* Unique artifact names.
* Make the right name unique.
* More progress
* Improvements
* Don't need new maccore.
* cleanup
* Remove debug spew.
* Realign stuff.
* duh
* More path fixes.
* zippity zip.
* Show publishing errors.
* dependencies
* Not fake results anymore.
* Tweaks
* Merge xml files
* [sampletester] Bump nuget timeout to 5 minutes.
The TodoREST.iOS test seems to use a lot of packages, and it can apparently
take a while to restore them, so give a few more minutes.
* Improvements.
* Fix glob.
* [sampletester] Fix fetching of sample repository and hash.
* [sampletester] Centralize verbosity for mtouch and mmp.
* Fixed
`/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/builds/mono-ios-sdk-destdir/ios-sources/external/linker/src/linker/Linker.Steps/OutputStep.cs(110,15): error CS0246: The type or namespace name ‘OutputException’ could not be found (are you missing a using directive or an assembly reference?) [/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tools/mmp/mmp.csproj]`
* Changed the name of the method that is used from linker. Because of this commit 6be26771b9
* Added `OutputException.cs` file on `mtouch.csproj`.
* Removing enter_gc_safe and exit_gc_safe because now it's already gc_safe in this part of code, after a mono change.
* Added known exceptions to LLVM exception list.
* Needs `ifdef` because of this https://github.com/mono/mono/pull/17260.
* Bump MIN_MONO_VERSION to 6.8.0.41 and point MIN_MONO_URL to the PR.
* Add ENABLE_IOS=1 and ENABLE_MAC=1.
* Added switch to disable packaged mono build
* [Tests] Ignore tests that fail on 32b.
Ignore the test on 32b, and filled issue: https://github.com/mono/mono/issues/17752
* [Tests] Ignore a couple of tests causing OOM.
Hopefully fixes https://github.com/xamarin/maccore/issues/1659 for good.
* Ignore `MM0135` test on Catalina+ because it needs Xcode 9.4.
* [monotouch-test] Add null checks for teardown when test didn't run because of a too early OS version.
* [CFNetwork]: Http 2.0 requires OS X 10.11 or later.
Check whether `_HTTPVersion2_0` is available and fallback to HTTP 1.1 otherwise.
* #7346
* This bumps Mono to use https://github.com/mono/mono/pull/17645 (which is the 2019-10 backport
of https://github.com/mono/mono/pull/17628).
* The big user-visible change is in regards to certificate validation, everything below are just
some minor adjustments to tests.
CoreFX uses a completely new `HttpClientHandler` implementation called `SocketsHttpHandler`,
which you can find at https://github.com/dotnet/corefx/tree/release/3.0/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler.
Since this is not based on the web stack anymore, it does not use any of the related APIs such
as `ServicePointManager` or `WebException`.
There is a new API called `HttpClientHandler.ServerCertificateCustomValidationCallback`.
- https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.8
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Unix.cs (L154)
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Windows.cs (L383)
The `ServicePointManager.ServerCertificateValidationCallback` is no longer invoked and on
certificate validation failure, `AuthenticationException` (from `System.Security.Authentication`)
is thrown instead of `WebException`.
At the moment, the `NSUrlSessionHandler` still uses it's own validation callback and also still
throws `WebException` on failure; we should probably look into making this consistent with the
other handlers.
* `HttpContent.SerializeToStreamAsync()` is now `protected` (changed from `protected internal`).
- src/Foundation/NSUrlSessionHandler.cs: changed overload accordingly.
- src/System.Net.Http/CFContentStream.cs: likewise.
* `HttpHeaders.GetKnownHeaderKind()` is an internal Mono API.
There is a new internal API called `System.Net.Http.PlatformHelper.IsContentHeader(key)`
which exists in both the old as well as the new implementation.
The correct way of doing it with the CoreFX handler is
`HeaderDescriptor.TryGet (key, out var descriptor) && descriptor.HeaderType == HttpHeaderType.Content`
* `HttpClientHandler.MaxRequestContentBufferSize` is now longer supported, you can set it to
any non-negative value, the getter will always return 0.
See c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Core.cs (L18).
- tests/linker/ios/link sdk/HttpClientHandlerTest.cs: removed assertion from test.
* `HttpMessageInvoker.handler` is a `protected private` field - in the CoreFX handler, it is
called `_handler` and `private`. This is accessed via reflection by some of the tests, which are
now using the new name.
- tests/mmptest/src/MMPTest.cs: here
- tests/mtouch/MTouch.cs: here
* tests/monotouch-test/System.Net.Http/MessageHandlers.cs:
Adjust `RejectSslCertificatesServicePointManager` to reflect the certificate validation
changes described above.
- FIXME: There was an `Assert.Ignore()` related to `NSUrlSessionHandler` and macOS 10.10;
I removed that to reenable the test because the description linked to an old issue in
the private repo that was referenced by several "Merged" PR's, so it looked to me that
this might have already been fixed - and I also didn't see why it would fail there.
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
After this commit we will have more application but they will be the
appropiate size so that they can be built with the linker for iOS 32b.
It is important to note that the following apps WILL CONTINUE to fail
since the dlls need to be splitted in mono:
* mscorlib tests
* Mono BCL tests group 5 - Which is monotouch_System.Core_xunit-test.dll
and is too large.
Manuel de la Pena
Přemek Vysoký
Sebastien Pouliot
Alex Soto
Chris Hamons
Rolf Bjarne Kvinge
Whitney Schmidt
monojenkins
Marius Ungureanu
Waleed Chaudhry
Bruno Garcia
Pramit Mallick
Vincent Dondain