I would have liked to automatically get all the references required by bindings-test.dll,
but that turned out to be not as simple as I first thought (since bindings-test is
a library project, the exact dependencies actually depend on the type of the executable
project that references it, which means it's quite difficult to calculate the final
references given just the binding project).
Instead just hardcode the fact that there's a reference to nunit.framework.dll, and
the corresponding on-disk path (we still look in the actual csproj to get the version
of nunit.framework.dll).
* Create a simple Xamarin.Utils.Execution class that can handle all our
process execution needs:
* Captures or streams stdout/stderr (in UTF8).
* Supports async
* Supports a timeout
* Does not depend on any other source file we have, only uses BCL API.
* Have the execution helper classes from mtouch/mmp
(Xamarin.BundlerDriver.RunCommand) and the tests
(Xamarin.Tests.ExecutionHelper) use this new class.
* Some simplifications were made:
* All API that took a string array for the environment now takes a
Dictionary<string, string>.
* The Driver.RunCommand methods were split out to a separate file. This
file also contains a Verbosity field, which is conditioned on not being
in mtouch nor mmp, which makes including this file from other projects
simpler (such as bgen - in particular bgen was modified to use this
Verbosity field instead of its own).
And delete the test case about Xcode 4.4, that's a bit old by now.
However, keep the Xcode 5.1.1 test case (which I've confirmed to work by
installing Xcode 5.1.1), since we'll probably want to have the test around for
when we update our min Xcode requirement (which is currently 6.0).
* Move much of ErrorHandler.cs into a partial class in ErrorHandler.tools.cs,
which is referenced by mtouch and mmp (but not our runtime).
* Add ErrorHandler.runtime.cs for runtime-specific bits, including a simpler
version of ErrorHandler.Show. In particular this gets rid of the call to
Environment.Exit, which should never happen at runtime.
* Rename MonoTouchException and MonoMacException to ProductException, which
allows us to remove a lot of ifdefs.
* This required moving Application.LoadSymbols and Target.LoadSymbols to
shared mtouch/mmp code.
* Rearrange files in Xamarin.Mac a bit to ease code sharing between mmp and
mtouch, by putting mono's static and dynamic libraries in
/Library/Frameworks/Xamarin.Mac.framework/Versions/Current/Sdks/Xamarin.macOS.sdk
to match how Xamarin.iOS does it.
* Don't use 'usr' as an intermediate directory. This removes another special
case.
* Share many of the functions and properties that return specific directories,
and document (as comments) what each function/property is supposed to
return.
* [mtouch] Handle a failure to launch the native linker better by showing better error messages.
dotnet will throw a Win32Exception if the command line is too long, so handle
that scenario. Also handle any other Win32Exceptions and show a better error
message.
* Make MT5217 an error to avoid multiple potentially confusing errors.
Goals
* Reflect Apple nullability annotations in our bindings using C#8
* No warnings when building bindings
Non-Goals
* Update (add or fix) `[NullAllowed]` to match Apple headers (next phase)
* Make the generator or internal code fully nullable aware (`nowarn` is used)
Notes
* Apple's own annotations are not 100% accurate :(
* Where known issue exists we have _fixed_ our attributes to match reality :)
* We also do additional null-checks internally that might seems not required (better safe than sorry).
The generator project file still needs some custom logic, because we can't put
the generated makefile fragment next to the csproj (that would break the
API/generator diff).
* Unify target framework code between mtouch and mmp.
* Simplify the code in mmp: have three possible valid target frameworks for
most of code, and add special code to handle setting any other valid target
frameworks to redirect to one of those three valid target frameworks (and
warn if given any of those valid, but not "main", target frameworks). Any
other code can then depend on the target framework having exactly one of
those specific values, which means we can make IsUnified* variables
convenience properties instead.
* Unify a bit more of the argument parsing code between mtouch and mmp, since
that made a few other things easier.
* Add TargetFramework.IsValidFramework to have one validation implementation.
* Move the implementation of TargetFramework.MonoFrameworkDirectory to mmp
itself, it's not really related to the target framework.
* Remove Driver.IsUnified and IsClassic from mmp, they're not used anymore.
* Formally deprecate --xamarin-[full|system]-framework in mmp, they've really been deprecated for many years.
* Remove LinkerOptions.TargetFramework, it's not used anymore.
* Get rid of mmp's userTargetFramework fried, it's duplicated with the
targetFramework field.
* Add a few tests, and tweak others a bit.
Breaking changes:
* Both mtouch and mmp require --target-framework now. The only direct
consumers should be the MSBuild tasks, which already pass --target-framework
all the time. This simplifies code, and removes assumptions.
* [mtouch][mmp] Share error.cs and make it safer against FormatException
Beside the obvious code sharing [1] this makes the error reporting code
safer against `FormatException`.
Even with code review this happened a few times in the past - we get
a string that can't format correctly in some error handling code. This
means the `FormatException` must be fixed before we can see the
underlying root issue (delaying bug fixes and support).
Recent changes made for localization makes it harder to review the
original strings (and even harder for translated strings) for
correctness. This means we can end up with something like
```
String.Format ("{0} {1}", "foo");
``
and easily miss it (since they split between different files) if there
are several changes inside a PR.
Finally sprinkle some unit tests :)
[1] we should move everything to use `ProductException` but that will be
for another PR...
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
This optimization can be enabled when it's not possible to use the
managed linker (e.g. **Don't link**) or when the managed linker cannot
remove references to deprecated types that would cause an application
to be rejected by Apple.
References to the existing types will be renamed, e.g. `UIWebView` to
`DeprecatedWebView`, in every assemblies.
The type definition is also renamed (for validity) and all custom
attributes on the types and their members will be removed.
Code inside the members will be replaced with a
`throw new NotSupportedException ();`.
The msbuild test app `MyReleaseBuild` has been updated to test that the
optimization is working as expected (device builds are slow so reusing
this test has little impact in test time).
Basically the test ensure that `UIWebView` is used and cannot be removed
by the compiler (optimization) or the managed linker (since it's
referenced). Since the optimization is enabled then we can `grep` then
final `.app` directory to ensure there's no mention of `UIWebView` inside
any of the files that would be submitted.
The application can be run, by itself, and will turn green if OK, red if
`DeprecatedWebView` can't be found (skeleton replacement for `UIWebView`)
or orange if a `NotSupportedException` is thrown.
Finally introspection tests have been updated to skip over the deprecated
(and renamed) types. It should not be an issue right now, since this
optimization is not enabled by default, but it made testing easier.
* [msbuild] Provide the correct value for the operating system for tvOS and watchOS to a few tasks. Fixes#6200. (#7226)
The problem with #6200 was that we'd pass -mios-version-min=x.y to the metal
tool even for tvOS apps. This fixes it so that now pass -mtvos-version-min.
Fixes https://github.com/xamarin/xamarin-macios/issues/6200.
* [msbuild] Add support for Metal in the simulator. Fixes#7392. (#7983)
Don't remove the entire script, because I believe there's code out there that
checks for the existence of the smcs script to determine whether Xamarin.iOS
is installed or not.
When building extensions, we first store all the mtouch arguments in a file
when msbuild builds the extension, and then when msbuild builds the main
project, we load those arguments again and actually build the extension at the
same time as we build the main app.
As such, it's important to make sure that when we reload the extension
arguments we end up with the exact same build configuration as the first time.
Unfortunately that was not the case regarding the interpreter: we
automatically set the 'EnableRepl' value in Main according to whether the
interpreter was enabled or not, but Main is not called after re-loading the
arguments when building extensions.
Fix this by moving the logic that automatically sets 'EnableRepl' to somewhere
that is executed when re-loading arguments when building extensions.
Fixes https://github.com/xamarin/xamarin-macios/issues/7780.
When building extensions, we first store all the mtouch arguments in a file
when msbuild builds the extension, and then when msbuild builds the main
project, we load those arguments again and actually build the extension at the
same time as we build the main app.
As such, it's important to make sure that when we reload the extension
arguments we end up with the exact same build configuration as the first time.
Unfortunately that was not the case regarding the interpreter: we
automatically set the 'EnableRepl' value in Main according to whether the
interpreter was enabled or not, but Main is not called after re-loading the
arguments when building extensions.
Fix this by moving the logic that automatically sets 'EnableRepl' to somewhere
that is executed when re-loading arguments when building extensions.
Fixes https://github.com/xamarin/xamarin-macios/issues/7780.
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
Bump mono to get the new splited test dlls and add them to be ran in
xharness. Special logic is used for mscorlib so we make sure that all
the 'parts' of the test dll do have the same configurations.
Co-Authored-By: Waleed Chaudhry <54864665+wachaudh@users.noreply.github.com>
* Fixed
`/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/builds/mono-ios-sdk-destdir/ios-sources/external/linker/src/linker/Linker.Steps/OutputStep.cs(110,15): error CS0246: The type or namespace name ‘OutputException’ could not be found (are you missing a using directive or an assembly reference?) [/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tools/mmp/mmp.csproj]`
* Changed the name of the method that is used from linker. Because of this commit 6be26771b9
* Added `OutputException.cs` file on `mtouch.csproj`.
* Removing enter_gc_safe and exit_gc_safe because now it's already gc_safe in this part of code, after a mono change.
* Added known exceptions to LLVM exception list.
* Needs `ifdef` because of this https://github.com/mono/mono/pull/17260.
* Bump MIN_MONO_VERSION to 6.8.0.41 and point MIN_MONO_URL to the PR.
* Add ENABLE_IOS=1 and ENABLE_MAC=1.
* Added switch to disable packaged mono build
* [Tests] Ignore tests that fail on 32b.
Ignore the test on 32b, and filled issue: https://github.com/mono/mono/issues/17752
* [Tests] Ignore a couple of tests causing OOM.
Hopefully fixes https://github.com/xamarin/maccore/issues/1659 for good.
* Ignore `MM0135` test on Catalina+ because it needs Xcode 9.4.
* [monotouch-test] Add null checks for teardown when test didn't run because of a too early OS version.
* [CFNetwork]: Http 2.0 requires OS X 10.11 or later.
Check whether `_HTTPVersion2_0` is available and fallback to HTTP 1.1 otherwise.
* #7346
* This bumps Mono to use https://github.com/mono/mono/pull/17645 (which is the 2019-10 backport
of https://github.com/mono/mono/pull/17628).
* The big user-visible change is in regards to certificate validation, everything below are just
some minor adjustments to tests.
CoreFX uses a completely new `HttpClientHandler` implementation called `SocketsHttpHandler`,
which you can find at https://github.com/dotnet/corefx/tree/release/3.0/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler.
Since this is not based on the web stack anymore, it does not use any of the related APIs such
as `ServicePointManager` or `WebException`.
There is a new API called `HttpClientHandler.ServerCertificateCustomValidationCallback`.
- https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.8
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Unix.cs (L154)
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Windows.cs (L383)
The `ServicePointManager.ServerCertificateValidationCallback` is no longer invoked and on
certificate validation failure, `AuthenticationException` (from `System.Security.Authentication`)
is thrown instead of `WebException`.
At the moment, the `NSUrlSessionHandler` still uses it's own validation callback and also still
throws `WebException` on failure; we should probably look into making this consistent with the
other handlers.
* `HttpContent.SerializeToStreamAsync()` is now `protected` (changed from `protected internal`).
- src/Foundation/NSUrlSessionHandler.cs: changed overload accordingly.
- src/System.Net.Http/CFContentStream.cs: likewise.
* `HttpHeaders.GetKnownHeaderKind()` is an internal Mono API.
There is a new internal API called `System.Net.Http.PlatformHelper.IsContentHeader(key)`
which exists in both the old as well as the new implementation.
The correct way of doing it with the CoreFX handler is
`HeaderDescriptor.TryGet (key, out var descriptor) && descriptor.HeaderType == HttpHeaderType.Content`
* `HttpClientHandler.MaxRequestContentBufferSize` is now longer supported, you can set it to
any non-negative value, the getter will always return 0.
See c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Core.cs (L18).
- tests/linker/ios/link sdk/HttpClientHandlerTest.cs: removed assertion from test.
* `HttpMessageInvoker.handler` is a `protected private` field - in the CoreFX handler, it is
called `_handler` and `private`. This is accessed via reflection by some of the tests, which are
now using the new name.
- tests/mmptest/src/MMPTest.cs: here
- tests/mtouch/MTouch.cs: here
* tests/monotouch-test/System.Net.Http/MessageHandlers.cs:
Adjust `RejectSslCertificatesServicePointManager` to reflect the certificate validation
changes described above.
- FIXME: There was an `Assert.Ignore()` related to `NSUrlSessionHandler` and macOS 10.10;
I removed that to reenable the test because the description linked to an old issue in
the private repo that was referenced by several "Merged" PR's, so it looked to me that
this might have already been fixed - and I also didn't see why it would fail there.
## Miscellaneous fixes
* Fixed
`/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/builds/mono-ios-sdk-destdir/ios-sources/external/linker/src/linker/Linker.Steps/OutputStep.cs(110,15): error CS0246: The type or namespace name ‘OutputException’ could not be found (are you missing a using directive or an assembly reference?) [/Users/builder/jenkins/workspace/xamarin-macios/xamarin-macios/tools/mmp/mmp.csproj]`
* Changed the name of the method that is used from linker. Because of this commit 6be26771b9
* Added `OutputException.cs` file on `mtouch.csproj`.
* Removing enter_gc_safe and exit_gc_safe because now it's already gc_safe in this part of code, after a mono change.
* Added known exceptions to LLVM exception list.
* Needs `ifdef` because of this https://github.com/mono/mono/pull/17260.
* Bump MIN_MONO_VERSION to 6.8.0.41 and point MIN_MONO_URL to the PR.
* Add ENABLE_IOS=1 and ENABLE_MAC=1.
* Added switch to disable packaged mono build
* [Tests] Ignore tests that fail on 32b.
Ignore the test on 32b, and filled issue: https://github.com/mono/mono/issues/17752
* [Tests] Ignore a couple of tests causing OOM.
Hopefully fixes https://github.com/xamarin/maccore/issues/1659 for good.
* Ignore `MM0135` test on Catalina+ because it needs Xcode 9.4.
* [monotouch-test] Add null checks for teardown when test didn't run because of a too early OS version.
* [CFNetwork]: Http 2.0 requires OS X 10.11 or later.
Check whether `_HTTPVersion2_0` is available and fallback to HTTP 1.1 otherwise.
## Bring HttpClient from CoreFX
* #7346
* This bumps Mono to use https://github.com/mono/mono/pull/17645 (which is the 2019-10 backport
of https://github.com/mono/mono/pull/17628).
* The big user-visible change is in regards to certificate validation, everything below are just
some minor adjustments to tests.
### SocketsHttpHandler
CoreFX uses a completely new `HttpClientHandler` implementation called `SocketsHttpHandler`,
which you can find at https://github.com/dotnet/corefx/tree/release/3.0/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler.
Since this is not based on the web stack anymore, it does not use any of the related APIs such
as `ServicePointManager` or `WebException`.
### Certificate Validation Changes
There is a new API called `HttpClientHandler.ServerCertificateCustomValidationCallback`.
- https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.8
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Unix.cs (L154)
- c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Windows.cs (L383)
The `ServicePointManager.ServerCertificateValidationCallback` is no longer invoked and on
certificate validation failure, `AuthenticationException` (from `System.Security.Authentication`)
is thrown instead of `WebException`.
At the moment, the `NSUrlSessionHandler` still uses it's own validation callback and also still
throws `WebException` on failure; we should probably look into making this consistent with the
other handlers.
### Minor adjustments related to internal Mono APIs
* `HttpContent.SerializeToStreamAsync()` is now `protected` (changed from `protected internal`).
- src/Foundation/NSUrlSessionHandler.cs: changed overload accordingly.
- src/System.Net.Http/CFContentStream.cs: likewise.
* `HttpHeaders.GetKnownHeaderKind()` is an internal Mono API.
There is a new internal API called `System.Net.Http.PlatformHelper.IsContentHeader(key)`
which exists in both the old as well as the new implementation.
The correct way of doing it with the CoreFX handler is
`HeaderDescriptor.TryGet (key, out var descriptor) && descriptor.HeaderType == HttpHeaderType.Content`
### Minor adjustments to tests.
* `HttpClientHandler.MaxRequestContentBufferSize` is now longer supported, you can set it to
any non-negative value, the getter will always return 0.
See c1778515a3/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Core.cs (L18).
- tests/linker/ios/link sdk/HttpClientHandlerTest.cs: removed assertion from test.
* `HttpMessageInvoker.handler` is a `protected private` field - in the CoreFX handler, it is
called `_handler` and `private`. This is accessed via reflection by some of the tests, which are
now using the new name.
- tests/mmptest/src/MMPTest.cs: here
- tests/mtouch/MTouch.cs: here
* tests/monotouch-test/System.Net.Http/MessageHandlers.cs:
Adjust `RejectSslCertificatesServicePointManager` to reflect the certificate validation
changes described above.
- FIXME: There was an `Assert.Ignore()` related to `NSUrlSessionHandler` and macOS 10.10;
I removed that to reenable the test because the description linked to an old issue in
the private repo that was referenced by several "Merged" PR's, so it looked to me that
this might have already been fixed - and I also didn't see why it would fail there.
Turn older #7165 prototype into an experimental feature. It can be
enabled by adding `--optimize=experimental-xforms-product-type` to the
**Additional mtouch arguments** of the project.
ref: https://github.com/xamarin/xamarin-macios/pull/7165
Turn older #7165 prototype into an experimental feature. It can be
enabled by adding `--optimize=experimental-xforms-product-type` to the
**Additional mtouch arguments** of the project.
ref: https://github.com/xamarin/xamarin-macios/pull/7165
* Implement a different escaping/quoting algorithm for arguments to System.Diagnostics.Process.
mono changed how quotes should be escaped when passed to
System.Diagnostic.Process, so we need to change accordingly.
The main difference is that single quotes don't have to be escaped anymore.
This solves problems like this:
System.ComponentModel.Win32Exception : ApplicationName='nuget', CommandLine='restore '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable/CellCustomTable.sln' -Verbosity detailed -SolutionDir '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable'', CurrentDirectory='/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories', Native error= Cannot find the specified file
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0029f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-08/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:778
ref: https://github.com/mono/mono/pull/15047
* Rework process arguments to pass arrays/lists around instead of quoted strings.
And then only convert to a string at the very end when we create the Process
instance.
In the future there will be a ProcessStartInfo.ArgumentList property we can
use to give the original array/list of arguments directly to the BCL so that
we can avoid quoting at all. These changes gets us almost all the way there
already (except that the ArgumentList property isn't available quite yet).
We also have to bump to target framework version v4.7.2 from v4.5 in several
places because of 'Array.Empty<T> ()' which is now used in more places.
* Parse linker flags from LinkWith attributes.
* [sampletester] Bump to v4.7.2 for Array.Empty<T> ().
* Fix typo.
* Rename GetVerbosity -> AddVerbosity.
* Remove unnecessary string interpolation.
* Remove unused variable.
* [mtouch] Simplify code a bit.
* Use implicitly typed arrays.
This has a couple of advantages:
* It makes it easier to add a catalyst version of these libraries (because it
becomes cumbersome to build for catalyst when the build rules assumes we're
building for both simulator and device).
* It makes it easier to create an xcframework of our libraries, because the
contents in an xcframework is split like this.
Rolf Bjarne Kvinge
Chris Hamons
Manuel de la Pena
Sebastien Pouliot
Alex Soto
Waleed Chaudhry
monojenkins
Vincent Dondain
Jo Shields
Alexander Köplinger