It ends up asserting:
> * Assertion at /Users/runner/work/1/s/src/mono/mono/mini/interp/interp.c:7543, condition `!interp_init_done' not met
because it's already been called.
VSTS is using https and not ssh to clone the repos, this means than if
we have any bash that uses git, it will fail. Cred store fixes this by
removing the need to pass the token in all bash scripts.
We remove allways the cred store once we are done.
If we set the variables as output variables they are not accessible by
the bash scripts resulting in a warning.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
We want to continue and generate the pkgs, the failure of this step only
results in failing tests.
fixes: https://github.com/xamarin/maccore/issues/2376
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
This change perse does not do other thing than removing the use of
virtuanenv and remove the installation of python-magic.
We have taken this approach because the images that will be used to provision the CI
bots will have python-magic installed by default. We want to do this
because:
1. CI should not be modifying the machine, we do not install any pkg.
2. python-magic should be present, if not, we will get an import error
and we will know how to fix it (re-image).
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
This change perse does not do other thing than removing the use of
virtuanenv and remove the installation of python-magic.
We have taken this approach because the images that will be used to provision the CI
bots will have python-magic installed by default. We want to do this
because:
1. CI should not be modifying the machine, we do not install any pkg.
2. python-magic should be present, if not, we will get an import error
and we will know how to fix it (re-image).
This way the bin directory doesn't unnecessarily get an updated timestamp, so
that some build targets in the dotnet/ directory don't rebuild unnecessarily.
Today both `mtouch` and `mmp` are copying the entire `.framework`
directories inside the `.app\[Contents\]Frameworks\` directory.
However not everything in a framework is required at runtime. The most
common unrequired files would be headers (`Headers/*.h`) and modules
(`Modules/*`).
Looking at Xcode build output we can see something like:
```
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -exclude Headers -exclude PrivateHeaders -exclude Modules -exclude \*.tbd -bitcode-strip replace-with-marker -bitcode-strip-tool
```
which excludes a few more, less common, files.
This _builtin_ command is not available externally (for us to re-use)
but it hints that Xcode is likely using `rsync` to avoid copying part of
the files.
Note: the builtin command also _likely_ calls `bitcode_strip` too (or has
similar code embedded) and `mtouch` already does so too
There's a cost to spawning an external process, like `rsync`, which we
avoid by having our own file copier, which clones files (almost zero
cost). That does not support excluding files, but deleting files is also
very cheap. Testing shows copying a framework to be less than 1 ms, even
with with extra deletion step.
* Tweak `GetRealPath` to optionally not to warn if the path does not exists
since, in this case, it's a check we want to do after resolving the path
This fixes several (5) MTouch tests looking for specific (and no extra)
warnings
```
Unable to canonicalize the path '/Users/builder/azdo/_work/2/s/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory195/testApp.app/Frameworks/Mono.framework/CVS': No such file or directory (2).
```
because:
> If the file to be deleted does not exist, no exception is thrown.
https://docs.microsoft.com/en-us/dotnet/api/system.io.file.delete?view=net-5.0
and yes this is different from `Directory.Delete` and PR https://github.com/xamarin/xamarin-macios/pull/10441
* Fix MT0015 test failure
The MT0015 test creates a directory where a file is expected.
That's fine except the code for handling this was a bit weird. It
worked because of a `TryDelete` on the path, which avoided the
`UnauthorizedAccessException` when `File.Delete` is used on a path.
Then later there's a `Directory.Exists` check that would throw...
The code now does the `Directory.Exists` first and only call
`File.Delete` if it returns false. The throwing of the exception
is kept since the code (and test) are already present (and this
minimize changes and chance of other surprises)
* [dotnet-linker] Add the RemoveUserResources sub step.
This comes with a few changes to the RemoveUserResources sub step as well:
* Bail out earlier if we're in the simulator (no need to do any processing at
all - we know at the very beginning if we're building for the simulator).
* Do a positive simulator check, instead of a negative device check (because
Mac[Catalyst] are neither devices - so they pass the negative device check).
* Remove all the conditional mtouch/mmp code, and figure out at runtime which
resource prefixes we need to check for.
It was also necessary to change where steps are added to the pipeline: we have
to remove resources before the OutputStep, but at the same time we have to do
it after the ExtractBindingLibraries step, otherwise the
ExtractBindingLibraries step won't find any binding libraries to extract. So
move the ExtractBindingLibraries, LoadNonSkippedAssemblies and
ListExportedSymbols to before the OutputStep (to keep their internal order),
and then add the RemoveUserResources after those.
This fixes the following link sdk/link all test when running on device:
BundledResources.ResourcesTest
[FAIL] Bundled : No resources
Expected: 0
But was: 2
* [tests] Update the BuildInterdependentBindingProjects test now that we're removing resources.
This allows us to undo a workaround we made for a missing API in the linker
reference assembly (an AnnotationStore.AddPreservedMethod overload).
This also requires a change to use MessageContainer.CreateCustomErrorMessage
instead of MessageContainer.CreateErrorMessage, because apparently having a
reference assembly doesn't mean there can't be incompatible changes in it 😒.
Github has a limited size for messages in comments. If we did reach that
limit, we create a gist to show all the results.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
TL&DR: This PR
1. Removes the creation of the `.dSYM` based on `Debug Information` [1]
2. Adds dSYM support to XM msbuild (now shared with XI implementation)
3. Archive the `.dSYM` directories (plural) properly, e.g.
```
msbuild -p:Configuration=Release -p:ArchiveOnBuild=true
```
Why ? The long story...
Historically `.dSYM` for Xamarin.Mac have not been very useful, largely
because (most of) the code is JITed so not much is known before runtime.
So they were simply not generated during the builds...
However AOT options were added to Xamarin.Mac, making them potentially
more useful. Also symbols from `libmono` and other native libraries /
frameworks can prove useful when diagnosing application crashes.
Unsurprisingly developers looking to get symbols eventually found _a way_
[1] to get a `.dSYM` for their applications - but it was not quite
correct because:
* setting the debug information option meant that `mmp` would be supplied with `-debug`. This disables several optimizations that are, by default, enabled for release builds. IOW generating symbols should have no effect on the executing code (but it had);
* it was produced when compiling the native launcher, so the symbols coverage was incomplete. How much depends if mono was statically or dynamically linked. However this would not cover any AOTed code nor bundled libraries or user frameworks.
* the .dSYM was produced inside the `x.app/Contents/MacOS/`, side-by-side with the native executable, which makes it part of the **signed** `.app` and also part of the created (and signed) `.pkg`. This had a large impact on the application's, disk and download, size(s). Manually (re)moving the `.dSYM` means re-signing the app and re-creating (and signing) the `.pkg` is not a good solution.
[1] https://forums.xamarin.com/discussion/139705/how-to-symbolicate-a-xam-mac-crash-log
Additional fixes
* Use `Directory.Move` instead of running the `mv` command
While the result is identical there is a cost to spawn several `mv`
processes. Doing it in parallel (might have) helped but that setup
also comes at a cost.
`Directory.Move` the four `.dylib.dSYM` of an app takes 1 ms, while
the existing code took 17 ms to do the same.
* Fix building mmptest since the DeleteDebugSymbolCommand constant is not present (nor used) anymore
* Bump to .NET 6.0.100-alpha.1.21060.3.
* Fix dotnet command line arguments.
* dotnet build: the project file must be the first argument.
* dotnet build/publish: use the documented verbosity format.
* Update version number in tests.
* [tests/introspection] Adjust introspection to cope with different library names in the new .NET version.
* [tests/link sdk] Adjust the LinkSdkRegressionTest.SpecialFolder test according to the new version of .NET 6.
* [tests/link sdk] Preserve a required method in System.Private.CoreLib to work around a bug in .NET 6.
Ref: https://github.com/dotnet/runtime/issues/46908.
* Revert "[CI][VSTS] Add the donet 6 pkg as a dependency. (#10348)"
This reverts commit 6de4e717e7.
There's no need to provision .NET 6, it's done automatically.
* [dotnet-linker] Add support for writing to the same MSBuild output items multiple times.
* Split parts of LinkerConfiguration.WriteOutputForMSBuild into a FlushOutputForMSBuild
method (the part that does the actual writing).
* Make WriteOutputForMSBuild just store the items in a dictionary.
* Add a DoneStep that runs at the very end and that writes out the MSBuild output
items.
* [dotnet-linker] Link with GSS when building for iOS/Mac Catalyst.
Add a ComputeNativeBuildFlagsStep, which computes the flags to pass to the native
compiler + native linker. This is currently a very simple implementation, but it
will become more complex as support for missing features are added.
GSS is required because of libSystem.Net.Security.Native.a:
Undefined symbols for architecture arm64:
"___gss_c_nt_hostbased_service_oid_desc", referenced from:
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_c_nt_user_name_oid_desc", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_cred_no_ci_flags_x_oid_desc", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_krb5_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_ntlm_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"___gss_spnego_mechanism_oid_desc", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
l___const.NetSecurityNative_AcquireCredSpNego.gss_mech_spnego_OID_set_desc in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_accept_sec_context", referenced from:
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcquireAcceptorCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_acquire_cred_with_password", referenced from:
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_delete_sec_context", referenced from:
_NetSecurityNative_DeleteSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_display_status", referenced from:
_NetSecurityNative_DisplayMinorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_DisplayMajorStatus in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_import_name", referenced from:
_NetSecurityNative_ImportUserName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ImportPrincipalName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_indicate_mechs", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_init_sec_context", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_inquire_context", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_oid_equal", referenced from:
_NetSecurityNative_InitSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitSecContextEx in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_AcceptSecContext in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_buffer", referenced from:
_NetSecurityNative_ReleaseGssBuffer in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_cred", referenced from:
_NetSecurityNative_ReleaseCred in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_name", referenced from:
_NetSecurityNative_GetUser in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_ReleaseName in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_release_oid_set", referenced from:
_NetSecurityNative_IsNtlmInstalled in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_set_cred_option", referenced from:
_NetSecurityNative_InitiateCredSpNego in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
_NetSecurityNative_InitiateCredWithPassword in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_unwrap", referenced from:
_NetSecurityNative_Unwrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
"_gss_wrap", referenced from:
_NetSecurityNative_Wrap in libSystem.Net.Security.Native.a(pal_gssapi.c.o)
ld: symbol(s) not found for architecture arm64
Remove the misleading title 'Device tests..' for 'Tests..'. The message
already contains the context of the test execution, it can be:
* Build
* VSTS: device tests iOS32b
* VSTS: device tests tvOS
* VSTS: device tests iOS
Removing the 'Device' word is enough to do not confuse users.
fixes: https://github.com/xamarin/maccore/issues/2358
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Since 'packages' is a common dir name that is ignored. Revert the change
in the .gitignore and rename the template path since it just means a one
liner change in the entry yaml file.
fixes: https://github.com/xamarin/maccore/issues/2359
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
TD&LR: This PR simplifies how we refer to user frameworks and fixes both
warnings and non-optimal (app) output.
Much longer story:
Additional testing on macOS showed some build-time warnings and an
[extra (dupe) file](a20f8aba41 (diff-54fd7d9cd5deae57f30195be0a43133eace03c1132401741a317e0ae8d5e13fdR34)).
Logs shows that we referred to the xcframework several times, where once
should have been enough.
```
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework
/native-reference:/Users/poupou/git/spouliot/xcframework/Universal.xcframework/macos-arm64_x86_64/Universal.framework/Universal
```
The first `/native-reference` line produced a warning like:
```
MMP warning MM2006: Native library 'Universal.xcframework' was referenced but could not be found.
```
which makes sense as the tools (both `mmp` and `mtouch`) are not, by
design, aware of (unresolved) xcframeworks.
Removing `{NativeReference}` from `Xamarin.Mac.Common.targets` (and
`Xamarin.iOS.Common.targets`) as it has already been processed by
`_ExpandNativeReferences` solves this.
The other part of the issue (next two lines) is because `msbuild` does
not track changes to directories like it does for files - and the
workaround (in `_ExpandNativeReferences`) had to be copied in other
places (both XI and XM `_CompileToNative`) and that was not enough (and
would eventually need to be duplicated again and again).
This could lead to duplicate entries (i msbuild logs) like
```
NativeReferences
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework
../../Universal.xcframework/macos-arm64_x86_64/Universal.framework/Univeral
```
which maps to our extra entries.
In order to simplify things we make the `_ExpandNativeReferences` resolve
the full path to the library name (not the `.framework` directory) which
simplifies both `_CompileToNative` and ensure a single way (at least for
`msbuild`) to provide this data to the tools (`mmp` and `mtouch`).
Using a file, instead of a directory, is also more consistent for the
existing `-framework` option, e.g. we provide the names like:
```
--framework=CoreLocation
--framework=ModelIO
```
So adding a full path that include the name is more appropriate, e.g.
``` --framework=/Users/poupou/git/master/xamarin-macios/tests/xharness/tmp-test-dir/xcframework-test760/bin/AnyCPU/Debug/bindings-xcframework-test.resources/XTest.xcframework/ios-i386_x86_64-simulator/XTest.framework/XTest
```
Finally for macOS applications it turns out we were embedding yet another
copy of the framework's library inside the `MonoBundle`, which is clearly
wrong, because of the last entry.
```
$ l bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
-rwxr-xr-x 1 poupou staff 167152 2 Dec 16:16 bin/Release/xcf-mac.app/Contents/MonoBundle/Universal
```
The tool now checks if a provided library is inside a framework (or not)
which is a good validation to have anyway when it gets called directly,
i.e. not thru `msbuild`.
Remove the misleading title 'Device tests..' for 'Tests..'. The message
already contains the context of the test execution, it can be:
* Build
* VSTS: device tests iOS32b
* VSTS: device tests tvOS
* VSTS: device tests iOS
Removing the 'Device' word is enough to do not confuse users.
fixes: https://github.com/xamarin/maccore/issues/2358
Since 'packages' is a common dir name that is ignored. Revert the change
in the .gitignore and rename the template path since it just means a one
liner change in the entry yaml file.
fixes: https://github.com/xamarin/maccore/issues/2359
The prefix of the location of the logs depends on the platform where it
was executed (else we step on them) but that was not added in the env
var use by xharness.
fixes: https://github.com/xamarin/maccore/issues/2349
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The GitHup url to be used to create comments in PRs is diff to the one
for comments. Use the build reason AND the changeID to identify if we
are building due to a PR and use the correct url.
fixes: https://github.com/xamarin/maccore/issues/2356
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
Co-authored-by: Alex Soto <alex@alexsoto.me>
The wrong variable was used and therefore the dir path was used instead
of the mac pkg name.
fixes: https://github.com/xamarin/maccore/issues/2353
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
User frameworks for macOS often uses symlinks (as Xcode creates them
this way).
This cause problem cause the symlink is on the binary and we expected
the `_CodeSignature` directory to by side-by-side with the binary. This
was missing and cause exceptions when codesigning such frameworks.
A second problem happened because `mmp` use `lipo -thin` to remove
non-required architectures. However when a framework has symlinks, like:
```
├── Frameworks
│ └── Universal.framework
│ ├── Resources -> Versions/Current/Resources
│ ├── Universal -> Versions/Current/Universal
│ └── Versions
│ ├── A
│ │ ├── Resources
│ │ │ └── Info.plist
│ │ ├── Universal
│ │ └── _CodeSignature
│ │ └── CodeResources
│ └── Current -> A
```
then this actually replaced the (very small) symlink with a thin version
of the framework. Which means the original one was still _fat_ and the
whole app was now larger than the original version.
Sample used: https://github.com/spouliot/xcframework/tree/main/xamarin/xcf-mac
The GitHup url to be used to create comments in PRs is diff to the one
for comments. Use the build reason AND the changeID to identify if we
are building due to a PR and use the correct url.
fixes: https://github.com/xamarin/maccore/issues/2356
Co-authored-by: Alex Soto <alex@alexsoto.me>
The prefix of the location of the logs depends on the platform where it
was executed (else we step on them) but that was not added in the env
var use by xharness.
fixes: https://github.com/xamarin/maccore/issues/2349
* [tests] Build test-libraries for Mac Catalyst.
* [msbuild] Add support for Mac Catalyst binding projects.
* [mtouch] Allow frameworks for Mac Catalyst apps.
* [mtouch] Put frameworks in the expected location for Mac Catalyst apps.
* [msbuild] Create the Resources directory before trying to put files in it.
* [src] Generate the string constants for each framework from our list of frameworks.
This makes it less error prone to add new frameworks (one less place to add
them), and easier to add new platforms (only one place to make sure the list
of frameworks is correct).
This also revelead a few issues in our list of frameworks (missing frameworks,
frameworks that have been moved, etc.), which have been fixed.
* [mmp] Generate the library path -> namespace map for the linker to optimize away dlopen statements.
* Fix indentation to be just as broken as the surrounding indentation.
* Remove unused Cecil reference.
* Fix c&p oversight.
* Adjust more Mac Catalyst API.
* CFNetwork is its own top-level framework.
Rework the code to generate an API comparison and a generator comparison: we
now clone xamarin-macios into a different directory, checks out the hash we
want to compare against, build the whole thing, and then compare the results.
This is somewhat slower than the previous implementation (which would checkout
the previous hash and build that on top of the existing build), but since we
don't have to build mono, it shouldn't make too much of a difference (the
biggest chunk of our build is building src/, and we had to do that anyway with
the previous implementation as well).
The upside is that it should be much more reliable.
Fixes https://github.com/xamarin/maccore/issues/2034.
Fixes https://github.com/xamarin/maccore/issues/1467.
Fixes https://github.com/xamarin/maccore/issues/1059.
Maybe https://github.com/xamarin/maccore/issues/1173 too.
* Install the Mac Catalyst versions of the mono libraries and BCL.
* The BCL is the same as the one for Xamarin.iOS, which means it has to be post-processed a bit to work with a Xamarin.MacCatalyst.dll
* Build our runtime for Mac Catalyst.
* Build a Xamarin.MacCatalyst.dll with the Mac Catalyst API (it compiles, but I haven't looked at the API surface at all). This PR assumes we're going to have a new TargetFrameworkIdentifier for Mac Catalyst, but a final decision has not been made (see https://github.com/dotnet/runtime/issues/44882), so this may change.
* Build a Xamarin.iOS.dll that contains type forwarders to Mac Catalyst for all the types that exist in both Mac Catalyst and Xamarin.iOS.
* Add support to xharness for running introspection on Mac Catalyst (there are a lot of failures because the API surface is wrong)
* Add support to our msbuild tasks and mtouch for building Mac Catalyst apps. This basically comes down to adding a new case in numerous places to either do things the iOS way or the macOS way, depending on each case.
* Add a __MACCATALYST__ define (which is in addition to the __IOS__ define).
In some places we have to provide the macOS version, and in other places the
iOS version. Add a map and the corresponding code to convert between the two,
and use them when needed.
In this case we're like iOS simulator builds: symbols may not be required under a
few circumstances, one of which is AOT compilation, which doesn't happen for catalyst
apps, thus symbols are always required for catalyst apps.
* Bumps mono binaries to include x86_64 watchOS support
* Build runtime/registrar x86_64 slices
* Produce a 64 bit version of Xamarin.WatchOS.dll
* Allow building x86_64 for watch simulators in mtouch
* Let xharness know about x86_64
* [tests] Add x86_64 arch to test-libraries
* Make dotnet package aware of x64
* [ObjCRuntime] Fix computing if we're calling a stret function or not in a 64-bit watchOS simulator.
* [xharness] Re-enable some watchOS tests.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
New commits in mono/mono:
* mono/mono@ac596375c7 Add support for OP_FCONV_TO_I to mini-arm64.c. (#20548)
* mono/mono@392fe5b87b [2020-02][watchOS] Add simwatch64 support (#20552)
* mono/mono@a22ed3f094 Fix potential crash for Encoder.Convert (#20522)
* mono/mono@970783731f Bump to F# 5.0 (#20511)
* mono/mono@32ab5066f7 Bump msbuild to fix a build issue
* mono/mono@93a7fe77e8 Ensure special static slots respect alignment. (#20506)
* mono/mono@3db5b35841 [debugger] Switch to GC Unsafe in signal handler callbacks (#20495)
* mono/mono@af315f44c4 [2020-02][corlib] ThreadAbortException protection for ArraySortHelper (#20468)
* mono/mono@ca11fb0fd8 [2020-02] Bump ikvm-fork to include https://github.com/mono/ikvm-fork/pull/20 (#20452)
Diff: be2226b5a1..ac596375c7
This avoids some code using one value, and other code using another value.,
and things don't quite work as expected.
In particular it makes it so that passing -v to bgen makes errors show the
stack trace of the corresponding exception.
This is done early so we can resolve the inner framework, inside the
xcframework, and let the existing framework support do most of the
work.
The resolving code has unit tests. Custom projects for "NoEmbedding"
exists for all supported platforms and executed by xharness.
A sample `xcframework` with tests projects is also available
[here](https://github.com/spouliot/xcframework).
The xcframework test case is based on Rolf's earlier/partial implementation.
https://github.com/rolfbjarne/xamarin-macios/commit/xcframework
Things to note:
Do not rename a framework (like XTest) to use it in an xcframework
(like XCTest). That will fail at codesign but won't give anything
useful. You might think signing the framework (instead of the inner
binary) would solve it. It does, as it codesign, but then the app
crash at startup. At some point you realize some symbols are still
using XTest (not XCTest) and then you can delete several other weird
workarounds (like for `ld`) because all of it was cause by this
never identified rename.
dSYM support (and tests) to be done in a separate PR.
Currently we put the implementation assemblies for all Xamarin.iOS platforms
in the same directory. This makes it impossible to have different
implementations for the same assembly in different platforms: in particular,
we're going to want a special version of Xamarin.iOS.dll for Mac Catalyst
(that will just have type forwarders into Xamarin.MacCatalyst.dll), that that
assembly will go into the Mac Catalyst-specific directory of implementation
assemblies.
* [tools] Disable Objective-C exception handling for macOS .NET apps due to a missing feature in the runtime.
We need support for dllmaps, and that isn't working on macOS yet.
* Fix to only affect .NET
* [watchOS] Add x86_64 simulator support
* Build runtime/registrar x86_64 slices
* Produce a 64 bit version of Xamarin.WatchOS.dll
* Allow building x86_64 for watch simulators in mtouch
* Let xharness know about x86_64
* [tests] Add x86_64 arch to test-libraries
* Make dotnet package aware of x64
* [ObjCRuntime] Fix computing if we're calling a stret function or not in a 64-bit watchOS simulator.
* [xharness] Re-enable some watchOS tests.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
This involves a few changes:
* Change everything to reference net6.0 instead of net5.0
* Update various variables to be NET6* instead of NET5*
* Reorder build logic to account for that our targets are imported earlier in
the build process:
In the latest .NET 6, our Workloads.targets is imported earlier in the
build. This requires a few changes, because we still need to run most of
our logic later in the process, which we do by adding targets files we
want imported later to the AfterMicrosoftNETSdkTargets property.
What we're loading as soon as possible:
* Our version information (Xamarin.Shared.Sdk.Versions.targets)
* The supported OS versions
(Microsoft.<platform>.Sdk.SupportedTargetPlatforms.targets)
* The default OS version
(Xamarin.Shared.Sdk.TargetFrameworkInference.targets).
This is all information that the .NET build require early on.
Changes:
* Rename all files that are loaded early to *.props.
* Updated documentations to reflect these changes.
* Remove Microsoft.<platform>.TargetFrameworkInference.targets, these
files aren't used and don't contain anything useful.
* Move the logic to calculate _ComputedTargetFrameworkMoniker has been
delayed to later, because it needs TargetFrameworkMoniker set.
* Add a StoreAttributesStep to store attributes that are removed by the
linker, but that the static registrar needs.
In particular, in .NET 6 the linker removes the
System.Runtime.CompilerServices.ExtensionAttribute, which the static
registrar needs to handle category methods properly.
This involved copying and slightly modifying the RemoveAttributesBase
code.
Add a StoreAttributesStep to store attributes that are removed by the linker, but
that the static registrar needs.
In particular, in .NET 6 the linker removes the System.Runtime.CompilerServices.ExtensionAttribute,
which the static registrar needs to handle category methods properly.
This involved copying and slightly modifying the RemoveAttributesBase code.
Improve error reporting in dotnet-linker by not requiring an instance of a LinkerConfiguration
to report errors. This is accomplished by making the LinkerConfiguration.Report method
a static method.
Otherwise reporting errors before we've successfully created a LinkerConfiguration
turns out to be troublesome (we end up throwing another exception, usually a NullReferenceException,
which is just confusing).
Using provisionator could result in the UI prompt being shown which will
make the pipeline hang until there is human interaction.
Move the step to be one of the first ones to ensure provisionator does
not get stuck.
Make sure to redirect the dotnet build output to our custom output directory
when building the previous commit so that it doesn't interfere with the normal
build output.
Also add validation for this.
* Continue using our own error handling logic, and print our problems to stderr.
* Also use the linker's messaging facilities to report a more generic error,
in case stderr doesn't show up for some reason.
Because just this is kind of useless:
error MT2301: The linker step 'Setup' failed during processing.
now it will say:
error MT2301: The linker step 'Setup' failed during processing: <hopefully something useful here>
Sometimes we wrap exceptions to add more information to what's happening, but
that may end up worse if we don't print out the wrapped exceptions.
At the same time we don't want to flood the user with information if they
didn't ask for it, so only show nested exceptions if they're something we
raised ourselves.
* [dotnet-linker] Catch any exceptions from our custom steps and show them using our error reporting logic.
* Letting the linker handle the exceptions will not result in a particularly
good experience, because the linker will crash.
* We can also show better information, since we have more knowledge about many
of the exceptions we raise ourselves.
* [dotnet] Make ConfigurationAwareSubStep inherit from ExceptionalSubStep.
This required a minor modification to ExceptionalSubStep to allow for custom reporting.
* [dotnet] Implement ConfigurationAwareStep's exception handling like it's done in ExceptionalSubStep.
I also changed the error message slightly to hopefully make it a bit more
comprehensible when translated (since the step name won't be translated, we'll
end up with a message that mixes English with the translated string).
* [tests] Fix sample tests to take into account when a platform has been disabled.
* [devops] The INCLUDE_MAC/INCLUDE_IOS variables might not exist.
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
* Ship mlaunch in the iOS, tvOS and watchOS NuGets. It should probably go into
a separate NuGet (to avoid shipping the same mlaunch executable in three different
packages), but that can be done at a later stage.
* Add a GetMlaunchArguments task that computes the mlaunch arguments to install
or launch an app in either the simulator or on device.
* Implement the MSBuild logic to make the Run target (provided by .NET) launch
mlaunch (for iOS, tvOS and watchOS) or the built app (for macOS). This is done
by setting the RunCommand and RunArguments properties (which the Run target uses)
to the correct values.
Ideally I'd would make 'dotnet run' work too, but that runs into a different problem which
I haven't figured out yet:
A fatal error was encountered. The library 'libhostpolicy.dylib' required to execute the application was not found in '/Users/rolf/work/maccore/onedotnet/xamarin-macios/tests/dotnet/MySingleView/bin/Debug/net5.0-ios/ios-x64/'.
Failed to run as a self-contained app.
- The application was run as a self-contained app because '/Users/rolf/work/maccore/onedotnet/xamarin-macios/tests/dotnet/MySingleView/bin/Debug/net5.0-ios/ios-x64/MySingleView.runtimeconfig.json' did not specify a framework.
- If this should be a framework-dependent app, specify the appropriate framework in '/Users/rolf/work/maccore/onedotnet/xamarin-macios/tests/dotnet/MySingleView/bin/Debug/net5.0-ios/ios-x64/MySingleView.runtimeconfig.json'.
That's for a different pull request though.
Ref: https://github.com/xamarin/net6-samples/issues/35.
The static registrar may need access to types that have been linked away, so
store unmarked types so that the static registrar can access them later.
This also makes all the monotouch-test variations green, so enable them all.
Fixes this monotouch-test when all optimizations are enabled:
MonoTouchFixtures.ObjCRuntime.RegistrarTest
[FAIL] TestProtocolRegistration : UIApplicationDelegate/17669
Expected: True
But was: False
at MonoTouchFixtures.ObjCRuntime.RegistrarTest.TestProtocolRegistration() in xamarin-macios/tests/monotouch-test/ObjCRuntime/RegistrarTest.cs:line 1350
The .NET linker comes with a way to remove attributes (by passing '--link-attributes
some.xml' as a command-line argument), but this has a few problems:
* We'd need to figure out which attributes to remove before running the linker,
but the code to figure out which optimizations have been enabled (and which attributes
should be removed) is in our custom linker code. We'd need to refactor a big chunk
of code to move this logic out of our custom linker code.
* We need to keep the removed attributes around, because the static registrar needs
them. Our custom linker logic is not notified for removed attributes, which means
we'd need to store all attributes for the attribute types we're interested in (as
opposed to this solution, where we only store attributes that are actually removed).
* The attributes we want removed may contain references to types we don't want
linked away. If we ask the linker to remove those attributes, then the types may
be linked away as well, and there's no good way around this.
The end result is that a custom step is the best solution for now.
Fixes these monotouch-test tests when enabling all optimizations:
Xamarin.BindingTests.ProtocolTest
[FAIL] OnlyProtocol : [Protocol] IP1
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.OnlyProtocol() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 47
[FAIL] ProtocolWithBaseType : [Protocol] IP2
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.ProtocolWithBaseType() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 79
[FAIL] ProtocolWithBaseTypeAndModel : [Protocol] IP3
Expected: 0
But was: 1
at Xamarin.BindingTests.ProtocolTest.ProtocolWithBaseTypeAndModel() in /Users/rolf/work/maccore/main/xamarin-macios/tests/bindings-test/ProtocolTest.cs:line 115
Fixes https://github.com/xamarin/xamarin-macios/issues/9562.
It also fixes this monotouch-test when enabling all optimizations:
[FAIL] BlockReturnTest : ObjCRuntime.RuntimeException : Invalid DelegateProxyAttribute for the return value for the method MonoTouchFixtures.ObjCRuntime.RegistrarTest+BlockReturnTestClass.MethodReturningBlock: DelegateType (ObjCRuntime.Trampolines+SDRegistrarTestBlock) specifies a type without a 'Handler' field.
This fixes a startup crash in the simulator with Xcode 12:
Library not loaded: /usr/lib/libnfshared.dylib
Referenced from: /Applications/Xcode_12.0.0-GMb.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/System/Library/Frameworks/CoreNFC.framework/CoreNFC
Reason: no suitable image found. Did find:
/usr/lib/libnfshared.dylib: mach-o, but not built for platform iOS-sim
Ref: https://stackoverflow.com/q/63915728/183422
Make the bgen tests pass in the path to the attribute library, platform
assembly and all the .NET reference assemblies to bgen. This way we execute
these tests using the .NET version of everything.
This way we know at runtime what's available and what's not, which means that
the runtime won't try to register assemblies when the dynamic registrar has
been linked away.
Fixes this startup crash in monotouch-test when all optimizations have been enabled:
Unhandled Exception:
ObjCRuntime.RuntimeException: The runtime function register_assembly has been linked away.
* [src] Build the .NET version of our product assemblies using a .NET 5 BCL.
We're not shipping the .NET product assemblies in any stable release, so we
can use a preview version of the .NET 5 BCL.
Also:
* Add all the nuget feeds we need to the top-level NuGet.config, even for .NET
5/6, there shouldn't be any conflicts with stable feeds since we use exact
version numbers.
* Generate a top-level global5.json which is copied to every directory that
needs a .NET 5 global.json (overriding the .NET 3.1 global.json in the root
directory).
* Use the expected dotnet binary during our local build.
* [tests] Fix the bgen tests to use .NET 5.
* [xharness] Set the current directory to the project directory when running .NET tests.
This way we end up using the dotnet version that's configured in global.json for the tests.
* Use the existing information we have in the Frameworks class to determine
whether a particular framework works in the simulator or not.
* Show a warning (MX5223) when we run into such a framework, but only if
the linker is enabled (otherwise we'll often get warnings for API the developer
doesn't use).
* [xcode12] Initial bump for Xcode 12 GM
* [tests][intro] We cannot load CoreNFC framework on iOS simulator anymore
* [tests][xtro] New Metal API are not abstract (before XAMCORE_4_0)
* [tests][intro] Fix crash when CSLocalizedString 'description' selector is called
* [tests][xtro] Add support for excluding platforms
Based on `Make.config` variables `INCLUDE_[IOS|TVOS|WATCH|MAC]`
This required moving some entries (common -> macOS) to keep the
sanitizer happy.
* [xtro] Ignore Intents watchOS differences since they will likely match iOS in the future
* Update to use Xcode 12 GMb instead of the old GM
* [tests][xtro] Remove OSLog for iOS and tvOS (changed in GM)
* [tests][msbuild] Disable FrameworkListTest based on the active/disabled platforms
* [tests][msbuild] Track new directory/file inside CoreML projects
* [mlaunch] Bump maccore and disable mlaunch if mac build is disabled
New commits in xamarin/maccore:
* xamarin/maccore@ba332d4d07 Disable mlaunch if Mac is not built (#2314)
Diff: 87a96d21c9..ba332d4d07
Co-authored-by: Sebastien Pouliot <sebastien.pouliot@microsoft.com>
* Fix links that point to master to point to main instead.
* Implement support in the sample tester for specifying the default branch for
each sample repo.
* Fix various text / documentation to say 'main' instead of 'master.'
* Push to 'main' instead of 'master' in xamarin-macios-data.
* Fix xharness to make 'main' the special branch with regards to documentation tests as opposed to 'master'.
* Fix various CI to use 'main' instead of 'master'.
This is a backport of PR #9561
Some appextension mtouch code had to be moved to shared code. This code is currently
only used for iOS/tvOS/watchOS, but it will eventually be applicable to macOS as
well.
This makes it possible to re-use the registrar code in dotnet-linker.
Fixes these linkall tests:
Linker.Shared.OptimizeGeneratedCodeTest
[FAIL] IsARM64CallingConvention : optimized: no ldsfld instruction
Expected: 0
But was: 1
at Linker.Shared.BaseOptimizeGeneratedCodeTest.IsARM64CallingConvention() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 527
[FAIL] SetupBlockPerfTest : At least 6x speedup
Expected: greater than 6
But was: 1.0876440665344851d
at Linker.Shared.BaseOptimizeGeneratedCodeTest.SetupBlockPerfTest() in /Users/rolf/work/maccore/main/xamarin-macios/tests/linker/BaseOptimizeGeneratedCodeTest.cs:line 120
And linkall is now green for .NET/Debug.
* [dotnet] Pass the Optimize flags from the extra bundler arguments to the linker configuration.
Also call Application.InitializeCommon to initialize the application instance. The
important part here is that InitializeCommon calls Optimizations.Initialize to compute
the default optimizations. It also calls Set*ExceptionMode and sets the default EnableCoopGC
value (so we don't need to call/set those anymore), and it does a few other initialization
tasks which we don't need yet, but eventually will.
And finally remember to parse the bundler arguments before using them in the dotnet
build logic. How did this not cause problems before? 🤦
* [tests] Set the verbosity using the additional args instead of an internal variable.
The internal _BundlerVerbosity variable is overwritten now (with the verbosity
value from the additional args).
* [xharness] Disable tvOS generation for the introspection/.NET test, it incorrect and needs fixing.
The type references are not cleaned (anymore?) and what's in memory can
be different from what will be saved to disk (which is the part that
matter).
So before linking we can check for type references (in a module) but
after linking need to see if it resolve (which means the definition,
of the reference, can still be found) and, just be be thorough, check
that's it's marked (if found).
Refactor the Optimizations class to have no conditionally compiled code, which makes
it re-usable from our dotnet-linker code.
Also return any errors or warnings instead of showing/throwing them, which makes
the caller able to show them using whatever means is easiest for the caller.
One test needed an update to the list of valid optimizations, because we now have
a per-platform map of valid optimizations, instead of just a iOS/tvOS/watchOS vs
macOS split ('remove-unsupported-il-for-bitcode' is only valid for watchOS, and now
we say so, while we previously said it was a valid optimization for iOS and tvOS
as well, even though we'd warn about it and do nothing if you tried to set it).
The Assembly.IsFrameworkAssembly property is used in two places:
* In Driver.IsBoundAssembly to return early when determining if an assembly has any NSObject subclasses: c1c5b9aac6/tools/mtouch/mtouch.cs (L1155-L1168)
* In Assembly.ExtractNativeLinkInfo to return early when looking for assemblies with LinkWith attributes: c1c5b9aac6/tools/common/Assembly.cs (L150-L154)
In both cases this definition of framework assembly works today and seems likely to work in the future as well.
I also went through and looked at all the usages of Profile.IsSdkAssembly, and it's used to:
* Decide which assemblies are selected for "link sdk"
* Decide which assemblies are considered an 'sdk' assembly for creating a user framework of all the sdk assemblies
* Bail out early when deciding whether:
* An assembly references the product assembly (Xamarin.iOS.dll, etc.)
* An assembly can contain references to UIWebView
* An assembly can contain user resources
* An assembly is a binding project / has third-party native resources
* An assembly needs the dynamic registrar
* An assembly has FieldAttributes whose native fields must be preserved by the native linker
In all cases our .NET definition of 'SDK' seems to work both for now and in the future.
There are also a few usages which does not apply to .NET, so I've ignored them:
* When looking for a few BCL APIs that must be preserved (MobileApplyPreserveAttribute.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
* When linking away parameter names (MonoTouchMarkStep.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
This also means:
* Adding the ExceptionalSubStep step to the build.
* Adding a few workarounds for missing/different linker API.
This fixes this startup crash in link all:
2020-08-26 19:56:03.936330+0200 link all[45665:6121665] Could not register the assembly 'link all': ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeParameters()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.get_Signature()
at Registrar.Registrar.ObjCMethod.get_Trampoline()
at Registrar.DynamicRegistrar.RegisterMethod(ObjCMethod method)
at Registrar.DynamicRegistrar.OnRegisterType(ObjCType type)
at Registrar.Registrar.RegisterTypeUnsafe(Type type, List`1& exceptions)
at Registrar.Registrar.RegisterAssembly(Assembly assembly)
2020-08-26 19:56:03.939327+0200 link all[45665:6121665] System.AggregateException: One or more errors occurred. (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.) (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.) (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.)
---> ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeReturnType()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.ValidateSignature(List`1& exceptions)
--- End of inner exception stack trace ---
---> (Inner Exception #1) ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeReturnType()
at Registrar.Registrar.VerifyInSdk(List`1& exceptions, ObjCMethod method)
at Registrar.Registrar.ObjCType.Add(ObjCMethod method, List`1& exceptions)
at Registrar.Registrar.RegisterTypeUnsafe(Type type, List`1& exceptions)<---
---> (Inner Exception #2) ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeParameters()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.ValidateSignature(List`1& exceptions)<---
=================================================================
Native Crash Reporting
=================================================================
Got a abrt while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x10d57eb0e - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
0x10d52b437 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_native_crash
0x10d57e365 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
0x7fff51c005fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_platform.dylib : _sigtramp
0x0 - Unknown
0x7fff51af0b7c - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
0x10d32478f - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_unhandled_exception_handler
0x10d5eef68 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_invoke_unhandled_exception_hook
0x10d52ade0 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_exception_internal
0x10d5296c1 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_exception
0x10d579582 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_amd64_throw_exception
0x10d91b5b0 - Unknown
0x10d3244eb - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_process_managed_exception
0x10d324367 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_process_managed_exception_gchandle
0x10d336851 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_main
0x10d217ced - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/link all : main
0x7fff51a231fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdyld.dylib : start
Fixes this linker crash when building the link sdk test:
ILLink : error IL1012: IL Linker has encountered an unexpected error. Please report the issue at https://github.com/mono/linker/issues [/Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/xharness/tmp-test-dir/link sdk33/link sdk.csproj]
Fatal error in IL Linker (TaskId:212)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object. (TaskId:212)
at Mono.Linker.Steps.SubStepsDispatcher.DispatchType(TypeDefinition type) (TaskId:212)
at Mono.Linker.Steps.SubStepsDispatcher.BrowseTypes(Collection`1 types) (TaskId:212)
at Mono.Linker.Steps.SubStepsDispatcher.BrowseAssemblies(IEnumerable`1 assemblies) (TaskId:212)
at Mono.Linker.Steps.SubStepsDispatcher.Mono.Linker.Steps.IStep.Process(LinkContext context) (TaskId:212)
at Mono.Linker.Pipeline.Process(LinkContext context) (TaskId:212)
at Mono.Linker.Driver.Run(ILogger customLogger) (TaskId:212)
at Mono.Linker.Driver.Main(String[] args) (TaskId:212)
The command exited with code 134. (TaskId:212)
Ref: https://github.com/mono/linker/issues/1458
Fixes this startup crash in link all:
2020-08-26 19:56:03.936330+0200 link all[45665:6121665] Could not register the assembly 'link all': ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeParameters()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.get_Signature()
at Registrar.Registrar.ObjCMethod.get_Trampoline()
at Registrar.DynamicRegistrar.RegisterMethod(ObjCMethod method)
at Registrar.DynamicRegistrar.OnRegisterType(ObjCType type)
at Registrar.Registrar.RegisterTypeUnsafe(Type type, List`1& exceptions)
at Registrar.Registrar.RegisterAssembly(Assembly assembly)
2020-08-26 19:56:03.939327+0200 link all[45665:6121665] System.AggregateException: One or more errors occurred. (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.) (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.) (The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.)
---> ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeReturnType()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.ValidateSignature(List`1& exceptions)
--- End of inner exception stack trace ---
---> (Inner Exception #1) ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the return value in the method LinkAll.Attributes.SmartConsumer.GetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeReturnType()
at Registrar.Registrar.VerifyInSdk(List`1& exceptions, ObjCMethod method)
at Registrar.Registrar.ObjCType.Add(ObjCMethod method, List`1& exceptions)
at Registrar.Registrar.RegisterTypeUnsafe(Type type, List`1& exceptions)<---
---> (Inner Exception #2) ObjCRuntime.RuntimeException: The registrar can't convert from 'LinkAll.Attributes.SmartEnum' to 'Foundation.NSString' for the parameter 'value' in the method LinkAll.Attributes.SmartConsumer.SetSmartEnumValue.
at Registrar.Registrar.ObjCMethod.get_NativeParameters()
at Registrar.Registrar.ComputeSignature(Type DeclaringType, MethodBase Method, ObjCMember member, Boolean isCategoryInstance, Boolean isBlockSignature)
at Registrar.Registrar.ObjCMethod.ComputeSignature()
at Registrar.Registrar.ObjCMethod.ValidateSignature(List`1& exceptions)<---
=================================================================
Native Crash Reporting
=================================================================
Got a abrt while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x10d57eb0e - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
0x10d52b437 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_native_crash
0x10d57e365 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
0x7fff51c005fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_platform.dylib : _sigtramp
0x0 - Unknown
0x7fff51af0b7c - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
0x10d32478f - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_unhandled_exception_handler
0x10d5eef68 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_invoke_unhandled_exception_hook
0x10d52ade0 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_exception_internal
0x10d5296c1 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_handle_exception
0x10d579582 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libmonosgen-2.0.dylib : mono_amd64_throw_exception
0x10d91b5b0 - Unknown
0x10d3244eb - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_process_managed_exception
0x10d324367 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_process_managed_exception_gchandle
0x10d336851 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/libxamarin-debug.dylib : xamarin_main
0x10d217ced - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/5CB344E3-EC61-4720-92E6-4C8B91A67A85/link all.app/link all : main
0x7fff51a231fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdyld.dylib : start
This means:
* Move the parts of the ApplyPreserveAttribute step that we don't need for.NET
into a new MobileApplyPreserveAttribute step, and have mtouch and mmp use
that step instead of the ApplyPreserveAttribute step.
* Copy ApplyPreserveAttributeBase into dotnet-linker from the upstream tuner
source (with minor modifications) so that our ApplyPreserveAttribute step
compiles.
* Create a DotNetSubStepDispatcher class that we're going to use as our
substep dispatcher, create an instance of it and insert it into the list of
linker steps.
* Also a workaround for the lack of LinkContext.GetAssemblies (): add a step
that collects all the assemblies and stores them in a list, so that we can
have our own GetAssemblies implementation.
I filed a linker issue to see if we can get LinkContext.GetAssemblies ()
exposed to us: https://github.com/mono/linker/issues/1455.
Fixes this startup crash with the linkall test:
2020-08-26 19:47:10.219697+0200 link all[32709:6065783] Xamarin.iOS: Fatal error: failed to load the method 'ObjCRuntime.Runtime.Initialize'.
=================================================================
Native Crash Reporting
=================================================================
Got a abrt while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x104007b0e - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
0x103fb4437 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : mono_handle_native_crash
0x104007365 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
0x7fff51c005fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_platform.dylib : _sigtramp
0x0 - Unknown
0x7fff51af0b7c - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
0x103daad98 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_assertion_message
0x103dade77 - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_initialize
0x103dbf80b - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/libxamarin-debug.dylib : xamarin_main
0x103cd2f0d - /Users/rolf/Library/Developer/CoreSimulator/Devices/289E372A-501C-4499-A1A6-59C5B3B6A9AE/data/Containers/Bundle/Application/4A5F3968-6980-4E90-88A2-2E77AE039C40/link all.app/link all : main
0x7fff51a231fd - /Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 13.5.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libdyld.dylib : start
This is a partial/modified port of the initial linker support (bc88790201)
Co-authored-by: Sebastien Pouliot <sebastien@xamarin.com>
* [dotnet] Pass exception marshaling options to the linker configuration, and pass it along to the Application instance.
* [dotnet] Write the selected exception marshaling modes to the generated main file.
* [dotnet-linker] Set the default cooperative GC mode.
The code to select the default exception marshalling mode needs it.
* [mmp] Rename LinkMode.All to LinkMode.Full.
So that we can continue to use Enum.Parse<LinkMode> to parse 'Full' as the link mode.
* [dotnet] Implement support for our different link modes.
Tell the managed linker what to do with each input assembly depending the selected
link mode (link all, link sdk, don't link).
Once we have done all the check of the bot, set the status of the commit
to pending since the next task is going to run the tests.
This way we ensure that the pending status should be cleaned at the end
rather than be left behind.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
* Bump .NET 5 and the linker.
The old linker run into a problem that seems fixed in the new linker, so bump.
* [tests] Add a workaround for a .NET bug in the fsharplibrary tests.
It's now the default to produce reference assemblies, but the F# compiler
doesn't support producing reference assemblies, so when there's no reference
assembly afterwards the MSBuild tasks complain. So explicitly disable
reference assemblies for our F# library.
* [tests] Skip reference assemblies when iterating over produced assemblies in the .NET unit tests.
Also simplify the code a little bit.
This happens because producing reference assemblies is now the default.
Once we have done all the check of the bot, set the status of the commit
to pending since the next task is going to run the tests.
This way we ensure that the pending status should be cleaned at the end
rather than be left behind.
Add a Registrar step that is responsible for dealing with the registrar during the
build. Currently only supports the dynamic and partial static registrar, the full
static registrar support will come later.
Refactor Application creation to happen earlier, and to split out the cache
creation. This way we can create the Application instance before processing
the configuration, and as we process any configuration we can set properties
on the Application instance.
If we can't find the mscorlib assembly in the list of loaded assemblies, try to load
it explicitly. If we still can't find it the mscorlib assembly, look for System.Void
in any assembly. This shouldn't be a performance bottleneck, because we cache the
System.Void type, which means the lookup is only done once.
This makes System.Void lookup work when building with .NET as well, since there's
no mscorlib.dll there.
This is required when running mtouch and mmp to generate the partial static registrar
code for .NET.
* Make Driver.Verbosity the single place where we store the verbosity level.
* Respect any default verbosity by adding to the existing verbosity instead of
setting it directly.
* There's no need to set ErrorHelper.Verbosity, the Driver.Verbosity setter
already does it.
This means that for the .NET linker code we'll treat
~/.xamarin-bundler-verbosity like we treat ~/.mtouch-verbosity for mtouch, and
parse it to set the verbosity.
* Port the interdependent-binding-projects test to .NET (it's the simplest
test project we have with binding projects).
* Add a lot of the shared source code for mtouch/mmp to dotnet-linker, and
make it compile. Most issues were fixed by adding a few stubbed out classes,
since there are large chunks of the mtouch/mmp code we're not using yet, so
stubbing out while things are being implemented works fine.
* Add a step in dotnet-linker for loading the linker output (the linked
assemblies) into our bundler code.
* Add another step in dotnet-linker to extract native resources from binding
libraries.
* Augment the build process to take into account the native resources we found
in any binding libraries.
The tests are not executed in the same agent that sets the github
status, for that reason, the github comment is not giving the correct
bot name, but the name of the bot that executed the commit message and
not the tests.
Add an output var in the runTests step to set the bot name and pass it
to the comment job in a different bot.
Co-authored-by: Whitney Schmidt <whschm@microsoft.com>
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The credentials for maccore are downloaded to a pat file (to be found).
When we call make git-clean, because we do use the -x options, all
files are deleted, including the pat file.
We move to call git clean -xdf inside xamarin-macios, which will delete
the test result files.
Once we find the exact path pattern, we can update the make git-clean to
not remove them but this commit unblocks the failing CI builds.
Co-authored-by: Manuel de la Pena <mandel@microsoft.com>
The credentials for maccore are downloaded to a pat file (to be found).
When we call make git-clean, because we do use the -x options, all
files are deleted, including the pat file.
We move to call git clean -xdf inside xamarin-macios, which will delete
the test result files.
Once we find the exact path pattern, we can update the make git-clean to
not remove them but this commit unblocks the failing CI builds.
This fixes an issue where mtouch would complain about a missing --target-framework argument when it's not actually needed:
/Library/Frameworks/Xamarin.iOS.framework/Versions/Current/bin/mtouch --launchsim bin/iPhoneSimulator/Release/MyApp.app [...]
error MT0086: A target framework (--target-framework) must be specified.
what makes this worse is that passing --target-framework to mtouch makes
mlaunch fail, because mlaunch doesn't accept a --target-framework argument.
Turns out we don't actually _need_ to know, in every case we use this knowledge it's
a performance improvement to not process the framework assemblies, so skip this for
now, since there's no harm done (except to the planet) to do some extra processing
by processing all assemblies in these cases.
Rolf Bjarne Kvinge
VS MobileTools Engineering Service 2
Manuel de la Pena
Sebastien Pouliot
vs-mobiletools-engineering-service2
Mike Bond
Alex Soto
monojenkins
Filip Navara
Connor Adsit
Whitney Schmidt
Dorothy Tam