The Assembly.IsFrameworkAssembly property is used in two places:
* In Driver.IsBoundAssembly to return early when determining if an assembly has any NSObject subclasses: c1c5b9aac6/tools/mtouch/mtouch.cs (L1155-L1168)
* In Assembly.ExtractNativeLinkInfo to return early when looking for assemblies with LinkWith attributes: c1c5b9aac6/tools/common/Assembly.cs (L150-L154)
In both cases this definition of framework assembly works today and seems likely to work in the future as well.
I also went through and looked at all the usages of Profile.IsSdkAssembly, and it's used to:
* Decide which assemblies are selected for "link sdk"
* Decide which assemblies are considered an 'sdk' assembly for creating a user framework of all the sdk assemblies
* Bail out early when deciding whether:
* An assembly references the product assembly (Xamarin.iOS.dll, etc.)
* An assembly can contain references to UIWebView
* An assembly can contain user resources
* An assembly is a binding project / has third-party native resources
* An assembly needs the dynamic registrar
* An assembly has FieldAttributes whose native fields must be preserved by the native linker
In all cases our .NET definition of 'SDK' seems to work both for now and in the future.
There are also a few usages which does not apply to .NET, so I've ignored them:
* When looking for a few BCL APIs that must be preserved (MobileApplyPreserveAttribute.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
* When linking away parameter names (MonoTouchMarkStep.cs): this is to be done in the upstream .NET linker now, so it doesn't apply to our own code
Turns out we don't actually _need_ to know, in every case we use this knowledge it's
a performance improvement to not process the framework assemblies, so skip this for
now, since there's no harm done (except to the planet) to do some extra processing
by processing all assemblies in these cases.
The shared version isn't used by mmp yet as far as I can tell (mmp has its own logic
to copy assemblies), but sharing this code is the first step towards having the same
implementation as well.
This solves a rebuild problem if an assembly has an invalid or unsupported symbol
file, where we'd detect that the symbol file exists, and expect it to be copied,
but then the linker would drop it, causing us to always rebuild the app (this is
not the same as when a symbol file is out of date).
This happens for NUnitLite 3.12.0's nunit.framework.dll, which ships with an old-style
pdb.
Also add a warning that is shown when we detect that there's a symbol file, but it
couldn't be loaded for some reason.
* Move much of ErrorHandler.cs into a partial class in ErrorHandler.tools.cs,
which is referenced by mtouch and mmp (but not our runtime).
* Add ErrorHandler.runtime.cs for runtime-specific bits, including a simpler
version of ErrorHandler.Show. In particular this gets rid of the call to
Environment.Exit, which should never happen at runtime.
* Rename MonoTouchException and MonoMacException to ProductException, which
allows us to remove a lot of ifdefs.
* This required moving Application.LoadSymbols and Target.LoadSymbols to
shared mtouch/mmp code.
It's possible for a nuget to have dependencies that will bring satellite
assemblies into a project.
When doing so the satellite assemblies are copied to the right output
(e.g. `Debug`) directory, so they work fine at runtime. However it's
not 100% fine at build time, e.g. msbuild won't detected them as
satellite assemblies.
Something similar happens with `mtouch` (and `mmp`) since the satellite
assemblies won't be found inside a (culture-named) subdirectory from the
original assembly location. In our case this means the assemblies won't
be copied into the .app bundle (since we don't run from `Debug`) and
localization won't work properly.
The solution is to check for both the (culture-named) subdirectories
from the assembly location (like before) and, if nothing is found, also
try to locate them in the _build_ (like `Debug`) directory - so if
something else (nuget or custom build scripts) tried to outsmart the
build logic then we'll still bring those satellite assemblies in the
app bundle we produce.
https://github.com/xamarin/xamarin-macios/issues/7113
* Implement a different escaping/quoting algorithm for arguments to System.Diagnostics.Process.
mono changed how quotes should be escaped when passed to
System.Diagnostic.Process, so we need to change accordingly.
The main difference is that single quotes don't have to be escaped anymore.
This solves problems like this:
System.ComponentModel.Win32Exception : ApplicationName='nuget', CommandLine='restore '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable/CellCustomTable.sln' -Verbosity detailed -SolutionDir '/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories/ios-samples/WorkingWithTables/Part 3 - Customizing a Table\'s appearance/3 - CellCustomTable'', CurrentDirectory='/Users/vsts/agent/2.158.0/work/1/s/tests/sampletester/bin/Debug/repositories', Native error= Cannot find the specified file
at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0029f] in /Users/builder/jenkins/workspace/build-package-osx-mono/2019-08/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:778
ref: https://github.com/mono/mono/pull/15047
* Rework process arguments to pass arrays/lists around instead of quoted strings.
And then only convert to a string at the very end when we create the Process
instance.
In the future there will be a ProcessStartInfo.ArgumentList property we can
use to give the original array/list of arguments directly to the BCL so that
we can avoid quoting at all. These changes gets us almost all the way there
already (except that the ArgumentList property isn't available quite yet).
We also have to bump to target framework version v4.7.2 from v4.5 in several
places because of 'Array.Empty<T> ()' which is now used in more places.
* Parse linker flags from LinkWith attributes.
* [sampletester] Bump to v4.7.2 for Array.Empty<T> ().
* Fix typo.
* Rename GetVerbosity -> AddVerbosity.
* Remove unnecessary string interpolation.
* Remove unused variable.
* [mtouch] Simplify code a bit.
* Use implicitly typed arrays.
E.g. `MetalPerformanceShaders` is unusable in 10.3.1 on the simulator so
trying to build with the static registrar cause simlauncher to be rebuilt
with a **strong** (and not _weak_) reference to the framework. This cause
a crash then the application start
```
Dyld Error Message:
Library not loaded: /System/Library/Frameworks/MetalPerformanceShaders.framework/MetalPerformanceShaders
Referenced from: /Users/USER/Library/Developer/CoreSimulator/Devices/95679F95-CCE7-4733-8376-35E91E97039C/data/Containers/Bundle/Application/0F665B2D-ADAC-4CA0-BD04-33E968B3F268/FailerApp.app/FailerApp
Reason: no suitable image found. Did find:
/System/Library/Frameworks/MetalPerformanceShaders.framework/MetalPerformanceShaders: mach-o, but not built for iOS simulator
```
ref: https://github.com/xamarin/xamarin-macios/issues/5753
- Existing binding projects embed the native libraries within the assembly as managed resource
- This does not scale well and has performance implications
- This PR creates a new property, NoBindingEmbedding which when true processes the building and consumption of binding projects differently.
- Existing binding projects are not affected, they will continue as is
- I've written a full XM test suite and ported a subset to iOS. Since iOS only supports checked in projects, and I didn't want to make the existing situation worse by adding more, I only wrote tests that could use the existing test projects.
-When we complete some form of msbuild testing reform, we'll revisit these tests.
- Remove two files in MyiOSFrameworkBinding that are not used (we use copies elsewhere)
- Remove unnecessary sleep and fix broken touch command
- Output failing test log to console instead of test output
- VSfM does not handle thousands of lines of test failure message well
- Add ability to generate binding projects with LinkWith
Unify the code to detect frameworks that aren't supported in the simulator (we
had switches in two places, now this data is stored per framework).
Also remove some of the Metal frameworks for some of the platforms from these
lists (since they're now available in the simulator in some cases), which
fixes#4422.
It also seems CoreAudioKit is now available in the simulator (it wasn't in the
first Xcode 7 beta, but apparently added in a later beta. This made our
exclusion incorrect, but we never noticed).
https://github.com/xamarin/xamarin-macios/issues/4422
* [Compression] Ensure we use the correct lonking flags for older versions. Fixes#4129.
Libcompresison was added after iOs 9.0, TvOS 9.0, Mac 10.11 and Watch
2.0. We want to use weak in those older platforms.
Fixes issue https://github.com/xamarin/xamarin-macios/issues/4129
- https://github.com/xamarin/xamarin-macios/issues/3725
- These frameworks "CoreAudioKit Metal MetalKit MetalPerformanceShaders CoreNFC DeviceCheck"
were special cased, but that special case did do an SDK check.
- Create a helper method to share check
- Add test for MM0135
You were the preprocessor we wished C# had natively
Removing PMCS requires these changes:
* Remove XamCore from src/
* Remove XamCore from tools/
* Remove XamCore from runtime/
* nint/nuint enum conversion
* _compat_ enum conversion
* NSAction conversion
* Hand fix single API incorrectly converted by PMCS to unbreak compatibility
- Due to a bug in PMCS, the nuint was incorrectly converted in this API.
- However, as that ship as sailed, we must "fix" it until XAMCORE_4_0
* Update readme
* Bump macios-binaries
We already have this logic for frameworks we detect according to the namespace
of the used types, but not for frameworks we detect from P/Invokes.
Fix this by using the same framework exclusion logic for frameworks detected
from P/Invokes: don't link with frameworks not available in the current SDK.
https://bugzilla.xamarin.com/show_bug.cgi?id=59636
* [mmp] Track all sub-frameworks of ApplicationServices and CoreServices. Fixes#58415
The existing custom list does not cover the needs of user code,
e.g. a reference to LaunchServices did not bring CoreServices
https://bugzilla.xamarin.com/show_bug.cgi?id=58415
* [CoreNFC] Add CoreNFC Xcode 9 Beta 1 Bindings
CoreNFC is not exposed in iOS simulator, no framework/headers are found
makes sense since there is no NFC chip on it.
CoreNFC says it is available on tvOS but headers says just 3 enums
are so not enabling tvOS in this first beta.
* [mtouch] CoreNFC not available on sim, this makes mtouch aware of it
HFS normalizes filenames to Form D when files are stored. This means that an
assembly whose assembly name is stored in Form C might be stored in a file
whose filename is Form D (which you'll get if you use the Form C filename).
However, this is a problem when we've already loaded an assembly and if we
doesn't take normalization into account: we check the cache based on the
filename, but store in the cache based on the assembly name. If those two uses
different normalization schemes, bad things (bug #57266) happen.
So in these scenarios normalize strings before comparing them.
https://bugzilla.xamarin.com/show_bug.cgi?id=57266
* [mtouch] Improve how we make sure native symbols aren't stripped away. Fixes#51710 and #54417.
* Refactor required symbol collection to store more information about each
symbol (field, function, Objective-C class), and in general make the code
more straight forward.
* Implement support for generating source code that references these symbols,
and do this whenever we can't ask the native linker to keep these symbols
(when using bitcode). Additionally make it possible to do this manually, so
that the source code can be generated for non-bitcode platforms too (which
is useful if the number of symbols is enormous, in which case we might
surpass the maximum command-line length).
* Also make it possible to completely ignore native symbols, or ignore them on
a per-symbol basis. This provides a fallback for users if we get something
right and we try to preserve something that shouldn't be preserved (for
instance if it doesn't exist), and the user ends up with unfixable linker
errors.
* Don't collect Objective-C classes unless they're in an assembly with
LinkWith attributes. We don't need to preserve Objective-C classes in any
other circumstances.
* Implement everything for both Xamarin.iOS and Xamarin.Mac, and share the
code between them.
* Remove previous workaround for bug #51710, since it's no longer needed.
* Add tests.
https://bugzilla.xamarin.com/show_bug.cgi?id=54417https://bugzilla.xamarin.com/show_bug.cgi?id=51710
* [mtouch] Make sure to only keep symbols from the current app when code sharing.
This fixes a build problem with the interdependent-binding-projects test when
testing in Today Extension mode.
- Before this mmp was not adding -framework, -weak_framework consistently on non-static registrar use cases
- GatherFrameworks was previously not ported from mtouch, and did not work as DeploymentTarget was unset in mmp
- Added verbose prints so users can determine why various framework linkages are added
- Fixed an issue where duplicate were being added due to HandleFramework shoving args by hand
- Tested with auto test and https://github.com/chamons/xm-version-regression-test manual test
Implement support for sharing both code and resources between app extensions
and their container app:
* AOT-compiled code. Each shared assembly is only AOT-compiled once, and if
the assembly is built to a framework or dynamic library, it will also only
be included once in the final app (as a framework or dynamic library in the
container app, referenced directly by the app extension). If the assemblies
are built to static objects there won't be any size improvements in the app,
but the build will be much faster, because the assemblies will only be AOT-
compiled once.
* Any resources related to managed assemblies (debug files, config files,
satellite assemblies) will be put in the container app only.
Since these improvements are significant, code sharing will be enabled by
default.
Test results
============
For an extreme test project with 7 extensions (embedded-frameworks)[1]:
with code sharing cycle 9 difference
build time 1m 47s 3m 33s -1m 46s = ~50% faster
app size 26 MB 131 MB -105 MB = ~80% smaller
For a more normal test project (MyTabbedApplication)[2] - this is a simple application with 1 extension:
with code sharing cycle 9 difference
build time 0m 44s 0m 48s -4s = ~ 8% faster
app size 23 MB 37 MB -15 MB = ~40% smaller
Another tvOS app with one extension also show similar gains (MyTVApp)[3]:
with code sharing cycle 9 difference
build time 0m 22s 0m 48s -26s = ~54% faster
app size 22 MB 62 MB -40 MB = ~65% smaller
[1]: https://github.com/rolfbjarne/embedded-frameworks
[2]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTabbedApplication
[3]: https://github.com/xamarin/xamarin-macios/tree/cycle9/msbuild/tests/MyTVApp
Warn if mtouch loads an assembly from a different location than requested
(which might be because there are multiple assemblies with the same name).
Also rework the MT0023 check a bit by explicitly loading the root assembly
first, and then detecting if any loaded assemblies matches the root assembly.
This results in code that's a bit more obvious, and it also works correctly
with extensions (previously the entire MT0023 check was skipped for
extensions).
Create a custom AssemblyCollection class that contains a dictionary with
assembly identity (name) -> Assembly mapping.
This also means that we can detect if we end up loading multiple assemblies
with the same identity, and show an error in that case (even if that case
should never happen since we cache assemblies based on the identity, it's nice
to have code that ensures it).
Resource files for mac should be copied to the app bundle
* Move the Satellite code used by mac to tools/common/Assembly.cs
* Add EmbeddedResources test to xammac_tests
Event sequence:
* mtouch is executed with the linker disabled.
* The linker pipeline copies all input assemblies (since the linker is
disabled the assemblies don't change) into the PreBuild directory. This will
keep the original timestamps of the input assemblies.
* mtouch is executed again, when none of the input assemblies changed.
* The linker pipeline will re-execute, because it will see that at least one
of the input assemblies (at least the .exe) is newer than at least one of
the assemblies in the PreBuild directory (usually a framework assembly,
because those have the original timestamp from their install location).
Fix:
Touch all the assemblies in the PreBuild directory after the linker pipeline
executes the first time. This way the second time mtouch is executed, it will
find that all assemblies in the PreBuild directory have timestamps later than
all the input assemblies, so it will load the cached linked assemblies,
instead of re-executing the linker pipeline.
The native linker treats object files (.o) and static libraries (.a files,
which are archives of .o files) differently.
The native linker will always include object files into the executable:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ld foo.o -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
0000000000000fe0 T _xxx
However, if the object file is inside a static library:
$ echo "void xxx () {}" > foo.m
$ clang -c foo.m -o foo.o -arch x86_64
$ ar cru foo.a foo.o
$ ld foo.a -dylib -o foo.dylib -macosx_version_min 10.12 -arch x86_64
$ nm foo.dylib
<no output>
This means that our testing library (libtest.a) which is a fat library of
_object files_, do not show the problems reported in bug #51548.
So:
a) I've fixed the creation of libtest.a to be a fat library of _static
libraries_. This causes the `FastDev_LinkWithTest` test to fail exactly
like in bug #51548.
b) I've made mtouch pass `-u <native symbol>` to the native linker, for every
native symbol referenced in a managed assembly, when creating a dylib.
Amazingly this seems to work fine even with symbols to Objective-C classes
(`_OBJC_CLASS_$_<class name>`).
c) This also required adding support for collecting the Objective-C names of
all managed types registered with Objective-C to the linker. The
information is already available in the static registrar, but that would
require us to make sure the static registrar is executed before compiling
dylibs, which means those two tasks won't be able to run in parallel (also
there's no guarantee we'll even run the static registrar).
https://bugzilla.xamarin.com/show_bug.cgi?id=51548
Simplify code a bit to avoid constant null checking and also take advantage of
the fact that HashSet.Add returns if the value was added or not (to avoid a
Contains check).
So that there can be multiple caches in the same process (which we'll have
once mtouch can compile extensions and the container app in the same process).
-lsqlite3 is a linker flag, not a file to be linked with, so when
automatically determining that we need to pass -lsqlite3 we need to put it in
the right list of linker information.
Otherwise we may end up passing `-force_load -lsqlite3` to the linker (if the
assembly's ForceLoad flag is set), which won't compile.
https://bugzilla.xamarin.com/show_bug.cgi?id=49220
Rolf Bjarne Kvinge
Waleed Chaudhry
Sebastien Pouliot
Marek Safar
Chris Hamons
Martin Baulig
Manuel de la Pena
Sebastien Pouliot
Alex Soto
Alex Soto
Timothy Risi