Some samples from the `ios-samples` repo failed with an error in running
`ditto`:
```
Tool /usr/bin/ditto execution started with arguments: -rsrc /Users/ankit/dev/ios-samples/ios9/FilterDemoApp/FilterDemoAppExtension/bin/iPhoneSimulator/Debug/FilterDemoAppExtension.appex bin/iPhoneSimulator/Debug/FilterDemoApp.app/PlugIns/FilterDemoAppExtension.appex Environment variables being passed to the tool:
ditto: can't get real path for source '/Users/ankit/dev/ios-samples/ios9/FilterDemoApp/FilterDemoAppExtension/bin/iPhoneSimulator/Debug/FilterDemoAppExtension.appex' Tool /usr/bin/ditto execution finished.
/Library/Frameworks/Mono.framework/External/xbuild/Xamarin/iOS/Xamarin.iOS.Common.targets : error : Tool exited with code: 1. Output: ditto: can't get real path for source '/Users/ankit/dev/ios-samples/ios9/FilterDemoApp/FilterDemoAppExtension/bin/iPhoneSimulator/Debug/FilterDemoAppExtension.appex' [/Users/ankit/dev/ios-samples/ios9/FilterDemoApp/FilterDemoApp/FilterDemoApp.csproj]
```
The directory
`/Users/ankit/dev/ios-samples/ios9/FilterDemoApp/FilterDemoAppExtension/bin/iPhoneSimulator/Debug/FilterDemoAppExtension.appex`
did exist *after* the full build was over, but just before `ditto` ran,
it didn't[1].
But it should have been created as part of the `FilterDemoAppExtension`
build, before `ditto` was called as the `FilterDemoAppExtension` project
is referenced by `FilterDemoApp`. And this builds fine with xbuild.
Debugging+hair-pulling revealed:
a. `FilterDemoApp.csproj` has a `ProjectReference` pointing to
`FilterDemoAppExtension.csproj`
b. xbuild figures out the build order based on the .sln and the project
references and builds the individual projects in that order.
c. But msbuild doesn't do that anymore. It seems to build in "some"
order (probably the order in which the projects appear in the sln), and
depends on each project's `ResolveProjectReferences` target building
such referenced projects! So, msbuild starts with building
`FilterDemoApp` project and should have built the extension project, as
it is referenced, but that does not happen.
d. It does not happen because XI, in `_SeparateAppExtensionReferences`,
moves any `Extension` projects from `@(ProjectReferences)` to other
items to handle them itself in `Xamarin.iOS.Common.targets`. So, the
`ResolveProjectReferences` does not see the extension project at all.
e. But XI targets handle this in `_ResolveAppExtensionReferences`, just
like `ResolveProjectReferences` from x/msbuild targets. BUT.. they depend
on the xbuild's behavior, where if you are building a `.sln` file, then
any dependent projects have already been built, and thus it just skips
them!!
- Since, msbuild no longer does this.. the extension project is
not built and instead proceeds to creating the apple bundle.
So, the fix is effectively to remove the `$(BuildingSolutionFile) == true`
check from that target.
- The same change is required for Watch extensions target too.
- And the corresponding changes for XamMac
- This fixes ~7 demos in ios-samples.
---
1. The directory and the corresponding files existed *after* the full
build but not before the `ditto` execution because after `ditto`
caused the `FilterDemoApp` project to fail, msbuild continued to build
the remaining `FilterDemoAppExtension` project! too late :)
This makes us only put packages in one directory (saves disk space and time),
and it also makes project files in multiple solutions work properly
(mtouch.csproj is in tests/tests.sln and tests/mtouch/mtouch.sln).
Normally the removal of `HandleAction` would be a breaking change (and
require a stub). However Xamarin.WatchOS.dll is not final before C9 so
this fix will be backported to `master` and `cycle9` branches.
Considering the following binding code:
public enum HMAccessoryCategoryType {
[Field ("HMAccessoryCategoryTypeGarageDoorOpener")]
DoorOpener,
GarageDoorOpener = DoorOpener,
}
We must
1. Ensure that `HMAccessoryCategoryType.DoorOpener.GetConstant () ==
HMAccessoryCategoryType.GarageDoorOpener.GetConstant ()`;
This is done by using the numeric value of the enum member (instead of the name)
2. Ensure that `HMAccessoryCategoryTypeExtensions.GetValue ("HMAccessoryCategoryTypeGarageDoorOpener")`
always return the same enum value, i.e. it can **not** change between
XI versions (e.g. due to reflection ordering) as it could break
comparison code;
This is done by only adding a map to the member that has a [Field] and
means that:
2.1. the _favorite_ enum member should be the one with the [Field]; and
2.2. a [Field] value can only be used once per enum (or else we report
an BI1046 error). This also solve the duplicate code generation for
the constant loading code;
Reference:
* https://bugzilla.xamarin.com/show_bug.cgi?id=46285
Use metadata tokens instead of strings to find types and methods.
This makes the code to find methods more compact (a lot less strings in the
executable, and additionally in most cases a compact representation (32-bit
integer) of the corresponding metadata token and additional information can be
used, which results in less executable code (fewer parameters to methods,
etc)), resulting in smaller executables.
Size savings are around 200kb for dont link apps, and 20-60kb for linked apps
(this obviously varies a lot depending on how much has to registered by the
registrar).
| | Before | After | Diff |
|----------------|--------------:|--------------:|------------------:|
| dontlink/32bit | 102.810.144 | 102.609.456 | -200.688 = -0,20% |
| dontlink/64bit | 107.420.576 | 107.221.792 | -198.784 = -0,19% |
| linksdk/32bit | 40.957.296 | 40.936.864 | -20.432 = -0,05% |
| linksdk/64bit | 43.113.136 | 43.093.936 | -19.200 = -0,04% |
| linkall/32bit | 38.410.032 | 38.348.288 | -61.744 = -0,16% |
| linkall/64bit | 40.315.200 | 40.267.344 | -47.856 = -0,12% |
Additionally I've removed the `lazy_map` dictionary, which we populated at
startup and was used to map between Class instances and the corresponding
managed type's FullName, and instead iterate over a native array of Class ->
metadata token mappings whenever we need to look up the managed type for a
certain Class instance.
This is slightly slower for each type we need to look up (for a non-linked app
there might be a 2000-3000 entries in the native array, which would be
iterated instead of using a hashtable lookup), but it's only done once per
type and there's a significant startup memory improvement.
For a non-linked test app I get the following using the Xamarin profiler:
| | Before | After | Diff |
|-------------------|--------:|--------:|----------------:|
| Memory allocated | 2,8 MB | 2,4 MB | -0,4 MB = -14 % |
| Objects allocated | 43678 | 38463 | -5215 = -12 % |
| Private bytes | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
| Working set | 26,6 MB | 24,4 MB | -2,2 MB = -8,3% |
Right now the logic exists in a few places, both in and outside the
linker. We recently began to use part of the linker pipeline in normal /
all builds so it's easier to share (and unify) the code now.
The real gain is to avoid copying assemblies, in particular large ones,
more than strictly needed while building.
E.g. a build including a very large 1.3GB assembly, with several
native libraries embedded, save a lot of time avoiding the rewrites
mtouch (before)
Total time: 64202 ms
mtouch (after)
Total time: 34840 ms
* Add XM support for RemoveUserResourcesSubStep
* Tests supplied by @chamons
Use the 'unique' argument to MobileProvisionIndex's lookup methods
to only get back the most recent versions of each provisioning
profile so that we don't accidentally pick an older version.
Run the partial static registrar separately for 32-bit and 64-bit, since this
is required with the upcoming changes to embed metadata tokens inside the
generated output, because the metadata tokens are different between the 32-bit
and 64-bit versions of Xamarin.iOS.dll.
Also make sure to properly resolve the 32-bit and 64-bit assemblies correctly
(by setting the ArchDirectory on the assembly resolver), so that we don't pick
up the reference assembly (which does not have the right metadata tokens).
Additionally stop running the partial static registrar for MonoTouch.Dialog-1,
since with the upcoming changes to use metadata tokens in the generated
output, we won't support registering anything more than once. This shouldn't
make much of an impact, because MonoTouch.Dialog-1 is fairly small, and
doesn't take long to register in the dynamic registrar. For device builds (or
when the static registrar is selected) this has no effect, since in that case
we're registering everythinga anyway.
Link the executable with LinkWith attributes' libraries in the simulator when
doing incremental builds, since we don't create dylibs there.
Fixes several mtouch test failures:
1) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_LinkAll(Unified)
2) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_LinkAll(TVOS)
3) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_LinkSDK(Unified)
4) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_LinkSDK(TVOS)
5) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_NoLink(Unified)
6) Test Failure : Xamarin.MTouch.FastDev_NoFastSim_NoLink(TVOS)
7) Test Failure : Xamarin.MTouch.FastDev_Sim(Unified)
8) Test Failure : Xamarin.MTouch.FastDev_Sim(TVOS)
Update mdb files even if the corresponding assembly didn't change, because the
mdb can change even if the assembly didn't (if whitespace was modified in the
source code, causing code lines to move).
https://bugzilla.xamarin.com/show_bug.cgi?id=39535
If the cache is invalid we print a warning:
> A full rebuild will be performed because the cache is either incomplete or entirely missing.
and set `Driver.Force = true;` (in Application.cs).
This later means that extracting the native code is done on each target:
```
public static bool IsUptodate (string source, string target)
{
if (Driver.Force)
return false;
```
even if this is identical between 32 and 64 bits (targets). That's
inefficient and, for large binding libraries (e.g. > 1GB), has a
noticable impact on build time (see timestamps).
Considering that the cache is cleaned (when detected as invalid) then
this Force condition is not really needed.
E.g. in `IsUptodate`
* the first time (arch) it's called will have to extract the native library;
* if a 2nd arch is built (fat) then it will be found as present and will
not be extracted again
Removing the `Driver.Force = true;` in this condition let the `-f` option
continue to extract it twice, which can be useful in debugging and testing.
As such the check is not removed from `IsUptodate`
Timestamps (before)
Setup: 25 ms
Resolve References: 1605 ms
Extracted native link info: 10465 ms
...
Timestamps (after)
Setup: 24 ms
Resolve References: 1560 ms
Extracted native link info: 5473 ms
...
Total build times (from XS) was around 90-100 seconds so 5 seconds is
about 10%. The actual savings will depend on how much native code needs
to be extracted, but it should help most release builds (almost always
fat builds).
* Comment is wrong: the code is never executed in parallel
* If reached then RemoveResources will be called and already check for
symbols (and load them when needed). Just a simplification (it won't
really save time as it's not loaded twice)
IIRC this used to be needed with a (rather old) version of Cecil.
The current code does nothing as the Load will only hit the cache
and not other properties are changed (compared to the old commented
code)
* [mtouch] Add missing HasParameters check inside linker and static registrar
Without the checks new, empty collections can be allocated and their
whole and only purpose will be to iterate up to 0 (nop).
The checks saves a small amount of memory (collections) and time.
* [registrar] Fix method comparison when they have no parameters
The SDK does not ship any, non-tool, .exe files. In fact .exe _should_
never need to be resolved since the main .exe is always given as an input
to mtouch.
Still it's possible (if quite uncommon) to refer to other .exe assemblies
just like if they were .dll. However those can only come from the
RootDirectory (and not the other places that ship with the SDK).
This "fix" ensure lookups for *.exe is done only inside the RootDirectory
location, speeding up (a bit) resolving assemblies.
Native libraries are already linked into the dylib for the binding assembly,
which means that if we also link it into the main executable, the native code
ends up twice in the app (which is bad for many reasons).
https://bugzilla.xamarin.com/show_bug.cgi?id=42473
The container app may not reference the same third-party frameworks as
extensions, which means that we must make sure the extension's frameworks are
also included in the app bundle.
So when building extensions save a list of all third-party frameworks, and
then read that list and include those frameworks when building the main app.
https://bugzilla.xamarin.com/show_bug.cgi?id=45800
This fixes the following mmptests:
1) Classic_NewRefCount_Warns (Xamarin.MMP.Tests.MMPTests.Classic_NewRefCount_Warns)
2) SystemMono_SmokeTest (Xamarin.MMP.Tests.MMPTests.SystemMono_SmokeTest)
That fails like this with earlier versions of Mono 4.8:
Undefined symbols for architecture i386:
"_mono_btls_x509_lookup_method_mono_init", referenced from:
-u command line option
"_mono_btls_x509_name_list_add", referenced from:
-u command line option
"_mono_btls_x509_name_list_free", referenced from:
-u command line option
"_mono_btls_x509_name_list_get_count", referenced from:
-u command line option
"_mono_btls_x509_name_list_get_item", referenced from:
-u command line option
"_mono_btls_x509_name_list_new", referenced from:
-u command line option
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error MM5109: Native linking failed with error code 1. Check build log for details.
Managed exception marshaling interferes with the debugger, because it adds
exception handlers to executing code, which makes the Mono runtime think an
exception is handled when logically it's not (although technically it is).
The consequence is that the IDEs will only be notified when we re-throw the
exception after catching it, making it impossible for the IDEs to stop when
the exception is thrown (they will instead stop when we re-throw the
exception).
So disable managed exception marshaling (unless the user changed the default
behavior) when a debugger is attached.
This is the same behavior as Xamarin.Android.
https://bugzilla.xamarin.com/show_bug.cgi?id=45116
* [AVFoundation] Add new constructor for AVAssetExportSession.
Add a new constructor that takes an enum instead of a string to have
a strong typed version of the default constructor to reduce possible
errors when passing a random string.
Fixes bug https://bugzilla.xamarin.com/show_bug.cgi?id=44443
* Use smart enums as requested in the PR review.
* Set manually the enum int values.
This makes it possible to set linker flags per assembly:
[assembly: LinkWith (LinkerFlags = "-lsqlite3")]
Which is required when incremental builds is enabled and a particular assembly
needs special linker flags (because we don't propagate the global -gcc_flags
to each dylib we build when doing incremental builds).
Also add an option to set the dlsym mode for an assembly (using the LinkWith
attribute).
Alex Soto
Rolf Bjarne Kvinge
Manuel de la Pena
Ankit Jain
Vincent Dondain
Sebastien Pouliot
Timothy Risi
Miguel de Icaza
Jeffrey Stedfast
Chris Hamons