2016-04-21 15:57:02 +03:00
/ *
* Copyright 2014 Xamarin Inc . All rights reserved .
*
* Authors :
* Rolf Bjarne Kvinge < rolf @xamarin . com >
*
* /
using System ;
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
using System.Collections.Generic ;
2016-04-21 15:57:02 +03:00
using System.Diagnostics ;
2017-08-22 10:40:26 +03:00
using System.Globalization ;
2016-04-21 15:57:02 +03:00
using System.IO ;
using System.Linq ;
using System.Runtime.InteropServices ;
using System.Text ;
[mtouch] Rework how tasks are built.
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
2017-01-26 12:56:55 +03:00
using System.Threading.Tasks ;
2016-04-21 15:57:02 +03:00
2018-11-05 10:16:29 +03:00
using Xamarin.MacDev ;
2016-04-21 15:57:02 +03:00
using Xamarin.Utils ;
2018-02-05 18:26:29 +03:00
using ObjCRuntime ;
2016-04-21 15:57:02 +03:00
namespace Xamarin.Bundler {
public partial class Driver {
2016-12-23 20:50:35 +03:00
static void AddSharedOptions ( Application app , Mono . Options . OptionSet options )
2016-05-11 13:24:55 +03:00
{
2018-11-05 10:16:29 +03:00
options . Add ( "sdkroot=" , "Specify the location of Apple SDKs, default to 'xcode-select' value." , v = > sdk_root = v ) ;
options . Add ( "no-xcode-version-check" , "Ignores the Xcode version check." , v = > { min_xcode_version = null ; } , true /* This is a non-documented option. Please discuss any customers running into the xcode version check on the maciosdev@ list before giving this option out to customers. */ ) ;
2017-02-14 19:46:26 +03:00
options . Add ( "warnaserror:" , "An optional comma-separated list of warning codes that should be reported as errors (if no warnings are specified all warnings are reported as errors)." , v = >
{
try {
if ( ! string . IsNullOrEmpty ( v ) ) {
foreach ( var code in v . Split ( new char [ ] { ',' } , StringSplitOptions . RemoveEmptyEntries ) )
ErrorHelper . SetWarningLevel ( ErrorHelper . WarningLevel . Error , int . Parse ( code ) ) ;
} else {
ErrorHelper . SetWarningLevel ( ErrorHelper . WarningLevel . Error ) ;
}
} catch ( Exception ex ) {
ErrorHelper . Error ( 26 , ex , "Could not parse the command line argument '{0}': {1}" , "--warnaserror" , ex . Message ) ;
}
} ) ;
options . Add ( "nowarn:" , "An optional comma-separated list of warning codes to ignore (if no warnings are specified all warnings are ignored)." , v = >
{
try {
if ( ! string . IsNullOrEmpty ( v ) ) {
foreach ( var code in v . Split ( new char [ ] { ',' } , StringSplitOptions . RemoveEmptyEntries ) )
ErrorHelper . SetWarningLevel ( ErrorHelper . WarningLevel . Disable , int . Parse ( code ) ) ;
} else {
ErrorHelper . SetWarningLevel ( ErrorHelper . WarningLevel . Disable ) ;
}
} catch ( Exception ex ) {
ErrorHelper . Error ( 26 , ex , "Could not parse the command line argument '{0}': {1}" , "--nowarn" , ex . Message ) ;
}
} ) ;
2016-12-23 20:50:35 +03:00
options . Add ( "coop:" , "If the GC should run in cooperative mode." , v = > { app . EnableCoopGC = ParseBool ( v , "coop" ) ; } , hidden : true ) ;
2017-03-17 01:14:42 +03:00
options . Add ( "sgen-conc" , "Enable the *experimental* concurrent garbage collector." , v = > { app . EnableSGenConc = true ; } ) ;
2016-12-22 22:11:50 +03:00
options . Add ( "marshal-objectivec-exceptions:" , "Specify how Objective-C exceptions should be marshalled. Valid values: default, unwindmanagedcode, throwmanagedexception, abort and disable. The default depends on the target platform (on watchOS the default is 'throwmanagedexception', while on all other platforms it's 'disable')." , v = > {
2016-05-11 13:47:26 +03:00
switch ( v ) {
case "default" :
2016-12-23 20:50:35 +03:00
app . MarshalObjectiveCExceptions = MarshalObjectiveCExceptionMode . Default ;
2016-05-11 13:47:26 +03:00
break ;
case "unwindmanaged" :
case "unwindmanagedcode" :
2016-12-23 20:50:35 +03:00
app . MarshalObjectiveCExceptions = MarshalObjectiveCExceptionMode . UnwindManagedCode ;
2016-05-11 13:47:26 +03:00
break ;
case "throwmanaged" :
case "throwmanagedexception" :
2016-12-23 20:50:35 +03:00
app . MarshalObjectiveCExceptions = MarshalObjectiveCExceptionMode . ThrowManagedException ;
2016-05-11 13:47:26 +03:00
break ;
case "abort" :
2016-12-23 20:50:35 +03:00
app . MarshalObjectiveCExceptions = MarshalObjectiveCExceptionMode . Abort ;
2016-05-11 13:47:26 +03:00
break ;
case "disable" :
2016-12-23 20:50:35 +03:00
app . MarshalObjectiveCExceptions = MarshalObjectiveCExceptionMode . Disable ;
2016-05-11 13:47:26 +03:00
break ;
default :
2016-12-22 22:11:50 +03:00
throw ErrorHelper . CreateError ( 26 , "Could not parse the command line argument '{0}': {1}" , "--marshal-objective-exceptions" , $"Invalid value: {v}. Valid values are: default, unwindmanagedcode, throwmanagedexception, abort and disable." ) ;
2016-05-11 13:47:26 +03:00
}
} ) ;
2016-12-22 22:11:50 +03:00
options . Add ( "marshal-managed-exceptions:" , "Specify how managed exceptions should be marshalled. Valid values: default, unwindnativecode, throwobjectivecexception, abort and disable. The default depends on the target platform (on watchOS the default is 'throwobjectivecexception', while on all other platform it's 'disable')." , v = > {
2016-05-11 13:47:26 +03:00
switch ( v ) {
case "default" :
2016-12-23 20:50:35 +03:00
app . MarshalManagedExceptions = MarshalManagedExceptionMode . Default ;
2016-05-11 13:47:26 +03:00
break ;
case "unwindnative" :
case "unwindnativecode" :
2016-12-23 20:50:35 +03:00
app . MarshalManagedExceptions = MarshalManagedExceptionMode . UnwindNativeCode ;
2016-05-11 13:47:26 +03:00
break ;
case "throwobjectivec" :
case "throwobjectivecexception" :
2016-12-23 20:50:35 +03:00
app . MarshalManagedExceptions = MarshalManagedExceptionMode . ThrowObjectiveCException ;
2016-05-11 13:47:26 +03:00
break ;
case "abort" :
2016-12-23 20:50:35 +03:00
app . MarshalManagedExceptions = MarshalManagedExceptionMode . Abort ;
2016-05-11 13:47:26 +03:00
break ;
case "disable" :
2016-12-23 20:50:35 +03:00
app . MarshalManagedExceptions = MarshalManagedExceptionMode . Disable ;
2016-05-11 13:47:26 +03:00
break ;
default :
2016-12-22 22:11:50 +03:00
throw ErrorHelper . CreateError ( 26 , "Could not parse the command line argument '{0}': {1}" , "--marshal-managed-exceptions" , $"Invalid value: {v}. Valid values are: default, unwindnativecode, throwobjectivecexception, abort and disable." ) ;
2016-05-11 13:47:26 +03:00
}
} ) ;
2017-01-11 23:10:39 +03:00
options . Add ( "j|jobs=" , "The level of concurrency. Default is the number of processors." , v = > {
Jobs = int . Parse ( v ) ;
} ) ;
2017-04-19 14:17:05 +03:00
options . Add ( "embeddinator" , "Enables Embeddinator targetting mode." , v = > {
app . Embeddinator = true ;
} , true ) ;
[mtouch] Improve how we make sure native symbols aren't stripped away. Fixes #51710 and #54417. (#2162)
* [mtouch] Improve how we make sure native symbols aren't stripped away. Fixes #51710 and #54417.
* Refactor required symbol collection to store more information about each
symbol (field, function, Objective-C class), and in general make the code
more straight forward.
* Implement support for generating source code that references these symbols,
and do this whenever we can't ask the native linker to keep these symbols
(when using bitcode). Additionally make it possible to do this manually, so
that the source code can be generated for non-bitcode platforms too (which
is useful if the number of symbols is enormous, in which case we might
surpass the maximum command-line length).
* Also make it possible to completely ignore native symbols, or ignore them on
a per-symbol basis. This provides a fallback for users if we get something
right and we try to preserve something that shouldn't be preserved (for
instance if it doesn't exist), and the user ends up with unfixable linker
errors.
* Don't collect Objective-C classes unless they're in an assembly with
LinkWith attributes. We don't need to preserve Objective-C classes in any
other circumstances.
* Implement everything for both Xamarin.iOS and Xamarin.Mac, and share the
code between them.
* Remove previous workaround for bug #51710, since it's no longer needed.
* Add tests.
https://bugzilla.xamarin.com/show_bug.cgi?id=54417
https://bugzilla.xamarin.com/show_bug.cgi?id=51710
* [mtouch] Make sure to only keep symbols from the current app when code sharing.
This fixes a build problem with the interdependent-binding-projects test when
testing in Today Extension mode.
2017-06-02 19:29:19 +03:00
options . Add ( "dynamic-symbol-mode:" , "Specify how dynamic symbols are treated so that they're not linked away by the native linker. Valid values: linker (pass \"-u symbol\" to the native linker), code (generate native code that uses the dynamic symbol), ignore (do nothing and hope for the best). The default is 'code' when using bitcode, and 'linker' otherwise." , ( v ) = > {
switch ( v . ToLowerInvariant ( ) ) {
case "default" :
app . SymbolMode = SymbolMode . Default ;
break ;
case "linker" :
app . SymbolMode = SymbolMode . Linker ;
break ;
case "code" :
app . SymbolMode = SymbolMode . Code ;
break ;
case "ignore" :
app . SymbolMode = SymbolMode . Ignore ;
break ;
default :
throw ErrorHelper . CreateError ( 26 , "Could not parse the command line argument '{0}': {1}" , "--dynamic-symbol-mode" , $"Invalid value: {v}. Valid values are: default, linker, code and ignore." ) ;
}
} ) ;
options . Add ( "ignore-dynamic-symbol:" , "Specify that Xamarin.iOS/Xamarin.Mac should not try to prevent the linker from removing the specified symbol." , ( v ) = > {
app . IgnoredSymbols . Add ( v ) ;
} ) ;
2018-04-13 15:50:28 +03:00
options . Add ( "root-assembly=" , "Specifies any root assemblies. There must be at least one root assembly, usually the main executable." , ( v ) = > {
2017-11-16 18:02:34 +03:00
app . RootAssemblies . Add ( v ) ;
} ) ;
[mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp. (#3242)
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
2018-01-23 13:33:48 +03:00
options . Add ( "optimize=" , "A comma-delimited list of optimizations to enable/disable. To enable an optimization, use --optimize=[+]remove-uithread-checks. To disable an optimizations: --optimize=-remove-uithread-checks. Use '+all' to enable or '-all' disable all optimizations. Only compiler-generated code or code otherwise marked as safe to optimize will be optimized.\n" +
"Available optimizations:\n" +
" dead-code-elimination: By default always enabled (requires the linker). Removes IL instructions the linker can determine will never be executed. This is most useful in combination with the inline-* optimizations, since inlined conditions almost always also results in blocks of code that will never be executed.\n" +
" remove-uithread-checks: By default enabled for release builds (requires the linker). Remove all UI Thread checks (makes the app smaller, and slightly faster at runtime).\n" +
#if MONOTOUCH
2019-03-15 17:05:53 +03:00
" inline-isdirectbinding: By default enabled unless the interpreter is enabled (requires the linker). Tries to inline calls to NSObject.IsDirectBinding to load a constant value. Makes the app smaller, and slightly faster at runtime.\n" +
[mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp. (#3242)
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
2018-01-23 13:33:48 +03:00
#else
2018-03-02 16:33:27 +03:00
" inline-isdirectbinding: By default disabled, because it may require the linker. Tries to inline calls to NSObject.IsDirectBinding to load a constant value. Makes the app smaller, and slightly faster at runtime.\n" +
[mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp. (#3242)
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
2018-01-23 13:33:48 +03:00
#endif
#if MONOTOUCH
2019-03-15 17:05:53 +03:00
" remove-dynamic-registrar: By default enabled when the static registrar is enabled and the interpreter is not used. Removes the dynamic registrar (makes the app smaller).\n" +
[mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp. (#3242)
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
2018-01-23 13:33:48 +03:00
" inline-runtime-arch: By default always enabled (requires the linker). Inlines calls to ObjCRuntime.Runtime.Arch to load a constant value. Makes the app smaller, and slightly faster at runtime.\n" +
#endif
Optimize calls to BlockLiteral.SetupBlock to inject the block signature. (#3391)
* [linker] Optimize calls to BlockLiteral.SetupBlock to inject the block signature.
Optimize calls to BlockLiteral.SetupBlock[Unsafe] to calculate the block
signature at build time, and inject it into the call site.
This makes block invocations 10-15x faster (I've added tests that asserts at
least an 8x increase).
It's also required in order to be able to remove the dynamic registrar code in
the future (since calculating the block signature at runtime requires the
dynamic registrar).
* [mtouch/mmp] Add support for reporting errors/warnings that point to the code line causing the error/warning.
Add support for reporting errors/warnings that point to the code line causing
the error/warning by adding ErrorHelper overloads that take the exact
instruction to report (previously we defaulted to the first line/instruction
in a method).
* [tests] Add support for asserting filename/linenumber in warning messages.
* Make all methods that manually create BlockLiterals optimizable.
* [tests] Create a BaseOptimizeGeneratedCodeTest test that's included in both XI's and XM's link all test.
* [tests] Add link all test (for both XI and XM) to test the BlockLiteral.SetupBlock optimization.
* [tests] Add mtouch/mmp tests for the BlockLiteral.SetupBlock optimization.
* [tests][linker] Make the base test class abstract, so tests in the base class aren't executed twice.
* [tests][linker] Don't execute linkall-only tests in linksdk.
The optimization tests only apply when the test assembly is linked, and that
only happens in linkall, so exclude those tests in linksdk.
* [tests][mmptest] Update test according to mmp changes.
Fixes these test failures:
1) Failed : Xamarin.MMP.Tests.MMPTests.MM0132("inline-runtime-arch")
The warning 'MM0132: Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.' was not found in the output:
Message #1 did not match:
actual: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
Message #2 did not match:
actual: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'inline-runtime-arch'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
2) Failed : Xamarin.MMP.Tests.MMPTests.MM0132("foo")
The warning 'MM0132: Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.' was not found in the output:
Message #1 did not match:
actual: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
Message #2 did not match:
actual: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size, blockliteral-setupblock.'
expected: 'Unknown optimization: 'foo'. Valid optimizations are: remove-uithread-checks, dead-code-elimination, inline-isdirectbinding, inline-intptr-size.'
* [tests][linker] Fix typo.
Fixes this test failure:
1) SetupBlock_CustomDelegate (Linker.Shared.BaseOptimizeGeneratedCodeTest.SetupBlock_CustomDelegate)
Counter
Expected: 1
But was: 2
* [registrar] Minor adjustment to error message to match previous (and better) behavior.
Fixes this test failure:
1) Failed : Xamarin.Registrar.GenericType_WithInvalidParameterTypes
The error 'MT4136: The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<U>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)'' was not found in the output:
Message #1 did not match:
actual: 'The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<Foundation.NSObject>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)''
expected: 'The registrar cannot marshal the parameter type 'System.Collections.Generic.List`1<U>' of the parameter 'arg' in the method 'Open`1.Bar(System.Collections.Generic.List`1<U>)''
* [docs] mmp shows MM errors/warnings.
* [docs] Improve according to reviews.
* [tests] Fix merge failure causing test duplication.
2018-02-06 09:08:15 +03:00
" blockliteral-setupblock: By default enabled when using the static registrar. Optimizes calls to BlockLiteral.SetupBlock to avoid having to calculate the block signature at runtime.\n" +
2018-02-10 03:15:15 +03:00
" inline-intptr-size: By default enabled for builds that target a single architecture (requires the linker). Inlines calls to IntPtr.Size to load a constant value. Makes the app smaller, and slightly faster at runtime.\n" +
2018-02-12 18:59:39 +03:00
" inline-dynamic-registration-supported: By default always enabled (requires the linker). Optimizes calls to Runtime.DynamicRegistrationSupported to be a constant value. Makes the app smaller, and slightly faster at runtime.\n" +
2018-03-02 16:33:27 +03:00
#if ! MONOTOUCH
2019-03-15 17:05:53 +03:00
" register-protocols: Remove unneeded metadata for protocol support. Makes the app smaller and reduces memory requirements. Disabled when the interpreter is used or when the static registrar is not enabled.\n" +
2019-02-14 20:36:29 +03:00
" trim-architectures: Remove unneeded architectures from bundled native libraries. Makes the app smaller and is required for macOS App Store submissions.\n" +
2018-03-02 16:33:27 +03:00
#else
2019-03-15 17:05:53 +03:00
" register-protocols: Remove unneeded metadata for protocol support. Makes the app smaller and reduces memory requirements. Disabled, by default, to allow dynamic code loading.\n" +
2019-02-14 20:36:29 +03:00
" remove-unsupported-il-for-bitcode: Remove IL that is not supported when compiling to bitcode, and replace with a NotSupportedException.\n" +
2018-03-02 16:33:27 +03:00
#endif
2019-02-14 20:36:29 +03:00
"" ,
[mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp. (#3242)
* [mtouch/mmp] Give users more control over optimizations, and share more code between mtouch and mmp.
1. Add an --optimize flag to mtouch/mmp that allows users to select which
optimizations to apply (or not). This makes it easier to add future
optimizations, and allow users to disable any optimization that causes
problems without having to disable many other features.
2. Share as much optimization code as possible between mtouch and mmp. This
immediately gives a benefit to mmp, which has three new optimizations only
mtouch had: NSObject.IsDirectBinding inlining, IntPtr.Size inlining and
dead code elimination.
This results in ~6kb of disk space saved for a linked Xamarin.Mac app:
* link sdk: [Debug][1], [Release][2]
* link all: [Debug][3], [Release][4]
Testing also verifies that monotouchtest ([Debug][5], [Release][6]) has not
changed size at all, which means that no default optimizations have changed
inadvertedly.
[1]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--debug
[2]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-sdk-mac--release
[3]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--debug
[4]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#link-all-mac--release
[5]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonedebug64
[6]: https://gist.github.com/rolfbjarne/6b731e3b5ca6170355662e6505c3d492#monotouchtest-iphonerelease64
* [tools] Don't enable the IsDirectBinding optimization by default for Xamarin.Mac apps, it's not safe.
* Fix whitespace issues.
* [doc] Document optimizations.
* Officially support optimizations by adding them to the Versions.plist.
* [linker] Improve IntPtr.Size inliner + dead code eliminatior and add tests.
* Properly handle operands for the ldc_i4_s instruction (they're sbyte).
* Fix less-than condition to actually do a less-than comparison.
* Make sure to look up the bitness in the Target, not the Application, since
the Application's value will be incorrect when building fat apps (both
Is32Build and Is64Build will be true).
* Remove unnecessary checks for the IntPtr.Size inliner: this optimization
does not depend on other instructions than the IntPtr.get_Size call, so
remove the checks that verify surrounding instructions. This makes the
IntPtr.Size inliner kick in in more scenarios (such as the new tests).
* Add tests.
* [tests] Add mmp tests for optimizations.
* [tests] Fix XM optimization tests.
* [tests] Fix test build error.
2018-01-23 13:33:48 +03:00
( v ) = > {
app . Optimizations . Parse ( v ) ;
} ) ;
2017-11-16 18:02:34 +03:00
options . Add ( new Mono . Options . ResponseFileSource ( ) ) ;
2017-01-11 23:10:39 +03:00
}
static int Jobs ;
public static int Concurrency {
get {
return Jobs = = 0 ? Environment . ProcessorCount : Jobs ;
}
2016-05-11 13:24:55 +03:00
}
[mtouch/mmp] Print assembly references in verbose mode. (#2139)
* [mtouch/mmp] Print assembly references in verbose mode.
Sample output:
Loaded assembly 'unifiedtestapp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' from /Users/rolf/Projects/TestApp/bin/iPhoneSimulator/Debug/unifiedtestapp.exe
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: Xamarin.iOS, Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065
Loaded assembly 'mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/mscorlib.dll
Loaded assembly 'Xamarin.iOS, Version=0.0.0.0, Culture=neutral, PublicKeyToken=84e04ff9cfb79065' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/../../64bits/Xamarin.iOS.dll
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: System.Core, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: Mono.Security, Version=2.0.5.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756
References: System.Xml, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Loaded assembly 'System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/System.dll
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: Mono.Security, Version=2.0.5.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756
References: System.Xml, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Loaded assembly 'Mono.Security, Version=2.0.5.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/Mono.Security.dll
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Loaded assembly 'System.Xml, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/System.Xml.dll
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Loaded assembly 'System.Core, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e' from /work/maccore/master/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/System.Core.dll
References: mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
References: System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
I believe this will make it easier to diagnose cases where something references a desktop assembly.
* [mtouch/mmp] Share more code between mmp and mtouch.
2017-05-29 17:15:22 +03:00
public static int Verbosity {
get { return verbose ; }
}
2018-01-26 20:38:23 +03:00
public const bool IsXAMCORE_4_0 = false ;
2016-04-21 15:57:02 +03:00
#if MONOMAC
#pragma warning disable 0414
static string userTargetFramework = TargetFramework . Default . ToString ( ) ;
#pragma warning restore 0414
#endif
static TargetFramework ? targetFramework ;
public static bool HasTargetFramework {
get { return targetFramework . HasValue ; }
}
public static TargetFramework TargetFramework {
get { return targetFramework . Value ; }
set { targetFramework = value ; }
}
static void SetTargetFramework ( string fx )
{
#if MONOMAC
userTargetFramework = fx ;
#endif
switch ( fx . Trim ( ) . ToLowerInvariant ( ) ) {
#if MONOMAC
case "xammac" :
case "mobile" :
case "xamarin.mac" :
targetFramework = TargetFramework . Xamarin_Mac_2_0 ;
break ;
#endif
default :
TargetFramework parsedFramework ;
if ( ! Xamarin . Utils . TargetFramework . TryParse ( fx , out parsedFramework ) )
throw ErrorHelper . CreateError ( 68 , "Invalid value for target framework: {0}." , fx ) ;
#if MONOMAC
if ( parsedFramework = = TargetFramework . Net_3_0 | | parsedFramework = = TargetFramework . Net_3_5 )
parsedFramework = TargetFramework . Net_2_0 ;
#endif
targetFramework = parsedFramework ;
break ;
}
#if MTOUCH
if ( Array . IndexOf ( TargetFramework . ValidFrameworks , targetFramework . Value ) = = - 1 )
throw ErrorHelper . CreateError ( 70 , "Invalid target framework: {0}. Valid target frameworks are: {1}." , targetFramework . Value , string . Join ( " " , TargetFramework . ValidFrameworks . Select ( ( v ) = > v . ToString ( ) ) . ToArray ( ) ) ) ;
#endif
}
2019-08-28 14:54:26 +03:00
public static int RunCommand ( string path , string args )
{
return RunCommand ( path , args , null , ( Action < string > ) null ) ;
}
public static int RunCommand ( string path , string args , string [ ] env = null , StringBuilder output = null , bool suppressPrintOnErrors = false )
{
if ( output ! = null )
return RunCommand ( path , args , env , ( v ) = > { if ( v ! = null ) output . AppendLine ( v ) ; } , suppressPrintOnErrors ) ;
return RunCommand ( path , args , env , ( Action < string > ) null , suppressPrintOnErrors ) ;
}
public static int RunCommand ( string path , string args , string [ ] env = null , Action < string > output_received = null , bool suppressPrintOnErrors = false )
2016-04-21 15:57:02 +03:00
{
Exception stdin_exc = null ;
var info = new ProcessStartInfo ( path , args ) ;
info . UseShellExecute = false ;
info . RedirectStandardInput = false ;
info . RedirectStandardOutput = true ;
info . RedirectStandardError = true ;
System . Threading . ManualResetEvent stdout_completed = new System . Threading . ManualResetEvent ( false ) ;
System . Threading . ManualResetEvent stderr_completed = new System . Threading . ManualResetEvent ( false ) ;
2019-08-28 14:54:26 +03:00
var lockobj = new object ( ) ;
StringBuilder output = null ;
if ( output_received = = null ) {
2016-04-21 15:57:02 +03:00
output = new StringBuilder ( ) ;
2019-08-28 14:54:26 +03:00
output_received = ( line ) = > {
if ( line ! = null )
output . AppendLine ( line ) ;
} ;
}
2016-04-21 15:57:02 +03:00
if ( env ! = null ) {
if ( env . Length % 2 ! = 0 )
throw new Exception ( "You passed an environment key without a value" ) ;
for ( int i = 0 ; i < env . Length ; i + = 2 )
info . EnvironmentVariables [ env [ i ] ] = env [ i + 1 ] ;
}
if ( verbose > 0 )
Console . WriteLine ( "{0} {1}" , path , args ) ;
using ( var p = Process . Start ( info ) ) {
p . OutputDataReceived + = ( s , e ) = > {
if ( e . Data ! = null ) {
2019-08-28 14:54:26 +03:00
lock ( lockobj )
output_received ( e . Data ) ;
2016-04-21 15:57:02 +03:00
} else {
stdout_completed . Set ( ) ;
}
} ;
p . ErrorDataReceived + = ( s , e ) = > {
if ( e . Data ! = null ) {
2019-08-28 14:54:26 +03:00
lock ( lockobj )
output_received ( e . Data ) ;
2016-04-21 15:57:02 +03:00
} else {
stderr_completed . Set ( ) ;
}
} ;
p . BeginOutputReadLine ( ) ;
p . BeginErrorReadLine ( ) ;
p . WaitForExit ( ) ;
stderr_completed . WaitOne ( TimeSpan . FromSeconds ( 1 ) ) ;
stdout_completed . WaitOne ( TimeSpan . FromSeconds ( 1 ) ) ;
2019-08-28 14:54:26 +03:00
output_received ( null ) ;
2016-04-21 15:57:02 +03:00
if ( p . ExitCode ! = 0 ) {
// note: this repeat the failing command line. However we can't avoid this since we're often
// running commands in parallel (so the last one printed might not be the one failing)
2019-08-28 14:54:26 +03:00
if ( ! suppressPrintOnErrors ) {
// We re-use the stringbuilder so that we avoid duplicating the amount of required memory,
// while only calling Console.WriteLine once to make it less probable that other threads
// also write to the Console, confusing the output.
if ( output = = null )
output = new StringBuilder ( ) ;
output . Insert ( 0 , $"Process exited with code {p.ExitCode}, command:\n{path}\n" ) ;
Console . Error . WriteLine ( output ) ;
}
2016-04-21 15:57:02 +03:00
return p . ExitCode ;
2019-08-28 14:54:26 +03:00
} else if ( verbose > 0 & & output ! = null & & output . Length > 0 & & ! suppressPrintOnErrors ) {
2016-04-21 15:57:02 +03:00
Console . WriteLine ( output . ToString ( ) ) ;
}
if ( stdin_exc ! = null )
throw stdin_exc ;
}
return 0 ;
}
[mtouch] Rework how tasks are built.
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
2017-01-26 12:56:55 +03:00
public static Task < int > RunCommandAsync ( string path , string args , string [ ] env = null , StringBuilder output = null , bool suppressPrintOnErrors = false )
{
2019-08-28 14:54:26 +03:00
if ( output ! = null )
return RunCommandAsync ( path , args , env , ( v ) = > { if ( v ! = null ) output . AppendLine ( v ) ; } , suppressPrintOnErrors ) ;
return RunCommandAsync ( path , args , env , ( Action < string > ) null , suppressPrintOnErrors ) ;
}
public static Task < int > RunCommandAsync ( string path , string args , string [ ] env = null , Action < string > output_received = null , bool suppressPrintOnErrors = false )
{
return Task . Run ( ( ) = > RunCommand ( path , args , env , output_received , suppressPrintOnErrors ) ) ;
[mtouch] Rework how tasks are built.
The previous build system kept a forward-pointing single linked list of tasks
to execute: task X had a list of subsequent tasks to execute. If task X was
up-to-date, it was not created (and the next tasks were directly added to the
list of tasks to execute).
In this world it became complicated to merge output from tasks (for instance
if the output of task X and task Y should be a consumed by a single task
producing a single output, since the corresponding task would end up in both
X's and Y's list of subsequent tasks).
Example: creating a single framework from the aot-compiled output of multiple
assemblies.
So I've reversed the logic: now we keep track of the final output, and then
each task has a list of dependencies that must be built.
This makes it trivial to create merging tasks (for the previous example, there
could for instance be a CreateFrameworkTask, where its dependencies would be
all the corresponding AotTasks).
We also always create every task, and then each task decides when its executed
whether it should do anything or not. This makes it unnecessary to 'forward-
delete' files when creating tasks (say you have three tasks, A, B, C; B
depends on A, and C depends on B; if A's output isn't up-to-date, it has to
delete its own output if it exists, otherwise B would not detect that it would
have to re-execute, because at task *creation* time, B's input hadn't
changed).
Additionally make it based on async/await, since much of the work happens in
externel processes (and we don't need to spin up additional threads just to
run external processes). This makes us have less code run on background
threads, which makes any issues with thread-safety less likely.
2017-01-26 12:56:55 +03:00
}
2016-04-21 15:57:02 +03:00
#if ! MMP_TEST
static void FileMove ( string source , string target )
{
Application . TryDelete ( target ) ;
File . Move ( source , target ) ;
}
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
static void MoveIfDifferent ( string path , string tmp , bool use_stamp = false )
2016-05-11 14:00:20 +03:00
{
// Don't read the entire file into memory, it can be quite big in certain cases.
bool move = false ;
using ( var fs1 = new FileStream ( path , FileMode . Open , FileAccess . Read ) ) {
using ( var fs2 = new FileStream ( tmp , FileMode . Open , FileAccess . Read ) ) {
if ( fs1 . Length ! = fs2 . Length ) {
Log ( 3 , "New file '{0}' has different length, writing new file." , path ) ;
move = true ;
} else {
move = ! Cache . CompareStreams ( fs1 , fs2 ) ;
}
}
}
if ( move ) {
FileMove ( tmp , path ) ;
} else {
Log ( 3 , "Target {0} is up-to-date." , path ) ;
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
if ( use_stamp )
Driver . Touch ( path + ".stamp" ) ;
2016-05-11 14:00:20 +03:00
}
}
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
public static void WriteIfDifferent ( string path , string contents , bool use_stamp = false )
2016-04-21 15:57:02 +03:00
{
var tmp = path + ".tmp" ;
try {
if ( ! File . Exists ( path ) ) {
2016-11-21 15:30:46 +03:00
Directory . CreateDirectory ( Path . GetDirectoryName ( path ) ) ;
2016-04-21 15:57:02 +03:00
File . WriteAllText ( path , contents ) ;
Log ( 3 , "File '{0}' does not exist, creating it." , path ) ;
return ;
}
File . WriteAllText ( tmp , contents ) ;
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
MoveIfDifferent ( path , tmp , use_stamp ) ;
2016-05-11 14:00:20 +03:00
} catch ( Exception e ) {
File . WriteAllText ( path , contents ) ;
ErrorHelper . Warning ( 1014 , e , "Failed to re-use cached version of '{0}': {1}." , path , e . Message ) ;
} finally {
Application . TryDelete ( tmp ) ;
}
}
2016-04-21 15:57:02 +03:00
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
public static void WriteIfDifferent ( string path , byte [ ] contents , bool use_stamp = false )
2016-05-11 14:00:20 +03:00
{
var tmp = path + ".tmp" ;
2016-04-21 15:57:02 +03:00
2016-05-11 14:00:20 +03:00
try {
if ( ! File . Exists ( path ) ) {
File . WriteAllBytes ( path , contents ) ;
Log ( 3 , "File '{0}' does not exist, creating it." , path ) ;
return ;
2016-04-21 15:57:02 +03:00
}
2016-05-11 14:00:20 +03:00
File . WriteAllBytes ( tmp , contents ) ;
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
MoveIfDifferent ( path , tmp , use_stamp ) ;
2016-04-21 15:57:02 +03:00
} catch ( Exception e ) {
2016-05-11 14:00:20 +03:00
File . WriteAllBytes ( path , contents ) ;
2016-04-21 15:57:02 +03:00
ErrorHelper . Warning ( 1014 , e , "Failed to re-use cached version of '{0}': {1}." , path , e . Message ) ;
} finally {
Application . TryDelete ( tmp ) ;
}
}
#endif
internal static string GetFullPath ( )
{
return System . Reflection . Assembly . GetExecutingAssembly ( ) . Location ;
}
2018-11-05 10:16:29 +03:00
static string xcode_product_version ;
public static string XcodeProductVersion {
2018-06-09 04:45:24 +03:00
get {
2018-11-05 10:16:29 +03:00
return xcode_product_version ;
2018-06-09 04:45:24 +03:00
}
}
2016-04-21 15:57:02 +03:00
static Version xcode_version ;
public static Version XcodeVersion {
get {
return xcode_version ;
}
}
2017-08-22 10:40:26 +03:00
static void SetCurrentLanguage ( )
{
// There's no way to change the current culture from the command-line
// without changing the system settings, so honor LANG if set.
// This eases testing mtouch/mmp with different locales significantly,
// and won't run into issues where changing the system language leaves
// the tester with an incomprehensible system.
var lang_variable = Environment . GetEnvironmentVariable ( "LANG" ) ;
if ( string . IsNullOrEmpty ( lang_variable ) )
return ;
// Mimic how mono transforms LANG into a culture name:
// https://github.com/mono/mono/blob/fc6e8a27fc55319141ceb29fbb7b5c63a9030b5e/mono/metadata/locales.c#L568-L576
var lang = lang_variable ;
var idx = lang . IndexOf ( '.' ) ;
if ( idx > = 0 )
lang = lang . Substring ( 0 , idx ) ;
idx = lang . IndexOf ( '@' ) ;
if ( idx > = 0 )
lang = lang . Substring ( 0 , idx ) ;
lang = lang . Replace ( '_' , '-' ) ;
try {
var culture = CultureInfo . GetCultureInfo ( lang ) ;
if ( culture ! = null ) {
CultureInfo . DefaultThreadCurrentCulture = culture ;
2017-09-18 18:14:17 +03:00
Log ( 2 , $"The current language was set to '{culture.DisplayName}' according to the LANG environment variable (LANG={lang_variable})." ) ;
2017-08-22 10:40:26 +03:00
}
} catch ( Exception e ) {
ErrorHelper . Warning ( 124 , e , $"Could not set the current language to '{lang}' (according to LANG={lang_variable}): {e.Message}" ) ;
}
}
2017-11-16 18:02:34 +03:00
static void LogArguments ( string [ ] arguments )
{
2018-04-14 23:13:14 +03:00
if ( Verbosity < 1 )
2017-11-16 18:02:34 +03:00
return ;
if ( ! arguments . Any ( ( v ) = > v . Length > 0 & & v [ 0 ] = = '@' ) )
return ; // no need to print arguments unless we get response files
LogArguments ( arguments , 1 ) ;
}
static void LogArguments ( string [ ] arguments , int indentation )
{
Log ( "Provided arguments:" ) ;
var indent = new string ( ' ' , indentation * 4 ) ;
foreach ( var arg in arguments ) {
Log ( indent + StringUtils . Quote ( arg ) ) ;
if ( arg . Length > 0 & & arg [ 0 ] = = '@' ) {
var fn = arg . Substring ( 1 ) ;
LogArguments ( File . ReadAllLines ( fn ) , indentation + 1 ) ;
}
}
}
[mtouch/mmp] Fix tracking of whether the static registrar should run again or not. Fixes #641. (#3534) (#3536)
* [tests] Improve debug spew for the RebuildTest_WithExtensions test.
* [mtouch/mmp] Store/load if the dynamic registrar is removed or not into the cached link results.
Store/load if the dynamic registrar is removed or not into the cached link
results, so that we generate the correct main.m even if cached linker results
are used.
* [mtouch/mmp] The static registrar must not execute if we're loading cached results from the linker.
The static registrar must not execute if we're loading cached results from the
linker, because the static registrar needs information from the linker that's
not restored from the cache.
* [mtouch/mmp] Share Touch code.
* [mtouch/mmp] Make it possible to touch inexistent files (to create them).
* [mtouch/mmp] Fix tracking of whether the static registrar should run again or not.
The recent changes to support optimizing away the dynamic registrar caused the
Xamarin.MTouch.RebuildTest_WithExtensions test to regress.
The problem
-----------
* The linker now collects and stores information the static registrar needs.
* This information is not restored from disk when the linker realizes that it
can reload previously linked assemblies instead of executing again.
* The static registrar runs again (for another reason).
* The information the static registrar needs isn't available, and incorrect
output follows.
So fix 1: show an error if the static registrar runs when the linker loaded
cached results.
The exact scenario the test ran into is this:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk (this is an optimization to avoid
compiling the registrar.m file again unless needed).
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is newer than
registrar.m's timestamp and run again, but doesn't produce the right result
because it doesn't have the information it needs.
Considered solutions
--------------------
1. Only track timestamps, not file contents. This is not ideal, since it will
result in more work done: in particular for the case above, it would add a
registrar.m compilation in build #2, and linker rerun + static registrar
rerun + registrar.m compilation + final native link in build #3.
2. Always write the output of the static registrar, even if it hasn't changed.
This is not ideal either, since it will also result in more work done: for
the case above, it would add a registrar.m compilation + final native link
in build #3.
3. Always write the output of the static registrar, but track if it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it.
This only means the final native link in build #3 is added (see #5 for why
this is worse than it sounds).
4. Always write the output of the static registrar, but track it it changed or
not, and if it didn't, just touch registrar.o instead of recompiling it,
and track that too, so that the final native link in build #3 isn't needed
anymore. Unfortunately this may result in incorrect behavior, because now
the msbuild tasks will detect that the executable has changed, and may run
dsymutil + strip again. The executable didn't actually change, which means
it would be the previously stripped executable, and thus we'd end up with
an empty .dSYM because we ran dsymtil on an already stripped executable.
5. Idea #4, but write the output of the final link into a temporary directory
instead of the .app, so that we could track whether we should update the
executable in the .app or not. This is not optimal either, because
executables can be *big* (I've seen multi-GB tvOS bitcode executables), and
extra copies of such files should not be taken lightly.
6. Idea #4, but tell the MSBuild tasks that dsymutil/strip doesn't need to be
rerun even if the timestamp of the executable changed. This might actually
work, but now the solution's become quite complex.
Implemented solution
--------------------
Use stamp files to detect whether a file is up-to-date or not.
In particular:
* When we don't write to a file because the new contents are identical to the
old contents, we now touch a .stamp file. This stamp file means "the
accompanying file was determined to be up-to-date when the stamp was
touched."
* When checking whether a file is up-to-date, also check for the presence of a
.stamp file, and if it exists, use the highest timestamp between the stamp
file and the actual file.
Now the test scenario becomes:
* 1st build: everything is new and everything is built.
* 2nd build: contents of .exe changes, the linker runs again, the static
registrar runs again, but sees that the generated output didn't change, so
it doesn't write the new content to disk, but it creates a registrar.m.stamp
file to indicate the point in time when registrar.m was considered up-to-
date.
* 3rd build: only the .exe timestamp changes, the linker sees nothing changes
in the contents of the .exe and loads the previously linked assemblies from
disk, the static registrar sees that the .exe's timestamp is *older* than
registrar.m.stamp's timestamp and doesn't run again.
We only use the stamp file for source code (registrar.[m|h], main.[m|h],
pinvokes.[m|h]), since using it every time has too much potential for running
into other problems (for instance we should never create .stamp files inside
the .app).
Fixes these test failures:
1) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single","",False,System.String[])
single
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory371/testApp.app/testApp is modified, timestamp: 2/15/2018 3:04:11 PM > 2/15/2018 3:04:09 PM" >
2) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("dual","armv7,arm64",False,System.String[])
dual
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory375/testApp.app/testApp is modified, timestamp: 2/15/2018 3:06:03 PM > 2/15/2018 3:06:00 PM" >
3) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("llvm","armv7+llvm",False,System.String[])
llvm
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory379/testApp.app/testApp is modified, timestamp: 2/15/2018 3:07:14 PM > 2/15/2018 3:07:12 PM" >
4) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("debug","",True,System.String[])
debug
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory383/testApp.app/testApp is modified, timestamp: 2/15/2018 3:08:16 PM > 2/15/2018 3:08:13 PM" >
5) Failed : Xamarin.MTouch.RebuildTest_WithExtensions("single-framework","",False,System.String[])
single-framework
Expected: <empty>
But was: < "/Users/builder/data/lanes/5746/4123bf7e/source/xamarin-macios/tests/mtouch/bin/Debug/tmp-test-dir/Xamarin.Tests.BundlerTool.CreateTemporaryDirectory387/testApp.app/testApp is modified, timestamp: 2/15/2018 3:09:18 PM > 2/15/2018 3:09:16 PM" >
Fixes https://github.com/xamarin/maccore/issues/641
2018-02-20 13:43:23 +03:00
public static void Touch ( IEnumerable < string > filenames , DateTime ? timestamp = null )
{
if ( timestamp = = null )
timestamp = DateTime . Now ;
foreach ( var filename in filenames ) {
try {
var fi = new FileInfo ( filename ) ;
if ( ! fi . Exists ) {
using ( var fo = fi . OpenWrite ( ) ) {
// Create an empty file.
}
}
fi . LastWriteTime = timestamp . Value ;
} catch ( Exception e ) {
ErrorHelper . Warning ( 128 , "Could not touch the file '{0}': {1}" , filename , e . Message ) ;
}
}
}
public static void Touch ( params string [ ] filenames )
{
Touch ( ( IEnumerable < string > ) filenames ) ;
}
2018-05-29 20:08:34 +03:00
static int watch_level ;
static Stopwatch watch ;
public static int WatchLevel {
get { return watch_level ; }
set {
watch_level = value ;
if ( ( watch_level > 0 ) & & ( watch = = null ) ) {
watch = new Stopwatch ( ) ;
watch . Start ( ) ;
}
}
}
public static void Watch ( string msg , int level )
{
if ( ( watch = = null ) | | ( level > WatchLevel ) )
return ;
for ( int i = 0 ; i < level ; i + + )
Console . Write ( "!" ) ;
Console . WriteLine ( "Timestamp {0}: {1} ms" , msg , watch . ElapsedMilliseconds ) ;
}
2018-11-05 10:16:29 +03:00
internal static PDictionary FromPList ( string name )
{
if ( ! File . Exists ( name ) )
throw ErrorHelper . CreateError ( 24 , "Could not find required file '{0}'." , name ) ;
return PDictionary . FromFile ( name ) ;
}
const string XcodeDefault = "/Applications/Xcode.app" ;
static string FindSystemXcode ( )
{
var output = new StringBuilder ( ) ;
if ( Driver . RunCommand ( "xcode-select" , "-p" , output : output ) ! = 0 ) {
ErrorHelper . Warning ( 59 , "Could not find the currently selected Xcode on the system: {0}" , output . ToString ( ) ) ;
return null ;
}
return output . ToString ( ) . Trim ( ) ;
}
static string sdk_root ;
static string developer_directory ;
public static string DeveloperDirectory {
get {
return developer_directory ;
}
}
static void ValidateXcode ( bool accept_any_xcode_version , bool warn_if_not_found )
{
if ( sdk_root = = null ) {
sdk_root = FindSystemXcode ( ) ;
if ( sdk_root = = null ) {
// FindSystemXcode showed a warning in this case. In particular do not use 'string.IsNullOrEmpty' here,
// because FindSystemXcode may return an empty string (with no warning printed) if the xcode-select command
// succeeds, but returns nothing.
sdk_root = null ;
} else if ( ! Directory . Exists ( sdk_root ) ) {
ErrorHelper . Warning ( 60 , "Could not find the currently selected Xcode on the system. 'xcode-select --print-path' returned '{0}', but that directory does not exist." , sdk_root ) ;
sdk_root = null ;
} else {
if ( ! accept_any_xcode_version )
ErrorHelper . Warning ( 61 , "No Xcode.app specified (using --sdkroot), using the system Xcode as reported by 'xcode-select --print-path': {0}" , sdk_root ) ;
}
if ( sdk_root = = null ) {
sdk_root = XcodeDefault ;
if ( ! Directory . Exists ( sdk_root ) ) {
if ( warn_if_not_found ) {
// mmp: and now we give up, but don't throw like mtouch, because we don't want to change behavior (this sometimes worked it appears)
ErrorHelper . Warning ( 56 , "Cannot find Xcode in any of our default locations. Please install Xcode, or pass a custom path using --sdkroot=<path>." ) ;
return ; // Can't validate the version below if we can't even find Xcode...
}
throw ErrorHelper . CreateError ( 56 , "Cannot find Xcode in the default location (/Applications/Xcode.app). Please install Xcode, or pass a custom path using --sdkroot <path>." ) ;
}
ErrorHelper . Warning ( 62 , "No Xcode.app specified (using --sdkroot or 'xcode-select --print-path'), using the default Xcode instead: {0}" , sdk_root ) ;
}
} else if ( ! Directory . Exists ( sdk_root ) ) {
throw ErrorHelper . CreateError ( 55 , "The Xcode path '{0}' does not exist." , sdk_root ) ;
}
// Check what kind of path we got
if ( File . Exists ( Path . Combine ( sdk_root , "Contents" , "MacOS" , "Xcode" ) ) ) {
// path to the Xcode.app
developer_directory = Path . Combine ( sdk_root , "Contents" , "Developer" ) ;
} else if ( File . Exists ( Path . Combine ( sdk_root , ".." , "MacOS" , "Xcode" ) ) ) {
// path to Contents/Developer
developer_directory = Path . GetFullPath ( Path . Combine ( sdk_root , ".." , ".." , "Contents" , "Developer" ) ) ;
} else {
throw ErrorHelper . CreateError ( 57 , "Cannot determine the path to Xcode.app from the sdk root '{0}'. Please specify the full path to the Xcode.app bundle." , sdk_root ) ;
}
var plist_path = Path . Combine ( Path . GetDirectoryName ( DeveloperDirectory ) , "version.plist" ) ;
if ( File . Exists ( plist_path ) ) {
var plist = FromPList ( plist_path ) ;
var version = plist . GetString ( "CFBundleShortVersionString" ) ;
xcode_version = new Version ( version ) ;
xcode_product_version = plist . GetString ( "ProductBuildVersion" ) ;
} else {
throw ErrorHelper . CreateError ( 58 , "The Xcode.app '{0}' is invalid (the file '{1}' does not exist)." , Path . GetDirectoryName ( Path . GetDirectoryName ( DeveloperDirectory ) ) , plist_path ) ;
}
if ( ! accept_any_xcode_version ) {
if ( min_xcode_version ! = null & & XcodeVersion < min_xcode_version )
throw ErrorHelper . CreateError ( 51 , "{3} {0} requires Xcode {4} or later. The current Xcode version (found in {2}) is {1}." , Constants . Version , XcodeVersion . ToString ( ) , sdk_root , PRODUCT , min_xcode_version ) ;
if ( XcodeVersion < SdkVersions . XcodeVersion )
2018-11-06 17:06:10 +03:00
ErrorHelper . Warning ( 79 , "The recommended Xcode version for {4} {0} is Xcode {3} or later. The current Xcode version (found in {2}) is {1}." , Constants . Version , XcodeVersion . ToString ( ) , sdk_root , SdkVersions . Xcode , PRODUCT ) ;
2018-11-05 10:16:29 +03:00
}
Driver . Log ( 1 , "Using Xcode {0} ({2}) found in {1}" , XcodeVersion , sdk_root , XcodeProductVersion ) ;
}
2016-04-21 15:57:02 +03:00
}
}