Use mono's `external/linker` submodule reference instead of including
our own, separate, `linker` submodule reference.
This way we are more consistent with xamarin-macios, which also uses
mono's linker submodule.
There is some code which is conceptually, if not literally, duplicated
between the xamarin-android and the IDEs (Visual Studio,
Visual Studio for Mac), in particular code that deals with finding the
Android SDK and Java JDK locations.
This code has been split out into a new `xamarin-android-tools` repo
to facilitate sharing, without the IDEs needing to submodule
xamarin-android, which is quite large.
Cleanup the xamarin-android repo to use the new xamarin-android-tools
repo for Android SDK information/etc.
The [most recent xamarin-android/master build is not green][xa564],
because of a failure in the `api-compatibility` tests:
[xa564]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/564/
ABI BREAK IN: mscorlib.dll
<!-- start namespace System.Collections.Generic --> <div>
<h2>Namespace System.Collections.Generic</h2>
<h3>Removed Type <span class='breaking' data-is-breaking>System.Collections.Generic.CollectionExtensions</span></h3>
</div> <!-- end namespace System.Collections.Generic -->
The cause for the breakage is that
[xamarin-andorid-api-compatibility was updated for mono/2017-06][0],
and `mscorlib.dll` is incompatible between mono/2017-04 and
mono/2017-06.
[0]: ed8d914d08
Since Xamarin.Android d15-4 and current master are both using
mono/2017-04, the result is the above "ABI BREAK" message...
*all the time*.
Update xamarin-android/master to use
[xamarin-android-api-compatibility/d15-4][1], fixing the ABI BREAK
build error.
[1]: https://github.com/xamarin/xamarin-android-api-compatibility/tree/d15-4
Developers like API compatibility; it helps ensure that their
investment in our platform isn't a waste of time.
Add `tests/api-compatibility` and a new
`make run-api-compatibility-tests` target, part of
`make run-all-tests`, which compares the API of the assemblies in the
current build tree against a "known good" API description maintained
in `tests/api-compatibility/reference/*.xml.gz`, one `.xml.gz` file
per assembly that we are interested in.
The new [`xamarin-android-api-compatibility`][0] submodule contains
the current API description and Make targets to check and update the
API description. `xamarin-android-api-compatibility` uses the
utilities `mono-api-info` and `mono-api-html` to generate and compare
the API descriptions.
A submodule is used because the API descriptions can be quite large --
Mono.Android.xml is currently 54MB -- and we don't want the change
history on these files to "clog up" the xamarin-android repo.
[0]: https://github.com/xamarin/xamarin-android-api-compatibility
**Background**: many Android SDK features rely on [ProGuard][0], a
Java bytecode manipulation library. Some of these features include
linking (shrinking) bytecode and [multidex][1].
Xamarin.Android has historically relied on the version of ProGuard
bundled with the Android SDK, which is how the `<Proguard/>` task and
`@(ProguardConfiguration)` build actions work.
Unfortunately, the version of ProGuard bundled with the Android SDK
is, at this point, *ancient* (ProGuard v4.7; current version is v5.3),
and (more importantly) ProGuard v4.7 doesn't support JDK 1.8 bytecode.
Meaning if the end-developer references a Java library targeting
JDK 1.8 -- e.g. the Android Support Library v25 -- then the version of
ProGuard bundled with the Android SDK *cannot process that library*.
This in turn means that the `<Proguard/>` task and
`@(ProguardConfiguration)` build actions don't work/are useless.
Native Java development "works around" this problem by using Gradle as
the build system, and Java projects are configured to download and
install a newer version of ProGuard via Gradle.
Xamarin.Android projects do not use Gradle, nor will they.
(At least not in the foreseeable future.)
**Solution**: We have communicated with Eric Lafortune, maintainer of
ProGuard, about distributing a newer ProGuard with Xamarin.Android.
He replied:
> From a licensing point of view, you shouldn't need an exception to
> the GPL in many practical configurations:
>
> * if you indeed run ProGuard in a separate JVM
The `<Proguard/>` task is a [`ToolTask`][2], which operates by
creating a new process to invoke the specified tool. We thus are
*already* using ProGuard within a "separate JVM", and no architectural
change is required to comply with ProGuard's licensing without
altering Xamarin.Android's licensing.
The current ProGuard sources (v5.3.2) have been imported into the
[xamarin/proguard][3] module, and we will be using the
`xamarin/proguard` module as a git submodule for build purposes.
[0]: https://www.guardsquare.com/en/proguard
[1]: https://developer.android.com/studio/build/multidex.html
[2]: https://msdn.microsoft.com/en-us/library/microsoft.build.utilities.tooltask.aspx
[3]: http://github.com/xamarin/proguard/
Mono 4.8 has new SGEN bridge performance characteristics and
semantics, requiring that we bump the SGEN Bridge Version to 5.
Additionally, Mono 4.8 is using Cecil/master, which is
API INCOMPATIBLE with Cecil 0.9.6, and Mono sources are included in
`Xamarin.Android.Build.Tasks.dll`.
We could attempt to address the Cecil incompatibility by building
Mono's `external/cecil` projects, but that doesn't work because
Java.Interop assemblies *also* use Cecil, and
`Xamarin.Android.Build.Tasks.dll` uses `Java.Interop.Tools.Cecil.dll`,
which uses Cecil via NuGet package, so everything gets very intermixed
and confusing if Java.Interop tries to use e.g. Cecil 0.9.6 via nuget
while xamarin-android attempts to use Cecil via Mono's source.
Skip that problem, and bump to Java.Interop/a1d3ecc8, which in turn
uses the Cecil 0.10.0-beta1-v2 *Preview* NuGet package, which is
API compoatible with Cecil/master and mono/master.
Additionally, update xamarin-android's Cecil 0.9.6 use to likewise use
Cecil 0.10.0-beta1-v2.
This allows xamarin-android to build while using Mono 4.8 sources.
Finally, there's lots of "unrelated noise" in this patch, largely
because every time I add a project to the solution via Xamarin Studio,
Xamarin Studio decides to modify EVERY PROJECT IN THE REPO. Preserve
some of these changes -- and "fix" others -- so that I can reduce the
"visual noise" of future Xamarin Studio changes.
The commit implements building of LLVM and cross-compilers to support
Xamarin.Android/Mono AOT. LLVM and cross-compilers can be built for
both the host platform (Linux and OS/X at the moment) as well as
cross-compiled for 32-bit and 64-bit Windows platforms.
Windows builds are done with MXE toolchain on OS/X and with the packaged
mingw-w64 toolchain on Linux (tested on Ubuntu 16.04 ONLY).
Also introducing a new set of MSBuild properties that contain information
about the host system. Some of those properties (HostOS, HostCC, HostCXX
for instance) have been moved from Configuration.props to better support
auto-detection. A new script, build-tools/scripts/generate-os-info, is
invoked as part of `make prepare` to generate file that contains the
new properties. The generated file is required for the build to work and
is also host-specific (it mustn't be moved between different machines)
Cross compiler builds require access to a configured Mono build tree, in
order to generate C structure offsets header file that is used by the AOT
compilers to properly generate AOT-ed binaries. Therefore, even if a JIT
target is not enabled in the configuration, enabling a cross-compiler for
some target will configure Mono for that JIT target but it will NOT build
it, to save time. To facilitate this, the _MonoRuntimes items defined in
build-tools/mono-runtimes/mono-runtimes.projitems gain an additional metadata
item called `DoBuild` which will be set to `true` if the runtime actually needs
to be built, as opposed to just configured.
MXE builds are disabled on Linux as mingw-w64 works just fine.
A `make prepare` warning is issued for Linux hosts which have the binfmt_misc
module enabled and either Wine of Mono (cli) registered as PE32/PE32+ binary
interpreters. In such instance building of the Windows cross-compilers will
fail because Autotools determine whether software is being cross compiled by
building a test program and attempting to execute it. In normal circumstances
such an attempt will fail, but with Windows cross-compilation and either Wine
or Mono registered to handle the PE32 executables this attempt will succeed
thus causing the cross compilation detection to fail.
Currently to build cross compilers on Linux you need to generate the C structure
offsets header file on OS/X and copy the resulting headers to appropriate places
on Linux. The header files should be placed in
build-tools/mono-runtimes/obj/Debug/cross-*/
directories. The header files are:
{cross-arm,cross-arm-win}/aarch64-v8a-linux-android.h
{cross-arm64,cross-arm64-win}/armv5-none-linux-androideabi.h
{cross-x86,cross-x86-win}/i686-none-linux-android.h
{cross-x86_64,cross-x86_64-win}/x86_64-none-linux-android.h
Offsets header generation doesn't work on Linux atm because of missing support
for it in the Mono utility used to generate the offsets. Hopefully this limitation
will be removed in the near future and a start-to-end build of everything will be
possible on Linux.
It is now mandatory to run at least `make prepare-props` before Xamarin.Android
can be built. The target generates the OS-specific props file which is required
by the build. `make prepare` depends on the target.
Commit 4ec06ac9 *broke* generation of `.apk` files when migrating
from `Ionic.Zip.dll` to `System.IO.Compression` because this:
apk.AddFile (assembly.ItemSpec, "assemblies", compressionLevel: CompressionLevel.NoCompression);
doesn't work the way we thought/hoped it would.
Specifically, we require that assemblies be *stored*, uncompressed,
within the `.apk`, as we **mmap**(2) the `.apk` and hand off the
memory addresses of the loaded assemblies to mono for execution.
`CompressionLevel.NoCompression`, despite saying "no compression",
does *not* mean "store". It means "add the file 'normally', but tell
`DeflateStream` to use 'no compression'."
The result is that the zip entry doesn't say it's stored:
$ unzip -lv bin/Debug/*-Signed.apk | grep assemblies/
64512 Defl:N 64512 67% 06-16-16 10:38 6d65706b assemblies/Scratch.DebugRelease.dll
Which in turn means that the resulting `.apk` is *unusable*.
The fix? `System.IO.Compression` can't be used for this, and
`Ionic.Zip.dll` had other problems (why we tried to use
`System.IO.Compression` in the first place!), so instead we'll
use [libZip][0] and [LibZipSharp][1] to handle `.apk` files.
libzip is a well-maintained OSS library for manipulating zip files,
and LibZipSharp is a C# wrapper around libzip.
Add libzip and LibZipSharp to the build system so that they're
available on required paltforms, and update the `BuildApk` and
related tasks to use LibZipSharp instead of System.IO.Compression.
This allows assemblies to be properly stored in the `.apk`,
allowing apps to execute as intended.
$ unzip -lv bin/Debug/*-Signed.apk | grep assembl
64512 Stored 64512 0% 06-29-16 08:31 6d65706b assemblies/Scratch.DebugRelease.dll
[0]: http://www.nih.at/libzip/
[1]: https://github.com/grendello/LibZipSharp/
OpenTK sources are built using our own project file in order to leave
the upstream sources untouched. Only OpenTK 1.0 is provided in hope that
nobody is interested in 0.9 anymore :) For this reason compilation items
are placed in a separate project file. It also makes it easier to share
the "wrapper" project file for different versions of the library.
A new configuration property is added, $(OpenTKSourceDirectory), which,
surprisingly, points to the OpenTK submodule directory.
Certain Xamarin.Android features require that Mono be built for
Windows, e.g. the [AOT compilers][aot] require a build of mono that
executes on Windows to generate the AOT native libraries.
Unfortunately, building Mono on Windows continues to be a massive
PITA. (Autotools on Windows requires Cygwin/mingw, running shell
scripts on Windows is painfully slow, it's all brittle, etc.)
To work around this pain, we instead build the Mono/Windows binaries
on OS X, via [MXE][mxe], which produces a gcc-based cross-compiler
which generates Windows binaries and is executable from Unix.
This in turn requires that we have MXE, so add a
`_CreateMxeToolchains` target to `android-toolchain.targets` which
will build MXE. The installation prefix for MXE can be overridden via
the new `$(AndroidMxeInstallPrefix)` MSBuild property; it defaults to
`$HOME/android-toolchain/mxe`.
Rework the `$(AndroidSupportedAbis)` MSBuild property so that it
must include the "host" ABI, and add support for a new `host-win64`
value which will use MXE to generate 64-bit Windows binaries for
libmonosgen-2.0.dll and libMonoPosixHelper.dll.
We can't always process `host-$(HostOS)` because of an xbuild bug.
The scenario is that if you want to just build `host-win64`, the
obvious thing to do is:
cd build-tools/mono-runtimes
xbuild /p:AndroidSupportedAbis=host-win64
Alas, if `host-$(HostOS)` is always processed, this inexplicably
causes `host-$(HostOS)` to be re-rebuilt, which (1) is a waste of
time, and (2) fails -- inexplicably -- in the `_BuildRuntimes` target
because make(1) thinks that the configure flags have somehow changed,
which currently makes no sense at all. (When can we move to MSBuild?)
Changing `$(AndroidSupportedAbis)` so that `host-$(HostOS)` is
explicitly processed instead of implicitly processed allows working
around the above xbuild bug, as `host-$(HostOS)` won't be implicitly
processed on every build, but only when required.
Additionally, we add a new <Which/> MSBuild task so that we can
determine if a particular program is in `$PATH`. This is useful
because listing requirements within README.md is a road to pain --
e.g. xxd(1) is required to build `src/monodroid` but if it's missing
it'll still *build* but you'll instead get a *linker* failure because
the `monodroid_config` and `monodroid_machine_config` symbols aren't
present. Building MXE requires that even more programs be present
within $PATH, so explicitly check for these so that *useful* error
messages can be generated instead of obscure ones.
Finally, a note about autotools and generating Windows native
libraries: creation of `.dll` files *requires* that an appropriate
objdump be present so it can determine if e.g. `libkernel32.a` is an
import library or an archive. If `x86_64-w64-mingw32.static-objdump`
isn't found -- e.g. because $PATH doesn't contain it -- then no `.dll`
files will be created, and much head scratching will occur.
To rectify this, override the OBJDUMP and DLLTOOL values when invoking
`configure` so that that full paths are used and `$PATH` use is
reduced. (Who wants `x86_64-w64-mingw32.static-objdump` in `$PATH`?)
[aot]: https://developer.xamarin.com/releases/android/xamarin.android_5/xamarin.android_5.1/#AOT_Support
[mxe]: http://mxe.cc/
Relative URLs for the submodules will make it harder to work with
various versions of Mono and Java.Interop checked out in parallel
with the proprietary parts of Xamarin.Android as both parties
would use the same location for the "master" repository but different
hashes or branches. This would lead to frequent rebuilding of both
submodules as developers switch between the OSS and proprietary
repositories.
Additionally, using a relative URL like previously means that we
have no information where the submodule was checked out from. Should
the repository referred to in the submodule URL be removed from
filesystem, user might have a problem recreating it (the same applies
to cloning the xamarin-android repository in which case to get the
submodules initialized one has to first check out Mono and Java.Interop
in the directory above xamarin-android - but there's no information
where to check them out from)
Furthermore, updating the submodule would require going first to
the "master" repository above xamarin-android, pulling the changes
there, going back to xamarin-android to update/init submodules.
HTTPs URLs fix the original issue of requiring that the users are
logged in to GitHub in order to clone xamarin-android and it avoids all
the pitfalls outlined above.
(*Now* things are starting to come together...)
One of the important infrastructural pieces of Xamarin.Android is
Mono.Android.dll, a "binding" for the Android SDK `android.jar` file.
Bindings are generated by using the `generator` tool from
Java.Interop, along with tools/jnienv-gen, and various other programs.
This in turn requires adding a git submodule for Java.Interop.
Stitch all these pieces together so that we can take an
*API description* (stored in Profiles\api-*.xml.in) and generate a
binding for that API description.
The generated files are located in
obj\$(Configuration)\android-$(ApiLevel), and the resulting
Mono.Android.dll is copied to
$(topdir)\bin\$(Configuration)\xbuild-frameworks\MonoAndroid\$(XAFrameworkVersion).
One SNAFU: currently, Mono.Android.csproj conditinally <Import/>s a
Mono.Android.projitems file generated by Java.Interop's generator
tool, which contains a list of all the generated files.
When the project is first loaded, Mono.Android.projitems will not
exist, so on that initial build, source code will be generated but
xbuild won't re-read Mono.Android.projitems (once it exists). This
will result in a failing build.
Simply rebuild the project to get a valid build, or use the `make all`
Makefile target.
Add a new build-tools/mono-runtimes project to build the required Mono
runtimes for Xamarin.Android app execution on Android devices.
Xamarin.Android 6.0 ("cycle 6") includes Mono runtimes compiled for
five architectures:
* armeabi (armv5)
* armeabi-v7a (armv7)
* arm64-v8a (aarch64)
* x86
* x86_64
An additional "host" architecture is also built to build the
"monodroid" profile assemblies such as mscorlib.dll.
In the interest of expediency and not requiring that *six* different
runtimes be built -- which can greatly increase build times and
storage requirements -- the initial xamarin-android open-source build
will only build the armeabi-v7a (armv7) and "host" architectures.
Additionally, the <UnzipDirectoryChildren/> task needed to be altered
to use unzip(1) on OS X instead of using
System.IO.Compression.ZipFile, because the ZipFile implementation
included with Mono 4.4 doesn't respect file attributes such as the +x,
causing all files to be extracted as 644 (rw-r--r--), resulting
in an NDK toolchain which wouldn't work. To resolve this issue,
execution on OS X and Linux now uses unzip(1).
~~ Adding new architectures ~~
To build Mono for a new architecture, edit
build-tools/mono-runtimes/mono-runtimes.projitems and add a new
@(_MonoRuntime) entry for the desired architecture.
TODO: Figure out if there's a reasonable way to "opt-in" or "opt-out"
of CPU architectures so that it isn't an "all or nothing" prospect.
Radek Doulik
Alexander Köplinger
Jonathan Pryor
Marek Habersack
Ludovic Henry
Rodrigo Kumpera
Andi McClure
Dean Ellis
Duncan Mak