If the build fails we get a generic message to look at the
`build.log`. This is a pain because the build log is HUGE! and takes
ages to find the problem.
It seems NUnit does not include the `FailedBuildException.BuildLog`
property. Override the `Exception.StackTrace` property to include the
`FailedBuildException.BuildLog` data.
On Windows after formatting my machine (don't ask),
`PrepareWindows.targets` was failing with:
System.InvalidOperationException: Could not determine Android SDK location.
Looking into it, there appear to be two things wrong:
* `PrepareWindows.targets` should import `Configuration.props`
* The incoming properties are `$(AndroidSdkDirectory)` and
`$(AndroidNdkDirectory)` not `$(AndroidSdkPath)` and
`$(AndroidNdkPath)`.
I'm not quite sure how this *ever* worked, unless `AndroidSdkInfo`
was able to find *some* Android SDK on your machine, e.g. from a
prior Xamarin install.
I have been told that the `prepare-image-dependencies.sh` script
should also install `mono`, to ensure that the required mono exists.
Add a `@PKG_URLS@` variable to `prepare-image-dependencies.sh.in` to
support downloading and installing our minimum specified mono version.
Additionally, I discovered that `xbuild` bugs are still around; I just
keep forgetting their specifics. Case in point:
[Jenkins build #712][xa712] uploaded
[`prepare-image-dependencies.sh`][sh], and it's not good:
[xa712]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/
[sh]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/712/Azure/processDownloadRequest/xamarin-android/prepare-image-dependencies.shhttps://dl.google.com/android/repository/build-tools_r26.0.1-macosx.zip sdk/build-tools\$(XABuildToolsFolder)
...
https://dl.google.com/android/repository/android-15_r03.zip sdk/platforms\android-15
MSBuild properties contained within item metadata -- specifically
`%(AndroidSdkItem.DestDir)` -- aren't fully expanded, resulting in the
above value of `$(XABuildToolsFolder)`, which *should* be `26.0.1`.
Similarly, some install paths are using `\` (backslash) when they
should contain `/` (forward slash).
Jenkins has been updated to explicitly use `msbuild`:
$ make prepare-image-dependencies MSBUILD=msbuild
However, `\r` characters now appear in
`prepare-image-dependencies.sh`. Fix this by having the
`PrepareImageDependencies` target emit the MSBuild-generated file into
an intermediate file, then fix it up using **tr**(1).
Finally, correct the `brew install` command so that any errors are
ignored, for if a brew package is already installed, re-installing it
will result in an error.
Context: https://github.com/xamarin/xamarin-android/blob/master/tools/scripts/xabuild#L105
`xabuild`, the shell script, has some logic to set MSBuild properties
for `AndroidSdkDirectory` and `AndroidNdkDirectory`. The issue here is
on Windows, `xabuild.exe` is not invoked from this shell script. So we
need to replicate the logic in C# in `xabuild.exe`, but not break what
was fixed in #977.
Changes:
* Check for `ANDROID_SDK_PATH` or `ANDROID_NDK_PATH` environment
variables
* Run `Paths.targets` via a new MSBuild process, the same way as
`tools/scripts/xabuild`
* Set the `AndroidSdkDirectory` and `AndroidNdkDirectory`
properties in the config file if they are not blank
* `AndroidSdkDirectory` and `AndroidNdkDirectory` can still be
overridden by a user, because incoming values supersede what is
found in the config file
The `BuildAppCheckDebugSymbols()` test does not take into account
that when building in debug mode on monodroid the assemblies
are not embedded in the apk. As a result when trying to get
`Library1.pdb` from the apk we get a `null` back. This causes
a `NullReferenceException` when we try to get a length.
The fix is to look in the `assets` directory if we can't get
the file from the `.apk`. This will allow the test to work in
both xamarin-android and monodroid.
We are investigating building Xamarin.Android atop
Visual Studio Team System (VSTS) in addition to our current Jenkins
setup, and one of the issues we're running into is "bootstrapping":
VSTS is configured to create a "fresh" VM for each build.
**Pro**: It should allow builds to be more reliable, as previous build
artifacts won't be present, and thus won't cause/hide errors.
**Con**: *Previous build artifacts are not present*. Previous build
artifacts such as *downloading and extracting* the Android NDK & SDK,
using `brew` to install dependencies, building MXE...
Ensuring that the dependencies are installed through `make prepare`
can be quite time consuming. What we want is a way to ensure that the
"build image" -- *what's already installed* when the VM boots --
contains all of our desired dependencies.
Furthermore, we *don't* want to have the responsible parties checkout
and build xamarin-android in order to determine what the dependencies
should be.
Attempt to square this circle by adding a new
`make prepare-image-dependencies` target, which processes
`@(AndroidSdkItem)`, `@(AndroidNdkItem)`, `@(AntItem)`, and
`@(RequiredProgram)` to create a `prepare-image-dependencies.sh`
script which will download and install the required dependencies.
The generated shell script does *not* take the state of the machine
running `make prepare-image-dependencies` into consideration. This
allows the target to be executed on one machine, and the output run
on another.
$ make prepare-image-dependencies
# creates `prepare-image-dependencies.sh`
*Note*: `make prepare-image-dependencies` does not currently deal
with MXE. (Building MXE on the VSTS VM is *very* time consuming, so
it's something we need to take care of. It is not *yet* dealt with.)
Fixes: https://devdiv.visualstudio.com/DevDiv/XamarinVS/_queries?id=512752%2F&triage=true&_a=edit
The DesignTime build is generally run before any nuget packages
have been restored. As a result we end up with empty cache files.
Because those files exist and are considered uptodate, when the
main build finally happens they are not re-generated.
This causes errors like
error: Error: No resource found that matches the given name: attr 'colorAccent'.
on a new default template.
The DesignTime build should have its own set of cache files which
do not conflict with the main build. This way when the main build
runs the cache files will not be in place.
FixesL https://bugzilla.xamarin.com/show_bug.cgi?id=60080
In a previous attempt to fix an intellisense build (1cd582ec) we
caused this issue. The DesignTime build does NOT like our AsyncTask
as it locks the IDE.
This commit puts the old code back in place which skips the task
if we are in DesignTime mode. But so we can easily test it, we
emit an message that our test can pick up to ensure we are doing
the correct thing. Without the message we have no way of knowing
if the task is skipped.
Commit 8643ded9 tried to make sure `xbuild` worked like `msbuild`
when it came to `'$(DebugType)' == 'None'`. The "fix" however
was not correct.
When `$(DebugType)` is `None` under `msbuild`, it changes to
`portable` and `$(DebugSymbols)` gets set to `true`. Additionally,
when `$(DebugType)` is *empty* it gets set to `portable` but then
`$(DebugSymbols)` is left alone.
So we need to get `xbuild` to do the same. Otherwise we get the
wrong runtimes. But by the time we get to
`Xamarin.Android.Common.targets` it is too late to change the
`$(DebugSymbols)` value: `xbuild` does not use the changes.
So we need to do this before `xbuild` gets a chance to set the
value of `$(DebugSymbols)`.
The trick is to alter the properties in
`Xamarin.Android.CSharp.targets` and
`Xamarin.Android.FSharp.targets` *before* we `<Import/>`
`Microsoft.*.targets`.
On jenkins is appears we are getting a situation where we
are picking up an empty directory for the FrameworkPath.
As a result we always get 0 when trying to pick up the
runtime that are available. So tests which check the runtimes
fail.
This commit adds a check for a 'libmono-android.release.so' file
so can ensure the required files exist in the folder. Also added
code to pick up the `CONFIGURATION` environment variable from
the Makefile.
The `dl.` CDN has more backends than `dl-ssl.` and is generally faster.
Also, change ant download to use HTTPS (in preparation for the world where plain
http is gone :P)
Context: https://github.com/nunit/nunit/releases/tag/3.7.1
Context: commit 6687dac5
The release notes for NUnit 3.7.1 mention a hang with parallelized
tests, so it is worth a try to see if updating fixes it the hangs
that we experienced and tried to workaround in 6687dac.
Attempt to quasi-"optimize" `Builder.BuildInternal()`: if we're going
to read `buildLogFullPath` a line at a time *anyway* -- to collect
the log file for later parsing into `Builder.LastBuildOutput` -- then
we should also do any associated line-oriented parsing as well.
Update `buildLogFullPath` processing to use a `StringBuilder` instead
of string concatenation, and search for the `Time Elapsed` message
while reading the file, instead of separately, *after* having read
the entire build log into memory.
Commit 8643ded9 added support for running the tests in `msbuild`.
Part of that change was altering the behaviour of `xbuild` to
match that of `msbuild`, regarding the way they handle
`$(DebugType) == "None"`.
The commit altered `Xamarin.Android.Common.targets` to reset
`$(DebugType)` to `portable` if `None` is detected. It also resets
`$(DebugSymbols)` to `false` in the same senario.
As a result the code these some tests which do the same thing
should not longer be required. Remove this redundant code.
The introduction of `xabuild` in commit f9d15dd2 omitted a design
requirement: `xabuild` needs to be runnable on a machine which does
*does not* have a xamarin-android build environment. The idea is that
e.g. Windows developers could download a Jenkins-built
`oss-xamarin.android*.zip` file, extract it, and run
`bin\Debug\bin\xabuild.exe` to use the extract artifacts *without*
requiring system-wide installation via `setup-windows.exe`.
This not-very-well-stated requirement brought along two mistakes:
1. `xabuild.exe` should *not* set the `$(AndroidSdkDirectory)` and
`$(AndroidNdkDirectory)` MSBuild properties. It *can't*; it could
be executed on ~any machine, in particular one which hasn't built
xamarin-android, so setting these properties to "build-tree"
values doesn't make sense and will fail.
2. Even if (1) weren't the case, commit f9d15dd2 still allowed the
`Xamarin.Android.Build.Tests` unit tests to use `xbuild` as the
MSBuild engine, via the `tools/scripts/xabuild` script.
This script *does not* always set `$(AndroidSdkDirectory)` or
`$(AndroidNdkDirectory)`.
As a result of (2), many unit tests *fail*:
Error executing task ResolveSdks: System.InvalidOperationException: Could not determine Android SDK location. Please provide `androidSdkPath`.
at Xamarin.Android.Tools.AndroidSdkInfo..ctor (System.Action`2[T1,T2] logger, System.String androidSdkPath, System.String androidNdkPath, System.String javaSdkPath) [0x0008e] in …/xamarin-android/external/xamarin-android-tools/src/Xamarin.Android.Tools.AndroidSdk/AndroidSdkInfo.cs:28
at Xamarin.Android.Tasks.MonoAndroidHelper.RefreshAndroidSdk (System.String sdkPath, System.String ndkPath, System.String javaPath) [0x00021] in …/xamarin-android/src/Xamarin.Android.Build.Tasks/Utilities/MonoAndroidHelper.cs:111
at Xamarin.Android.Tasks.ResolveSdks.RunTask () [0x001a0] in …/xamarin-android/src/Xamarin.Android.Build.Tasks/Tasks/ResolveSdksTask.cs:157
at Xamarin.Android.Tasks.ResolveSdks.Execute () [0x00002] in …/xamarin-android/src/Xamarin.Android.Build.Tasks/Tasks/ResolveSdksTask.cs:130
at Microsoft.Build.BuildEngine.TaskEngine.Execute () [0x00000] in …/mcs/class/Microsoft.Build.Engine/Microsoft.Build.BuildEngine/TaskEngine.cs:134
at Microsoft.Build.BuildEngine.BuildTask.Execute () [0x0008d] in …/mcs/class/Microsoft.Build.Engine/Microsoft.Build.BuildEngine/BuildTask.cs:101
Update `xabuild` to *never* explicitly set `$(AndroidSdkDirectory)`
and `$(AndroidNdkDirectory)`, and update `Xamarin.Android.Build.Tests`
to *always* explicitly provide these MSBuild properties. This allows
unit test execution to be more reliable, and will prevent
`xabuild.exe` from looking at "random" non-existent directories.
The `Xamarin.Android.Build.Tests` NUnit tests recently started
hanging after the tests were done.
Run tests with a single worker as a workaround until we figure out
why, unblocking Jenkins.
There are cases in which diagnostic MSBuild output is not including
important information such as `$(MSBuildExtensionsPath)`.
To help debug situations like this, if `MSBuildApp.Main` returns a
non-zero exit code, we can print the contents of xabuild.exe's config
file to the console.
Commit 286b9c28 added extra item to `@(_FilesToRegister)` as part of
the `IncrementalClean` target.
Unfortunately, one of the values omitted the `$`, resulting in an
"invalid" property name reference.
This commit fixes that issue.
Context: 8643ded9
Context: 286b9c28
Context: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/700/testReport/
`Xamarin.ProjectTools` was updated in commit 8643ded9 to use `msbuild`
(via `xabuild`) *by default*, and commit 286b9c28 updated
`Xamarin.ProjectTools` to further use `msbuild`/`xabuild` when the
`$USE_MSBUILD` environment variable is *not* `0`, and to continue
using `xbuild` when `$USE_MSBUILD` is `0`.
Meaning before 8643ded9 `xbuild` was used for
`Xamarin.Android.Build.Tests` execution, and afterward `msbuild`
was used. (This change wasn't explicitly clear to some.)
"Revert" this behavior; use `xbuild` *by default* for
`Xamarin.Android.Build.Tests` execution, by updating the
`RUN_NUNIT_TEST` make `define` so that the `$USE_MSBUILD`
environment variable is exported to the *value of* the
`$(USE_MSBUILD)` make variable, IFF it's not the empty string.
If `$(USE_MSBUILD)` *is* the empty string, export `USE_MSBUILD=0`,
which will cause `xbuild` to be used.
What his means is that a default unit test invocation:
make run-all-tests
will use `xbuild` to run `Xamarin.Android.Build.Tests`. However,
if `msbuild` is *explicitly opted into*, then `msbuild` is used:
$ make run-all-tests MSBUILD=msbuild
# -or-
$ make run-all-tests MSBUILD=some/path/to/msbuild USE_MSBUILD=1
Furthermore, if/when we change the default MSBuild engine to
`msbuild` in the future -- by altering
`build-tools/scripts/msbuild.mk` to set `MSBUILD=msbuild` and
`USE_MSBUILD=1` by default -- then `make run-all-tests` will
*implicitly* follow suit.
With luck, "reverting" to `xbuild` use will allow the
[current `Xamarin.Android.Build.Tests` failures][xa700t] -- all 20 of
them -- to be "fixed" until such time as all
`Xamarin.Android.Build.Tests` tests *actually* pass with `msbuild`.
[xa700t]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/700/testReport/
The test failed locally for me with:
```
…/xamarin-android/bin/Debug/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets:
error : '…/android-sdk-macosx/build-tools/23.0.0/apksigner' does not exist.
You need to install android-sdk build-tools 26.0.1 or above.
```
which is weird since it's looking into the 23.0.0 build tools which
doesn't have apksigner even though the test specifies build tools 26.0.1.
Turns out the test had a typo, it used `AndroidBuildToolsVersion`
instead of `AndroidSdkBuildToolsVersion` so it fell back to 23.0.0 :)
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=60185
Newer versions of NDK don't include API 4 anymore but two of our tests specified
that they required that particular version to build. This caused the tests to
fail because the android-4 platform path was nowhere to be found.
Bump the minimum SDK version to 9 (which is the lowest platform included in the NDK
as of now)
Sometimes out TimeOut tests fail if the build is a bit slow.
This is because we time the entire process of starting a new
process to bringing in ALL the output data.
So lets parse the build.log and pull our the reported MSBuild
"Time Elapsed" if we can.
There was a spelling mistake in the DesignTime build timing test.
Also it was reporting the wrong time! So this commit fixes that.
It also makes the test `NonParallelizable`, this is because when
its run in parallel with a bunch of other tests the timing is usually
off and the test fails.
There are circumstances when we don't want to use the latest
and greatest version of mono. On the CI servers we will want
to keep to certain versions so that we have a stable build.
But on developer machines we will want to allow the developer
to ignore such checks.
This commit adds support for `$(MonoRequiredMaximumVersion).
By default this proeprty is ignored. But on build systems we should pass:
/p:IgnoreMaxMonoVersion=False
to ensure that the max version is applied and we run against
a stable version of mono.
We need to output the Lint tool Path we use and the Lint Tool
version for diagnostic purposes. So if it does fail we have
the information we need to correct the issue.
Commit 8643ded reworked the msbuild tests to use msbuild. But
there was a problem with the logic.
string.IsNullOrEmpty (useMSBuild) || useMSBuild == "0"
well the Environment variable USE_MSBUILD will ALWAYS be empty.
Because we never set it. so the logic should be
!string.IsNullOrEmpty (useMSBuild) && useMSBuild == "0"
so that if it is set it has to be "0" to allow xbuild to be
used.
Also one of the tests was using `RunningMSBuild` before it was
calculated. So it was always false. So just added some logging to
print what we are using to the console. This forces `RunningMSBuild`
to be calculated.
One final issue was with the intermediate .pdb files being cleaned
up by `IncrementalClean`. Turns out we had a fix for the .mdb files
in `Xamarin.Android.Windows.targets`. So the solution there was to
add the same work around for .pdb files.
When working with the `Makefile` I noticed this command doesn’t run the
tests:
```
$ make run-nunit-tests
make: Nothing to be done for `run-nunit-tests’.
```
And this command runs the tests:
```
$ make run-nunit-tests SKIP_NUNIT_TESTS=1
```
I suspect the intention is that `ifneq` needs to be `ifeq`.
Do not use `%(TestApk.ResultsPath)` as base filename for performance
measurements anymore; it was changed recently by 385699a5. Instead,
pass own results filename to the `<ProcessLogcatTiming/>` and
`<ProcessPlotInput/>` tasks.
Also add a `ProcessLogcatTiming.LabelSuffix` parameter to the task,
so that it is easier to create merged measurements. It comes handy as
more tests are run in multiple configurations now. It also simplifies
the use of definitions files as we don't need one per configuration.
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=59193
Java does not include types to represent unsigned integers. This poses a problem
when either porting code from Java to managed languages or when attempting to
cast integer values between the Java and the managed land.
The issue described in the above bug could be fixed by adding appropriate
implicit and explicit operators to Java.Lang.Object in Xamarin.Android but that
would allow for behavior which may have adverse effects without any external
signs immediately visible to the developer.
Consider a situation when a minimum signed 32-bit integer value returned by Java
code is cast to the managed `uint` type - we end up with the same value but with
different sign and no indication given that such a thing happened. We could
up-cast the value to long but that changes the type of the result and is not
advisable, especially with implicit conversions. Also, even if the value was
up-cast to a type with a larger value range this would have to stop with the
64-bit integers since they can't be up-cast to any other primitive integer type.
Any casts between signed and unsigned integer types should be a conscious and
explicit action, thus the double cast `(ulong)(long)value` is considered the
correct behavior.
For those reasons we decided that the best action to to take is to actively
prevent direct casts from/to a managed unsigned integer type to/from a signed
Java integer type. This is implemented by way of adding a number of explicit and
implicit conversion operators to XA's Java.Lang.Object implementation that are
marked "obsolete" and being erroneous. This is done this way so that the code
attempting to perform such conversions won't build because the compiler, seeing
the attribute, will signal an error and abort the build.
In order to get the tests running using xabuild on monodroid
we kinda need it in the output directory. This commit adds
it as Content to the xabuild.csproj. It also updaes the Builder
to look for `xabuild` in the output directory first and then
fallback to `tools\scripts`.
This commit reworks the tests to pass on MSBuild. xbuild is
deprecated, so we should ensure that our system works on MSBuild.
Lots of changes. Highlights include:
We have to NOT use `$(BuildingInsideVisualStudio)` when building
under `msbuild`. Setting this property to `true` has a side effect
of telling `msbuild` "Dont bother to build @(ProjectReferences)".
This is because under VS that is something the IDE takes care of.
It turns out that the bug we had to do with `__NonExistantFile__`
causing the `csc` task to run ALL the time was fixed.
All of the `IsTargetSkipped` checks have been unified to call the
`BuildOutput.IsTargetSkipped()` method. It has also been updated
to include more target "skipped" formats which xbuild and MSBuild
generate.
There is also a difference in behaviour between xbuild and MSBuild.
You may well have noticed that ALL the tests that have had the
"debug" -> "release"
changes (including the debugger attribute one) are around
$(DebugType) == "None"
Under xbuild when `$(DebugType)` == "None" and
`$(DebugSymbols)` == "True", we end up with
`$(DebugType)` == "None" and `$(DebugSymbols)` == "True" (no changes).
This results in a debug runtime and apk.
However, under MSBuild when `$(DebugType)` == "None"
and `$(DebugSymbols)` == "True" we end up with
`$(DebugType)` == "portable" and `$(DebugSymbols)` == "False" (!).
This results in a release runtime and apk.
This is because `None` is NOT a valid value for `$(DebugType)` in
MSBuild, so it resets BOTH `$(DebugType)` and `$(DebugSymbols)`.
This results in difference of behaviour between the two systems.
Update `Xamarin.Android.Common.targets` to check for this, and update
`$(DebugType)` and `$(DebugSymbols)` so that xbuild behaves like
MSBuild. This improves consistency (and our own sanity!).
(This may break some QA tests, but this is now expected and the tests
should be updated to reflect this.)
Context: https://stackoverflow.com/questions/17709873/how-can-i-invoke-my-msbuild-target-when-msbuild-exe-starts-and-when-it-ends
There is a convention where an MSBuild file named
`Before.<SolutionName>.sln.targets` will get imported prior to a SLN
file getting built. This enables us to import `PrepareWindows.targets`
and enable `msbuild Xamarin.Android.sln /t:Prepare` to work
appropriately.
macOS/Linux remain unchanged, since `PrepareWindows.targets`
currently only works for Windows.
Add commit SHA1 to `$(AndroidMxeInstallPrefix)`, so that we can
revert mxe to an earlier version.
Most of the current "do not rebuild DEPENDENCY" logic is
"forward-only": it only works properly if you go *forward* in time,
not backwards in time.
For example, consider the `ForceBuild` target in
`build-tools/mono-runtimes/mono-runtimes.targets`: if the HEAD commit
within `external/mono` moves "backward" so that the resulting
timestamp of `autogen.sh` is *older* than e.g.
`bin\$(Configuration)\bcl-tests\bcl-tests.zip`, re-running the
`ForceBuild` target won't do anything.
`external/mono` and `mono-runtimes` are forward-only.
Usually this is considered acceptable. If it's truly a problem, nuke
the `bin` directory and rebuild.
"Forward-only" behavior is *not* always acceptable.
Case in point: `external/mxe`. Building MXE takes a significant
amount of time -- roughly 10+ minutes -- so we *cache* the MXE
toolchain into `$(AndroidMxeInstallPrefix)` -- which defaulted to
`$HOME/android-toolchain/mxe` -- and only rebuild the cached MXE
when `external/mxe` changes.
All well and good...except when MXE *cannot* be built *at all*.
Case in point: [MXE couldn't be built with Xcode 8.3][mxe1744].
We didn't notice this for a long period of time, because our Jenkins
machines are fairly stable; they built and cached MXE under
Xcode 8.1, and never had to rebuild MXE again.
[mxe1744]: https://github.com/mxe/mxe/issues/1744
The only reason we noticed is because some developers tried to build
MXE on new machines, with newer Xcode versions, and MXE didn't build.
This raises an interesting conundrum: for *cached* resources such
as MXE, it is *extremely* important to be able to *revert* to a
"known good" version. "Forward only" behavior is at odds with this;
you *can't* revert to a "known good" version, because it may have
been replaced!
Bump to mxe/xamarin/a926b16d, as this version *can* be built under
Xcode 9. (If we're lucky, it will *also* be able to build an updated
commit of `external/libzip`…)
Additionally, alter the semantics of `$(AndroidMxeInstallPrefix)` to
*require* that it *ends with* the abbreviated commit hash of the
`external/mxe` commit; in this commit, `$(AndroidMxeInstallPrefix)`
is `$HOME/android-toolchain/mxe-a926b16d`.
In particular, this allows us to *revert*: bumping `external/mxe`
will build MXE into a new, *unique*, directory, which means that any
previously built directory will not be modified. This in turn allows
us to "revert" the patch to use the previous MXE, if necessary.
The updated MXE now includes a few more "native" plugins in
`plugins/native/$(OS_SHORT_NAME)`. This caused issues on macOS where
the `patch` that MXE built ran into a problem because it relied on
newer APIs not available in older versions of macOS.
Fix this by disabling building the native plugins and instead rely on
the system ones by overriding `$(OS_SHORT_NAME)`.
[This mirrors what MXE is doing on Travis][travis].
[travis]: 16acc77a3d/.travis.yml (L6)
This commit fixes a couple of issues which appeared when
using `xabuild.exe` on macOS. Firstly if the symlink exists
already the SymbolicLink code will error out and fail the
build completely. This happens because if you run two or
more `xabuild.exe` instances at the same time they trip over
each other trying to create the symlink.
So rather than error'ing completely we should only error if
the link does NOT exist. If it does exist after an attempted
creation we should ignore the exception.
The other is about where we look for extensions. Xamarin.Android
is installed on macOS into
/Library/Frameworks/Mono.framework/External/xbuild
this was not included in the search path for MSBuild, so it never
manages to find the required `.targets` files.
JDK 9 reports a single-digit version value:
$ javac -version
javac 9
Unfortunately, this means that the `<Which/>` task won't properly
recognize JDK 9's `javac` as passing the
`%(RequiredProgram.MinimumVersion)` value of 1.8, because `<Which/>`
requires *at least* two digits, not one. (Because `System.Version`
required at least two digits; one would throw `ArgumentException`.)
Update the `<Which/>` task so that it supports single-digit version
values, treating them as if they had a "minor" value of `0`.
@clancey was trying to build xamarin-android, which failed:
$ make prepare
./build-tools/scripts/generate-os-info Configuration.OperatingSystem.props
xbuild /p:Configuration=Debug /p:_DebugFileExt=.pdb build-tools/dependencies/dependencies.mdproj
...
warning : Referenced Project ../../external/xamarin-android-tools/src/Xamarin.Android.Tools.AndroidSdk/Xamarin.Android.Tools.AndroidSdk.csproj not found, ignoring.
...
Xamarin.Android.BuildTools.PrepTasks/JdkInfo.cs(9,23): error CS0234: The type or namespace name 'Tools' does not exist in the namespace 'Xamarin.Android' (are you missing an assembly reference?)
The problem is that `make prepare-deps`, which builds
`build-tools/dependencies/dependencies.mdproj`, implicitly builds
`build-tools/xa-prep-tasks/xa-prep-tasks.csproj`, which in turn
depends on
`external/xamarin-android-tools/src/Xamarin.Android.Tools.AndroidSdk/Xamarin.Android.Tools.AndroidSdk.csproj`,
which is only reliably created by `make prepare-external`s
`git submodule update --init --recursive`.
Unfortunately, `make prepare-external` wasn't invoked until *after*
`make prepare-deps`, meaning there is no way for `make prepare` to
actually work on a clean install!
(This works on Jenkins because Jenkins automatically checks out all
submodules, so it never encounters this "clean checkout" scenario.)
Reorder the targets that `make prepare` executes so that
`make prepare-external` is run *before* `make prepare-deps`, ensuring
that git submodules exist before we attempt to use them.
`lint` 26.1.1 added a new MissingSuperCall check to `lint`. This
errors out if a class does NOT call `super.onCreate()`. Because
our generated code doesn't need to call `super.onCreate()`,
as the expectation is that the *managed subclass* will instead
invoke `base.OnCreate()`.
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=57914
This commit adds support for the `apksigner` tool which
ships with android. It is enabled by a new property
$(AndroidUseApkSigner)
When set to true the old `jarsigner` code will be bypassed.
The new system examines the manifest file to figure out the
min and max supported versions of android. This is then passed
to the `apksigner` along with the other information (like keys etc)
and allows `apksigner` to figure out what signing algorithm to use.
Additional parameters can be passed to the tool by the developer
via the `$(AndroidApkSignerAdditionalArguments)` property.
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=22074
Context: https://bugzilla.xamarin.com/show_bug.cgi?id=59293
Context: https://developer.xamarin.com/releases/android/xamarin.android_6/xamarin.android_6.1/#Improved_Java_Interface_Versioning_Support
**Background**:
Until [C# gains support for default interface methods][ifaces], there
is an "impedance mismatch" between Java interfaces and C# interfaces:
Java methods can be added to interfaces *without* requiring that
existing classes implementing those interfaces implement the "new"
methods. (*This is not* Java default interface methods!)
[ifaces]: https://github.com/dotnet/csharplang/blob/master/proposals/default-interface-methods.md
Consider [`android.database.Cursor`][cursor]:
[cursor]: https://developer.android.com/reference/android/database/Cursor.html
```java
// API-1
interface Cursor {
void close();
// ...
}
class MyCursor implements Cursor {
public void close() {}
}
```
The `Cursor` interface had several methods added to it over time --
*before* the Android SDK supported default interface methods! --
e.g. the addition of `Cursor.getType()` in API-11:
```java
interface Cursor {
// in API-1
void close();
// ...
// Added in API-11
int getType(int columnIndex);
}
```
Question: What happens when `MyCursor.getType()` is invoked, if/when
`MyCursor` *has not* been recompiled?
```java
Cursor c = new MyCursor(); // Implements API-1 API, *not* API-11!
c.getType(0); // Method added in API-11!
```
What happens is that an [`AbstractMethodError`][ame] is thrown.
[ame]: https://developer.android.com/reference/java/lang/AbstractMethodError.html
Compare to C#, in which *the type cannot be loaded* if an interface
has a method added:
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: VTable setup of type MyCursor failed
What this meant is [Bug #22074][b22074]: if you have a
Xamarin.Android library, and that library:
1. Has `$(TargetFrameworkVersion)`=v2.3 (API-10), and
2. Has a type which implements `Android.Database.ICursor`
and you then use the type (2) in an App which has
`$(TargetFrameworkVersion)`=v4.4, the app would crash with a
`TypeLoadException`.
[b22074]: https://bugzilla.xamarin.com/show_bug.cgi?id=22074
The fix, [introduced in Xamarin.Android 6.1][xa6.1], is the
`FixAbstractMethodsStep` step, which looks at every type in every
assembly to be included in the `.apk`, and checks to see if that type
is "missing" any abstract methods. If any such type is found, then
the "missing" method is *inserted*, and it will throw an
`AbstractMethodError`.
[xa6.1]: https://developer.xamarin.com/releases/android/xamarin.android_6/xamarin.android_6.1/#Improved_Java_Interface_Versioning_Support
For example, if an assembly `Lib.dll` built against API-10 has:
```csharp
public class MyCursor : Java.Lang.Object, Android.Database.ICursor {
// Implement `ICursor` methods from API-10
}
```
then when `Lib.dll` is packaged as part of an API-11+ app the
`FixAbstractMethodsStep` step will *add* the following method:
```csharp
int ICursor.GetType(int columnIndex)
{
throw new Java.Lang.AbstractMethodError();
}
```
**The Problem**: The Xamarin.Forms team ran into an issue with
[the debugger in Visual Studio][b59293]. During investigation it
turned out that our `FixAbstractMethodsStep` injects two methods
which were not needed, as the methods were already implemented.
[b59293]: https://bugzilla.xamarin.com/show_bug.cgi?id=59293
**The Fix**: Improve "missing abstract method" detection by looking
in `MethodDefinition.Overrides` to find the interface method and
compare it to the interface method we are looking for, not just
using the method name. This will be more resilient to compiler
changes -- not all compilers emit the same name for explicitly
implemented interface methods -- and prevent us from emitting
explicit interface methods when they're already present.
Context: 0077d151
Context: d1d9820a
A "funny" thing happened when commit e9daf5ea didn't build on Jenkins:
I realized that not all tests were run in all configurations. From
commit d1d9820a:
> Why are some tests Debug-only and some aren't?
The answer: time, primarily. Why run tests multiple times, when they
can be potentially time-consuming?
While tests can be slow, they're not always *that* slow -- except for
`Xamarin.Android.Build.Tests` and the BCL tests -- and even there,
program behavior can alter between Debug and Release configurations.
See in particular commit 0077d151, in which the BCL tests are run only
in the Debug configuration because tests *failed* when run in the
Release configuration.
The desire, then, is to run *all* tests in both Debug and Release
configurations. Yes, it'll take longer! So what! (Within reason:
`Xamarin.Android.Build.Tests` should only be run once!)
However, this raises two problems:
1. Filename collisions
2. Jenkins unit test display
Until now, all tests wrote files into a filename that didn't include
the Configuration, e.g. `TestResult-Mono.Android_Tests.xml`. If we did
run these tests twice, the second test invocation would overwrite the
first test invocation. This isn't desirable.
Then there's the display on Jenkins: if we did have e.g.
`TestResult-Mono.Android_Tests-Debug.xml` and
`TestResult-Mono.Android_Tests-Release.xml`, how will Jenkins display
that information? I haven't tested, but I would assume that one of two
things will occur, assuming reasonable Jenkins behavior:
1. Each test will be listed twice, e.g.
ApplicationContextIsApp
ApplicationContextIsApp
2. They'll be "merged" into a single entry.
Neither of these behaviors is desirable: if Debug passes but Release
fails, we need to be able to differentiate between them. Neither of
these possible renderings allows us to tell which configuration fails.
Solve both of these problems by introducing a new `<RenameTestCases/>`
task. This task takes three values of importance:
```xml
<RenameTestCases
Configuration="CONFIGURATION"
SourceFile="SOURCE"
DestinationFolder="DESTINATION"
/>
```
The `<RenameTestCases/>` task will read in `SOURCE`, and if `SOURCE`
is an XML file which we determine is NUnit2-formatted XML (root
element of `<test-case/>`), we will update every `//test-case/@name`
value so that it ends with ` / CONFIGURATION`. The updated XML is
then written to the `DESTINATION` directory, with a filename that
contains `CONFIGURATION`, and `SOURCE` is deleted.
Thus, if we have a Debug-configuration
`TestResult-Mono.Android_Tests.xml` file with XML fragment:
```xml
<test-case
name="Mono.Android_Tests, Android.AppTests.ApplicationTest.ApplicationContextIsApp"
...
/>
```
then `<RenameTestCases/>` will create the file
`TestResult-Mono.Android_Tests-Debug.xml` file with XML fragment:
```xml
<test-case
name="Mono.Android_Tests, Android.AppTests.ApplicationTest.ApplicationContextIsApp / Debug"
...
/>
```
This allows us to run tests in both Debug and Release configurations
while not inadvertently overwriting the `TestResults*.xml` files that
Jenkins reads, and ensuring that the Jenkins test result output is
rendered in a meaningfully useful fashion.
Aside: when updating `//test-case/@name`, the resulting value *cannot*
end in `)`. If it does, then the `(root)` package name issue fixed in
commit 23b2642e reappears for the `generator` unit tests.
**Completely random aside about the state of `xbuild`**:
A development version of `<RenameTestCases/>` was "saner", using
`ITaskItem[]` and not string:
```csharp
partial class RenameTestCases {
public ITaskItem[] SourceFiles {get; set;}
// vs.
// public string SourceFile {get; set;}
}
```
The problem is that the above, while entirely reasonable, did not work
at all correctly with `xbuild`:
```xml
<RenameTestCases SourceFiles="%(TestApk.ResultsPath)" />
```
Under `xbuild`, MSBuild properties would not be expanded, e.g.
`RenameTestCases.SourceFiles` would get a "bizarro" value of e.g.
`$(OutputPath)Mono.Android_Tests-Signed.apk`, which is *useless* and
would result in `FileNotFoundException`s.
MSBuild proper, of course, worked as desired.
TODO: Once this is merged, update the Jenkins Configuration page so
that instead of:
make run-all-tests V=1 || exit 1
it instead runs both Debug and Release configuration tests:
make run-all-tests SKIP_NUNIT_TESTS=1 V=1 || exit 1
make run-all-tests CONFIGURATION=Release V=1 || exit 1
Note that `$(SKIP_NUNIT_TESTS)` is specified so that we only run the
lengthy (1+hr!) `Xamarin.Android.Build.Tests` tests in the Release
configuration, not the Debug + Release configurations.
Context: https://github.com/jonathanpeppers/xabuild
What do we want? A *usable*, *parallel*, build tree.
To elaborate: we want to be able to have a "system" Xamarin.Android
install, and a "parallel" xamarin-android install, and be able to
*easily* switch between the two. (For various definitions of "easy";
here, we mean *command-line* use, not IDE use.)
On macOS, this (more or less) Just Works™, and is extremely handy
for testing bug fixes:
$ xbuild /t:Install Project.csproj
# Verify that some bug is triggered
$ xbuild /t:Uninstall Project.csproj
$ tools/scripts/xabuild /t:Install Project.csproj
# Verify that some bug is *fixed*
There's One Problem™ with this: MSBuild does not make this easy.
(Related: commit aa1db830.) Apps may rely on files located within
`$(MSBuildExtensionsPath)` or `$(TargetFrameworkRootPath)`, files
such as PCL profile assemblies, or 3rd party frameworks.
Meanwhile, on macOS, `xabuild` is *predicated* upon overriding
`$(TargetFrameworkRootPath)` and `$(MSBuildExtensionsPath)` and
`$(XBUILD_FRAMEWORK_FOLDERS_PATH)`, and creating a bunch of symlinks
to "fake out" `msbuild.exe` so that system-installed files such as
PCL assemblies can be found *through* the parallel environment.
It kinda/sorta works on macOS. It completely falls apart when using
Windows. There is no easy "symlink half the world" solution there.
Overriding `$(MSBuildExtensionsPath)` means that
`Microsoft.Common.targets` can't be found. Overriding
`$(TargetFrameworkRootPath)` means PCL files can't be found.
It's a mess.
Fortunately, more recent versions of MSBuild allow for some of these
properties to contain *multiple* directories instead of a single
directory, which means *there is a way* to support our desired
usable parallel install world order. We "just" need to e.g.
force `$(MSBuildExtensionsPath)` to contain *both* the in-tree
directory *and* the system directory:
```xml
<MSBuildExtensionsPath>In-Tree Directory; System Directory</MSBuildExtensionsPath>
```
Unfortunately, *this isn't easy*. Not all of these properties can be
overridden on the `msbuild.exe` command line. Worse, MSBuild doesn't
allow `;` to be part of a property value, as `;` is a property name
[separator char](https://msdn.microsoft.com/en-us/library/ms164311.aspx)
msbuild.exe /property:WarningLevel=2;OutDir=bin\Debug
The way to force MSBuild to accept multiple paths in a property value
is by providing a `.exe.config` file with the appropriate values.
~~ Enter `xabuild.exe` ~~
`xabuild.exe` is a nice wrapper around MSBuild for compiling
Xamarin.Android projects using a locally built version of
Xamarin.Android on your system. It seems to work on Windows, macOS, and
Linux and doesn’t require elevation or modifications to your system.
`xabuild.exe` works by doing the following:
1. Reference `MSBuild.exe` or `MSBuild.dll` depending on the platform
2. Creates symbolic links to `.NETPortable` and `.NETFramework`
directories inside the Xamarin.Android build output directory
3. Overrides MSBuild's `app.config` file to set various properties,
such as `$(MSBuildExtensionsPath)`.
4. Run MSBuild’s `Main()` method
~~ Usage ~~
On macOS, `tools/scripts/xabuild` has been updated to use
`xabuild.exe` when `$MSBUILD` is `msbuild:
$ MSBUILD=msbuild tools/scripts/xabuild /t:SignAndroidPackage samples/HelloWorld/HelloWorld.csproj
When `$MSBUILD` is `xbuild` (the current default), the previous
behavior of overriding `$MSBuildExtensionsPath` and
`$XBUILD_FRAMEWORK_FOLDERS_PATH` is still used.
On Windows, MSBuild 15.3 from Visual Studio 2017 is required.
Simply execute `xabuild.exe`:
> bin\Debug\bin\xabuild.exe Xamarin.Android-Tests.sln /p:XAIntegratedTests=False
Before `xabuild.exe` existed, `setup-windows.exe` would need to be
executed (as Administrator!) in order for `Xamarin.Android-Tests.sln`
to be built using `msbuild.exe`.
The results reporting acted weird and hid 2 failures, which were not
reported in the instrumentation result lines
INSTRUMENTATION_RESULT: failed=0
INSTRUMENTATION_RESULT: inconclusive=0
INSTRUMENTATION_RESULT: passed=20134
INSTRUMENTATION_RESULT: run=20466
INSTRUMENTATION_RESULT: nunit2-results-path=/data/data/Xamarin.Android.Bcl_Tests/files/.__override__/TestResults.xml
INSTRUMENTATION_RESULT: skipped=322
INSTRUMENTATION_CODE: -1
and were only reported in the
`TestResult-Xamarin.Android.Bcl_Tests-Release.xml` file.
From `adb logcat` output we can see the crash (same for both failures):
--------- beginning of crash
E/AndroidRuntime( 3826): FATAL EXCEPTION: main
E/AndroidRuntime( 3826): Process: Xamarin.Android.Bcl_Tests, PID: 3826
E/AndroidRuntime( 3826): android.runtime.JavaProxyThrowable: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NotSupportedException: Linked away.
E/AndroidRuntime( 3826): at (wrapper managed-to-native) System.Object:__icall_wrapper_ves_icall_object_new_specific (intptr)
E/AndroidRuntime( 3826): at MonoTests.System.Runtime.Remoting.ContextTest..ctor () [0x00000] in <0484cb939a1a4a72be4938b3c08edcaa>:0
E/AndroidRuntime( 3826): at (wrapper managed-to-native) System.Reflection.MonoCMethod:InternalInvoke (System.Reflection.MonoCMethod,object,object[],System.Exception&)
E/AndroidRuntime( 3826): at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00002] in <0e1d684ae38a4822aaf6364f06390ad6>:0
E/AndroidRuntime( 3826): --- End of inner exception stack trace ---
That lead us to [`mono/mono/metadata/object.c`][object.c] source,
where `mono_error_set_not_supported (error, "Linked away.");` is
used 6 times.
[object.c]: https://github.com/mono/mono/blob/b263d37/mono/metadata/object.c#L5303
One of the cases was related to:
```c
im = mono_class_get_method_from_name (klass, "CreateProxyForType", 1);
```
That led us to the linked away
`System.Runtime.Remoting.Activation.ActivationServices.CreateProxyForType()`
method.
Update `Xamarin.Android.Bcl-Tests/Resources/LinkerDescription.xml` to
preserve `ActivationServices.CreateProxyForType()`, allowing these
tests to succeed in a `LinkSDKOnly` environment.
Alexander Köplinger
Dean Ellis
Atsushi Eno
Jonathan Peppers
Jonathan Pryor
Marek Habersack
Radek Doulik