Update documentation for new megazord technology.

This commit is contained in:
Ryan Kelly 2019-06-28 15:59:52 +10:00 коммит произвёл Thom Chiovoloni
Родитель b0680ffb55
Коммит 422e3e055a
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 31F01AEBD799934A
6 изменённых файлов: 208 добавлений и 157 удалений

Просмотреть файл

@ -48,8 +48,7 @@ The code for these components is organized as follows:
applications.
* The [Swift bindings](components/logins/ios) for use by iOS applications.
* [./megazords/](megazords) contains infrastructure for bundling multiple rust
components into a single build artifact called a
"[megazord library](https://github.com/mozilla/application-services/blob/master/docs/product-portal/applications/consuming-megazord-libraries.md)"
components into a single build artifact called a "[megazord library](docs/design/megazords.md)"
for easy consumption by applications.
For more details on how the client libraries are built and published, please see

Просмотреть файл

@ -17,6 +17,9 @@ you work on app-services projects. We have:
* Docs for various infrastructure pieces:
* Our [Dependency Management Policies](./dependency-management.md)
* Our [Build and Publish Pipeline](./build-and-publish-pipeline.md)
* Architectural design docs:
* How [megazording](./design/megazords.md) works, and why we do it.
* The motivation and design of the [sync manager](./design/sync-manager.md).
* Howtos for specific coding activities:
* Code and architecture guidelines:
* [Guide to Building a Rust Component](./howtos/building-a-rust-component.md)

90
docs/design/megazords.md Normal file
Просмотреть файл

@ -0,0 +1,90 @@
# Megazording
Each Rust component published by Application Services is conceptually a stand-alone library, but for
distribution we compile all the rust code for all components together into a single `.so` file. This
has a number of advantages:
* Easy and direct interoperability between different components at the Rust level
* Cross-component optimization
* Reduced code size to to distributing a single copy of the rust stdlib, low-level dependencies, etc.
This process is affectionately known as "megazording" and the resulting artifact as a ***megazord library***.
On iOS, this process is quite straightforward: we build all the rust code into a single statically-linked
framework, and the consuming application can import the corresponding Swift wrappers and link in just the
parts of the framework that it needs at compile time.
On Android, the situation is more complex due to the way packages and dependencies are managed.
We need to distribute each component as a separate Android ARchive (AAR) that can be managed as a dependency
via gradle, we need to provide a way for the application to avoid shipping rust code for components that it
isn't using, and we need to do it in a way that maintanins the advantages listed above.
This document describes our current approach to meeting all those requirements on android.
## AAR Dependency Graph
We publish a separate AAR for each component (e.g. fxaclient, places, logins) that contains
*just* the Kotlin wrappers that expose it to Android. Each of these AARs depends on a separate
shared "megazord" AAR in which all the rust code has been compiled together into a single `.so` file.
The application's dependency graph thus looks like this:
[![megazord dependency diagram](https://docs.google.com/drawings/d/e/2PACX-1vTA6wL3ibJRNjKXsmescTfKTx0w_fpr5NcDIF_4T5AsnZfCi8UEEcav8vibocSyKpHOQOk5ysiDBm-D/pub?w=727&h=546)](https://docs.google.com/drawings/d/1owo4wo2F1ePlCq2NS0LmAOG4jRoT_eVBahGNeWHuhJY/)
This generates a strange inversion of dependency flow in our build pipeline:
* Each individual component defines both a rust crate and an android AAR.
* There is a special "full-megazord" component that also defines a rust crate and an android AAR.
* The full-megazord rust crate depends on the rust crates for each individual component.
* But the android AAR for each component depends on the android AAR of the full-megazord!
However, this has the benefit that we can use gradle's dependency-substitution features to easily manage
the rust code that is shipping in each application.
## Custom Megazords
By default, an application that uses *any* appservices component will include the compiled rust code
for *all* appservices components.
To reduce its code size, the application can use [dependency
substitution](https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.DependencySubstitutions.html) to
replace the "full-megazord" AAR with a custom-built megazord AAR containing only the components it requires.
Such an AAR can be built in the same way as the "full-megazord", and simply avoid depending on the rust
crates for components that are not required.
To help ensure this replacement is done safely at runtime, the `mozilla.appservices.support.native` provides
helper functions for loading the correct megazord `.so` file. The Kotlin wrapper for each component should
load its shared library by calling `mozilla.appservices.support.native.loadIndirect`, specifying both the
name of the component and the expected version number of the shared library.
XXX TODO: explain a bit about how it uses system properties to manage which library gets loaded.
## Unit Tests
XXX TODO: explain the `forUnitTests` thing here.
## Gotchas and Rough Edges
This setup mostly works, but has a handful of rough edges.
The `build.gradle` for each component needs to declare an explicit dependency on `project(":full-megazord")`,
otherwise the resulting AAR will not be able to locate the compiled rust code at runtime. It also needs to
declare a dependency between its build task and that of the full-megazord, for reasons. Typically this looks something
like:
```
tasks["generate${productFlavor}${buildType}Assets"].dependsOn(project(':full-megazord').tasks["cargoBuild"])
```
In order for unit tests to work correctly, the `build.gradle` for each component needs to add the `rustJniLibs`
directory of the full-megazord project to its `srcDirs`, otherwise the unittests will not be able to find and load
the compiled rust code. Typically this looks something like:
```
test.resources.srcDirs += "${project(':full-megazord').buildDir}/rustJniLibs/desktop"
```
The above also means that unittests will not work correctly when doing local substitutions builds,
because it's unreasonable to expect the main project (e.g. Fenix) to include the above in its build scripts.

Просмотреть файл

@ -1,8 +1,9 @@
# Using locally-published components in Fenix
Note: This is a bit tedious, so feel free to ask in slack if there's a better
way to test things. At the moment substitution is broken, so this is necessary
in more cases than we'd like.
Note: This is a bit tedious, and you might like to try the substitution-based
approach documented in [Development with the Reference Browser](./working-with-reference-browser.md).
That approach is still fairly new, and the local-publishing approach in this document
is necessary if it fails.
Note 2: This is fenix-specific only in that some links on the page go to the
`mozilla-mobile/fenix` repository, and that I'm describing `fenix`, however
@ -10,22 +11,7 @@ these steps should work for e.g. `reference-browser`, as well. (Same goes for
lockwise, or any other consumer of our components, but they may use a different
structure -- lockwise has no Dependencies.kt, for example)
1. If you've added a new project to the megazord:
1. In the gradle plugin's [`AppServicesExtension.kt`](AppServicesExtension),
add the new component to the relevant megazord. Note: you may have
already done this.
2. In the gradle plugin's [`build.gradle`](plugin-build-gradle), change
`ext.plugin_version` to end with `-TESTING$N`
<sup><a href="#note1">1</a></sup> where `$N` is some number
that you haven't used for this before.
Example: `ext.plugin_version = '0.4.4-TESTING3'`
3. Inside the `gradle-plugin` directory run `./gradlew publishToMavenLocal`.
It's important that you do this from inside the plugin's directory,
e.g. cwd must be `path/to/application-services/gradle-plugin`!
2. Inside the `application-services` repository root:
1. Inside the `application-services` repository root:
1. In [`.buildconfig-android.yml`](app-services-yaml), change
`libraryVersion` to end in `-TESTING$N` <sup><a href="#note1">1</a></sup>,
where `$N` is some number that you haven't used for this before.
@ -37,7 +23,7 @@ structure -- lockwise has no Dependencies.kt, for example)
the next step much faster.
3. Run `./gradlew publishToMavenLocal`. This may take between 5 and 10 minutes.
3. Inside the `android-components` repository root:
2. Inside the `android-components` repository root:
1. In [`.buildconfig.yml`](android-components-yaml), change
`componentsVersion` to end in `-TESTING$N` <sup><a href="#note1">1</a></sup>,
where `$N` is some number that you haven't used for this before.
@ -57,7 +43,7 @@ structure -- lockwise has no Dependencies.kt, for example)
5. Run `./gradlew publishToMavenLocal`.
4. Inside the `fenix` repository root:
3. Inside the `fenix` repository root:
1. Inside [`build.gradle`](fenix-build-gradle-1), add
`mavenLocal()` inside `allprojects { repositories { <here> } }`.
1. If you added a new project to the megazord (e.g. you went through the
@ -107,8 +93,6 @@ matched (e.g. all of the identifiers ended in `-TESTING3`, this is not required,
so long as you match everything up correctly at the end. This can be tricky, so
I always try to use the same number).
[AppServicesExtension]: https://github.com/mozilla/application-services/blob/594f4e3f6c190bc5a6732f64afc573c09020038a/gradle-plugin/src/main/kotlin/mozilla/appservices/AppServicesExtension.kt#L21-L55
[plugin-build-gradle]: https://github.com/mozilla/application-services/blob/594f4e3f6c190bc5a6732f64afc573c09020038a/gradle-plugin/build.gradle#L3
[app-services-yaml]: https://github.com/mozilla/application-services/blob/594f4e3f6c190bc5a6732f64afc573c09020038a/.buildconfig-android.yml#L1
[android-components-yaml]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/.buildconfig.yml#L1
[android-components-deps]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/buildSrc/src/main/java/Dependencies.kt#L37

Просмотреть файл

@ -6,7 +6,7 @@ This is a companion to the [equivalent instructions for the android-components r
Modern Gradle supports [composite builds](https://docs.gradle.org/current/userguide/composite_builds.html), which allows to substitute on-disk projects for binary publications. Composite builds transparently accomplish what is usually a frustrating loop of:
1. change library
1. publishing library snapshot to the local Maven repository
1. publish library snapshot to the local Maven repository
1. consume library snapshot in application
## Preparation
@ -35,7 +35,9 @@ rust.targets=x86
## Substituting projects
### Using local.properties
Both android-components and reference-browser have custom build logic for dealing with composite builds,
so you should be able to configure it by simply adding the path to the application-services repo
in the correct `local.properties` file:
In `android-components/local.properties`:
```groovy
@ -47,18 +49,12 @@ In `reference-browser/local.properties`:
substitutions.application-services.dir=../application-services
```
### Using settings.gradle
If this doesn't seem to work, or if you need to configure composite builds for a project that does
not contain this custom logic, add the following to `settings.gradle`:
In `android-components/settings.gradle`:
```groovy
includeBuild('../application-services') {
dependencySubstitution {
// As required.
substitute module('org.mozilla.appservices:fxaclient') with project(':fxaclient')
substitute module('org.mozilla.appservices:logins') with project(':logins')
substitute module('org.mozilla.appservices:places') with project(':places')
}
}
includeBuild('../application-services')
```
In `reference-browser/settings.gradle`:
@ -75,14 +71,7 @@ includeBuild('../android-components') {
// Gradle handles transitive dependencies just fine, but Android Studio doesn't seem to always do
// the right thing. Duplicate the transitive dependencies from `android-components/settings.gradle`
// here as well.
includeBuild('../application-services') {
dependencySubstitution {
// As required.
substitute module('org.mozilla.appservices:fxaclient') with project(':fxaclient')
substitute module('org.mozilla.appservices:logins') with project(':logins')
substitute module('org.mozilla.appservices:places') with project(':places')
}
}
includeBuild('../application-services')
```
## Caveat

Просмотреть файл

@ -6,141 +6,127 @@ sidebar_label: Consuming megazord libraries
# Megazord libraries
The Rust component libraries that Application Services publishes stand alone: each published Android
ARchive (AAR) contains managed code (`classes.jar`) and multiple `.so` library files (one for each
supported architecture). That means consuming multiple such libraries entails at least two `.so`
libraries, and each of those libraries includes the entire Rust standard library as well as
(potentially many) duplicated dependencies. To save space and allow cross-component native-code
Link Time Optimization (LTO, i.e., inlining, dead code elimination, etc) Application Services also
publishes aggregate libraries -- so called *megazord libraries* -- that compose multiple Rust
components into a single optimized `.so` library file. The managed code can be easily configured to
use such a megazord without significant changes.
Each Rust component published by Application Services is conceptually a stand-alone library, but they
all depend on a shared core of functionality for exposing Rust to Kotlin. In order to allow easy interop
between components, enable cross-component native-code Link Time Optimization, and reduce final application
size, the rust code for all components is compiled and distributed together as a single aggregate library
which we have dubbed a ***megazord library***.
There are two tasks we want to arrange. First, we want to substitute component modules for the
single aggregate megazord module (a process that we call "megazording"); second, we want to arrange
for native Rust code to be available to JVM unit tests. (They're related because the unit test
changes depend on the megazord used.)
Each Application Services component is published as an Android ARchive (AAR) that contains the managed code
for that component (`classes.jar`) and which depends on a separate "megazord" AAR that contains all of the
compiled rust code (`libmegazord.so`). For an application that consumes multiple Application Services components,
the dependency graph thus looks like this:
Both tasks are handled by the
[org.mozilla.appservices](https://github.com/mozilla/application-services/gradle-plugin/README.md)
Gradle plugin.
[![megazord dependency diagram](https://docs.google.com/drawings/d/e/2PACX-1vTA6wL3ibJRNjKXsmescTfKTx0w_fpr5NcDIF_4T5AsnZfCi8UEEcav8vibocSyKpHOQOk5ysiDBm-D/pub?w=727&h=546)](https://docs.google.com/drawings/d/1owo4wo2F1ePlCq2NS0LmAOG4jRoT_eVBahGNeWHuhJY/)
# Consuming megazords
While this setup is *mostly* transparent to the consuming application, there are a few points to be aware of
which are outlined below.
You'll need to:
## Initializing Shared Infrastructure
1. Choose a megazord from the [list of megazords](#megazords) that Application Services produces in automation.
1. [Apply](#apply-the-gradle-plugin) the `org.mozilla.appservices` Gradle plugin.
1. [Configure](#configure-the-gradle-plugin) the Gradle plugin.
1. [Call `.init()`](#configuring-the-consuming-application) in your `Application.onCreate()`.
1. [Verify](#verify-that-your-apk-is-megazorded) that your APK is megazorded.
The megazord AAR exposes a single additional JVM class, `mozilla.appservices.Megazord`, which the application
should initialize explicitly. This would typically be done in the `Application.onCreate()` method, like so:
## Megazords
```kotlin
import mozilla.appservices.Megazord
open class Application extends android.app.Application {
override fun onCreate() {
super.onCreate();
Megazord.init();
}
...
}
```
The `init()` method sets some Java system properties that help the component modules locate the compiled
rust code.
After initializing the Megazord, the application can configure shared infrastructure such as logging:
```kotlin
import mozilla.components.support.rustlog.RustLog
open class Application extends android.app.Application {
override fun onCreate() {
...
Megazord.init();
...
RustLog.enable()
...
}
}
```
Or networking:
```kotlin
import mozilla.components.lib.fetch.httpurlconnection.HttpURLConnectionClient
import mozilla.appservices.httpconfig.RustHttpConfig
open class Application extends android.app.Application {
override fun onCreate() {
...
Megazord.init();
...
RustHttpConfig.setClient(lazy { HttpURLConnectionClient() })
...
}
}
```
The configured settings will then be used by all rust components provided by the megazord.
## Using a custom Megazord
The default megazord library contains compiled rust code for *all* components published by Application Services.
If the consuming application only uses a subset of those components, it's possible to its package size and load
time by using a custom-built megazord library containing only the required components.
First, you will need to select an appropriate custom megazord. Application Services publishes several custom megazords
to fit the needs of existing Firefox applications:
| Name | Components | Maven publication |
| --- | --- | --- |
| `lockbox` | `fxaclient`, `logins` | `org.mozilla.appservices:lockbox-megazord` |
| `reference-browser` | `fxaclient`, `logins`, `places` | `org.mozilla.appservices:reference-browser-megazord` |
| `fenix` | `fxaclient`, `logins`, `places` | `org.mozilla.appservices:fenix-megazord` |
If your project needs an additional megazord, talk to #rust-components on Slack.
## Apply the Gradle plugin
<a alt="Version badge" href="https://plugins.gradle.org/plugin/org.mozilla.appservices.gradle-plugin">
<img align="left" src="https://img.shields.io/maven-metadata/v/https/plugins.gradle.org/m2/org/mozilla/appservices/org.mozilla.appservices.gradle.plugin/maven-metadata.xml.svg?label=org.mozilla.appservices&colorB=brightgreen" />
</a>
<br/>
Build script snippet for plugins DSL for Gradle 2.1 and later:
Then, simply use gradle's builtin support for [dependency
substitution](https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.DependencySubstitutions.html)
to replace the default "full megazord" with your selected custom build:
```groovy
plugins {
id 'org.mozilla.appservices' version '0.1.0'
}
```
Build script snippet for use in older Gradle versions or where dynamic configuration is required:
```groovy
buildscript {
repositories {
maven {
url 'https://plugins.gradle.org/m2/'
}
}
dependencies {
classpath 'gradle.plugin.org.mozilla.appservices:gradle-plugin:0.1.0"
configurations.all {
resolutionStrategy.dependencySubstitution {
substitute module("org.mozilla.appservices:full-megazord:X.Y.Z") with module("org.mozilla.appservices:fenix-megazord:X.Y.Z")
}
}
apply plugin: 'org.mozilla.appservices'
```
## Configure the Gradle plugin
If you would like a new custom megazord for your project, please reach out via #rust-components in slack.
To consume a specific megazord module, use something like:
## Running unit tests
Since the megazord library contains compiled native code, it cannot be used directly for running local unittests
(it's compiled for the android target device, not for your development host machine). To support running unittests
via the JVM on the host machine, we publish a special `forUnitTests` variant of the megazord library in which the
native code is compiled into a JAR for common desktop architectures.
Use dependency substitution to include it in your test configuration as follows:
```groovy
appservices {
defaultConfig {
// Megazord in all Android variants. The default is to not megazord.
megazord = 'lockbox' // Or 'reference-browser', etc.
enableUnitTests = false // Defaults to true.
}
```
If you need, you can configure per Android variant: see the
[plugin docs](https://github.com/mozilla/application-services/gradle-plugin/README.md).
## Configuring the consuming application
The megazord modules expose a single additional JVM class, like
`org.mozilla.appservices.{Lockbox,ReferenceBrowser}Megazord`. That class has a single static
`init()` method that consuming applications should invoke in their `Application.onCreate()` method,
like:
```xml
<manifest>
<application android:name=".Application" ...>
</application>
...
</manifest>
```
and:
```java
public class Application extends android.app.Application {
@Override
public void onCreate() {
super.onCreate();
org.mozilla.appservices.LockboxMegazord.init();
}
...
configurations.testImplementation.resolutionStrategy.dependencySubstitution {
substitute module("org.mozilla.appservices:full-megazord:X.Y.Z") with module("org.mozilla.appservices:full-megazord-forUnitTests:X.Y.Z")
}
```
The `init()` method sets some Java system properties that tell the component modules which megazord
native code library contains the underlying component native code.
If you are using a custom megazord library, substitute both the default and custom module with the `forUnitTests`
variant of your custom megazord:
## Verify that your APK is megazorded
After `./gradlew app:assembleDebug`, list the contents of the APK produced. For the Reference
Browser, this might be like:
```
./gradlew app:assembleGeckoNightlyArmDebug
unzip -l app/build/outputs/apk/geckoNightlyArm/debug/app-geckoNightly-arm-armeabi-v7a-debug.apk | grep lib/
```
You should see a single megazord `.so` library, like:
```
5172812 00-00-1980 00:00 lib/armeabi-v7a/libreference_browser.so
```
and no additional _component_ `.so` libraries (like `libfxaclient_ffi.so`). You will see additional
`.so` libraries -- just not component libraries, which are generally suffixed `_ffi.so`.
Then exercise your functionality on device and don't think about megazording again!
```groovy
configurations.testImplementation.resolutionStrategy.dependencySubstitution {
substitute module("org.mozilla.appservices:full-megazord:X.Y.Z") with module("org.mozilla.appservices:fenix-megazord-forUnitTests:X.Y.Z")
substitute module("org.mozilla.appservices:fenix-megazord:X.Y.Z") with module("org.mozilla.appservices:fenix-megazord-forUnitTests:X.Y.Z")
}
```