Add a full "how to add a component" howto (and various other doc updates) [ci skip] (#3987)

* Add a walkthrough for adding a new component from scratch.

There are a lot of non-obvious things you need to do when adding
a new component to this repo. I've tried to capture them all
in one reference guide, including screenshots for the part
where you have to mess around in the XCode GUI.

There are probably parts of this that could be simplified,
but let's start by writing the whole process down.

* Delete the "using protobuf-encoded data over Rust FFI" howto.

We have some existing components that use this approach, but any
new components should use UniFFI's builtin "record" types for
passing data rather than using protobuf. Thus, we don't expect
anyone to actually need this howto in future.

* Morph the "exposing rust code on android" howto into a simpler FAQ.

The existing doc was a hybrid howto/FAQ already, and we intend for
the "howto" part to be entirely replaced by "use UniFFI". So, let's
salvage content that's still useful to us and put it in a deliberate
FAQ.

* Remove the "consuming rust components on android" howto.

It's basically content-free, and the one link it has there about
megazords has been dead for quite some time. This doc is obviously
not providing any value in practice.

* Remove the "exposing rust components to swift" howto.

It's pretty meagre, and it was pretty much all about the megazord config changes,
which are now covered by the general "adding a new component" guide.
This commit is contained in:
Ryan Kelly 2021-03-29 11:39:14 +11:00 коммит произвёл GitHub
Родитель 20f940e34f
Коммит e29b4f772f
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
15 изменённых файлов: 323 добавлений и 741 удалений

121
docs/android-faqs.md Normal file
Просмотреть файл

@ -0,0 +1,121 @@
# Rust + Android FAQs
### How do I expose Rust code to Kotlin?
Use [UniFFI](https://mozilla.github.io/uniffi-rs/), which can produce Kotlin
bindings for your Rust code from an interface definition file.
If UniFFI doesn't currently meet your needs, please [open an issue](
https://github.com/mozilla/uniffi-rs/issues) to discuss how the tool can
be improved.
As a last resort, you can make hand-written bindings from Rust to Kotlin,
essentially manually performing the steps that UniFFI tries to automate
for you: flatten your Rust API into a bunch of `pub extern "C"` functions,
then use [JNA](https://github.com/java-native-access/jna) to call them
from Kotlin. The details of how to do that are well beyond the scope of
this document.
### How should I name the package?
Published packages should be named `org.mozilla.appservices.$NAME` where `$NAME`
is the name of your component, such as `logins`. The Java namespace in which
your package defines its classes etc should be `mozilla.appservices.$NAME.*`.
### How do I publish the resulting package?
Add it to `.buildconfig-android.yml` in the root of this repository.
This will cause it to be automatically included as part of our release
publishing pipeline.
### How do I know what library name to load to access the compiled rust code?
Assuming that you're building the Rust code as part of the application-services
build and release process, your `pub extern "C"` API should always be available
from a file named `libmegazord.so`.
### What challenges exist when calling back into Kotlin from Rust?
There are a number of them. The issue boils down to the fact that you need to be
completely certain that a JVM is associated with a given thread in order to call
java code on it. The difficulty is that the JVM can GC its threads and will not
let rust know about it.
JNA can work around this for us to some extent, at the cost of some complexity.
The approach it takes is essentially to spawn a thread for each callback
invocation. If you are certain youre going to do a lot of callbacks and they
all originate on the same thread, you can have them all run on a single thread
by using the [`CallbackThreadInitializer`](
https://java-native-access.github.io/jna/4.2.1/com/sun/jna/CallbackThreadInitializer.html).
With the help of JNA's workarounds, calling back from Rust into Kotlin isnt too bad
so long as you ensure that Kotlin cannot GC the callback while rust code holds onto it
(perhaps by stashing it in a global variable), and so long as you can either accept the overhead of extra threads being instantiated on each call or are willing to manage
the threads explicitly.
Note that the situation would be somewhat better if we used JNI directly (and
not JNA), but this would cause us to need to generate different Rust FFI code for
Android than for iOS.
Ultimately, in any case where there is an alternative to using a callback, you
should probably pursue that alternative.
For example if you're using callbacks to implement async I/O, it's likely better to
move to doing a blocking call, and have the calling code dispatch it on a background
thread. Its very easy to run such things on a background thread in Kotlin, is in line
with the Android documentation on JNI usage, and in our experience is vastly simpler
and less painful than using callbacks.
(Of course, not every case is solvable like this).
### Why are we using JNA rather than JNI, and what tradeoffs does that involve?
We get a couple things from using JNA that we wouldn't with JNI.
1. We are able to use the same Rust FFI code on all platforms. If we used JNI we'd
need to generate an Android-specific Rust FFI crate that used the JNI APIs, and
a separate Rust FFI crate for exposing to Swift.
2. JNA provides a mapping of threads to callbacks for us, making callbacks over
the FFI possible. That said, in practice this is still error prone, and easy
to misuse/cause memory safety bugs, but it's required for cases like logging,
among others, and so it is a nontrivial piece of complexity we'd have to
reimplement.
However, it comes with the following downsides:
1. JNA has bugs. In particular, its not safe to use bools with them, it thinks
they are 32 bits, when on most platforms (every platform Rust supports) they
are 8 bits. They've been unwilling to fix the issue due to it breaking
backwards compatibility (which is... somewhat fair, there is a lot of C89
code out there that uses `bool` as a typedef for a 32-bit `int`).
2. JNA makes it really easy to do the wrong thing and have it work but corrupt
memory. Several of the caveats around this are documented in the
[`ffi_support` docs](https://docs.rs/ffi-support/*/ffi_support/), but a
major one is when to use `Pointer` vs `String` (getting this wrong will
often work, but may corrupt memory).
We aim to avoid triggering these bugs by auto-generating the JNA bindings
rather than writing them by hand.
### How do I debug Rust code with the step-debugger in Android Studio
1. Uncomment the `packagingOptions { doNotStrip "**/*.so" }` line from the
build.gradle file of the component you want to debug.
2. In the rust code, either:
1. Cause something to crash where you want the breakpoint. Note: Panics
don't work here, unfortunately. (I have not found a convenient way to
set a breakpoint to rust code, so
`unsafe { std::ptr::write_volatile(0 as *const _, 1u8) }` usually is
what I do).
2. If you manage to get an LLDB prompt, you can set a breakpoint using
`breakpoint set --name foo`, or `breakpoint set --file foo.rs --line 123`.
I don't know how to bring up this prompt reliably, so I often do step 1 to
get it to appear, delete the crashing code, and then set the
breakpoint using the CLI. This is admittedly suboptimal.
3. Click the Debug button in Android Studio, to display the "Select Deployment
Target" window.
4. Make sure the debugger selection is set to "Both". This tends to unset
itself, so make sure.
5. Click "Run", and debug away.

Просмотреть файл

@ -0,0 +1,193 @@
# Adding a new component to Application Services
Each component in the Application Services repo has three parts (the Rust code,
the Kotlin wrapper, and the Swift wrapper) so there are quite a few moving
parts involved in adding a new component. This is a rapid-fire list of all
the things you'll need to do if adding a new component from scratch.
## The Rust Code
Your component should live under `./components` in this repo.
Use `cargo new --lib ./components/<your_crate_name>`to create a new library crate,
and please try to avoid using hyphens in the crate name.
See the [Guide to Building a Rust Component](./building-a-rust-component.md) for general
advice on designing and structuring the actual Rust code, and follow the
[Dependency Management Guidelines](../dependency-management.md) if your crate
introduces any new dependencies.
Use [UniFFI](https://mozilla.github.io/uniffi-rs/) to define how your crate's
API will get exposed to foreign-language bindings. By convention, put the interface
definition file at `./components/<your_crate_name>/<your_crate_name>.udl`. Use
the `builtin-bindgen` feature of UniFFI to simplify the build process, by
putting this in your `Cargo.toml`:
```
[build-dependencies]
uniffi_build = { version = "<latest version here>", features=["builtin-bindgen"] }
```
Include your new crate in the application-services workspace, by adding
it to the `members` and `default-members` lists in the `Cargo.toml` at
the root of the repository.
In order to be published to consumers, your crate must be included in the
["megazord"](../design/megazords.md) crate for each target platform:
* For Android, add it as a dependency in `./megazords/full/Cargo.toml` and
add a `pub use <your_crate_name>` to `./megazords/full/src/lib.rs`.
* For iOS, add it as a dependency in `./megazords/ios/rust/Cargo.toml` and
add a `pub use <your_crate_name>` to `./megazords/ios/rust/src/lib.rs`.
Run `cargo check -p <your_crate_name>` in the repository root to confirm that
things are configured properly. This will also have the side-effect of updating
`Cargo.lock` to contain your new crate and its dependencies.
## The Kotlin Bindings
Make a `./components/<your_crate_name>/android` subdirectory to contain
Kotlin- and Android-specific code. This directory will contain a gradle
project for building your Kotlin bindings.
Copy the `build.gradle` file from `./components/crashtest/android/` into
your own component's directory, and edit it to replace the references to
`crashtest.udl` with your own component's `.udl` file.
Create a file `./components/<your_crate_name>/uniffi.toml` with the
following contents:
```
[bindings.kotlin]
package_name = "mozilla.appservices.<your_crate_name>"
cdylib_name = "megazord"
```
Create a file `./components/<your_crate_name>/android/src/main/AndroidManifest.xml`
with the following contents:
```
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="org.mozilla.appservices.<your_crate_name>" />
```
In the root of the repository, edit `.buildconfig-android.yml`to add
your component's metadata. This will cause it to be included in the
gradle workspace and in our build and publish pipeline. Check whether
it builds correctly by running:
* `./gradlew <your_crate_name>:assembleDebug`
You can include hand-written Kotlin code alongside the automatically
generated bindings, by placing `.kt`` files in a directory named:
* `./android/src/test/java/mozilla/appservices/<your_crate_name>/`
You can write Kotlin-level tests that consume your component's API,
by placing `.kt`` files in a directory named:
* `./android/src/test/java/mozilla/appservices/<your_crate_name>/`.
So you would end up with a directory structure something like this:
* `components/<your_crate_name>/`
* `Cargo.toml`
* `uniffi.toml`
* `src/`
* Rust code here.
* `android/`
* `build.gradle`
* `src/`
* `main/`
* `AndroidManifest.xml`
* `java/mozilla/appservices/<your_crate_name>/`
* Hand-written Kotlin code here.
* `test/java/mozilla/appservices/<your_crate_name>/`
* Kotlin test-cases here.
Run your component's Kotlin tests with `./gradlew <your_crate_name>:test`
to confirm that this is all working correctly.
## The Swift Bindings
Make a `./components/<your_crate_name>/ios` subdirectory to contain
Swift- and iOS-specific code. The UniFFI-generated swift bindings will
be written to a subdirectory named `Generated`.
You can include hand-written Swift code alongside the automatically
generated bindings, by placing `.swift` files in a directory named:
`./ios/<your_crate_name>/`.
So you would end up with a directory structure something like this:
* `components/<your_crate_name>/`
* `Cargo.toml`
* `uniffi.toml`
* `src/`
* Rust code here.
* `ios/`
* `<your_crate_name>/`
* Hand-written Swift code here.
* `Generated/`
* Generated Swift code will be written into this directory.
Edit `megazords/ios/MozillaAppServices.h` and add an import line for your component,
like:
```
#import "uniffi_<your_crate_name>_Bridging-Header.h"
```
You will then need to add your component into the iOS ["megazord"](../design/megazords.md)
XCode project, which can only really by done using the XCode application,
which can only really be done if you're on a Mac.
Open `megazords/ios/MozillaAppServices.xcodeproj` in XCode.
In the Project navigator, add a new Group for your new component, pointing to
the `./ios/` directory you created above. Add the following entries to the Group:
* The `.udl` file for you component, from `../src/<your_crate_name>.udl`.
* Any hand-written `.swift `files for your component
* A sub-group named "Generated", pointing to the `./Generated/` subdirectory, and
containing entries for the files generated by UniFFI:
* `<your_crate_name>.swift`
* `uniffi_<your_crate_name>-Bridging-Header.h`
The result should look something like this:
![Screenshot of XCode Project Navigator](./img/xcode_add_component_1.png)
Click on the top-level "MozillaAppServices" project in the navigator,
then go to "Build Phases" and add `<your_crate_name>.udl` to the list
of "Compile Sources". This will trigger an XCode Build Rule that generates
the Swift bindings automatically. Also include any hand-written `.swift` files
in this list.
The result should look something like this:
![Screenshot of XCode Compile Sources list](./img/xcode_add_component_2.png)
In the same "Build Phases" screen, under the "Headers" section, add `uniffi_<your_crate_name>-Bridging-Header.h` to the list of Public headers.
The result should look something like this:
![Screenshot of XCode Headers list](./img/xcode_add_component_3.png)
Build the project in XCode to check whether that all worked correctly.
To add Swift tests for your component API, create them in a file under
`megazords/ios/MozillaAppServicesTests/`. Use this syntax to import
your component's bindings from the compiled megazord:
```
@testable import MozillaAppServices
```
In XCode, navigate to the `MozillaAppServicesTests` Group and add your
new test file as an entry. Select the corresponding target, click on
"Build Phases", and add your test file to the list of "Compile Sources".
The result should look something like this:
![Screenshot of XCode Test Setup](./img/xcode_add_component_4.png)
Use the XCode Test Navigator to run your tests and check whether
they're passing.

Просмотреть файл

@ -21,6 +21,12 @@ To repeat with emphasis - **please consider this a living document**.
We think components should be structured as described here.
## We build libraries, not frameworks
Think of building a "library", not a "framework" - the application should be in
control and calling functions exposed by your component, not providing functions
for your component to call.
## The "store" is the "entry-point"
[Note that some of the older components use the term "store" differently; we

Просмотреть файл

@ -1,14 +0,0 @@
## Guide to Consuming Rust Components on Android
Welcome!
It's great that you want to include our Rust Components into your Android app,
but we haven't written this guide yet.
In fact we're still deciding whether this guide is even necessary. For now you
probably want to find the corresponding component over in
[android-components](https://github.com/mozilla-mobile/android-components/) and
consume it from there.
You might also want to read up on [consuming megazord
libraries](https://mozilla.github.io/application-services/docs/applications/consuming-megazord-libraries.html).

Просмотреть файл

@ -1,182 +0,0 @@
# How to expose your Rust Component to Kotlin
Welcome!
It's great that you've built a Rust Component and want to expose it to Koltin,
but we haven't written a complete guide yet.
Please share your :+1: on [the relevant github
issue](https://github.com/mozilla/application-services/issues/599) to let us
know that you wanted it.
In the meantime, here are some preliminary notes.
## Option 1: Autogenerated bindings with UniFFI
We have been working on a tool called [UniFFI](https://github.com/mozilla/uniffi-rs/) to automatically
generate FFI bindings from the Rust code. It's probably worth a look for your use case, but
is still quite new and may not support everything you need.
The [autofill](/components/autofill) component provides a useful example, and
the [UniFFI tutorial](https://mozilla.github.io/uniffi-rs/Getting_started.html)
walks through the steps involved.
## Option 2: Hand-written bindings
If UniFFI doesn't currently meet your needs, you may need to write your own FFI
and bindings layer.
### High-level overview.
The [logins](/components/logins) component provides a useful example. We assume
you have written a nice core of Rust code in a [./src/](/components/logins/src)
directory for your component.
First, you will need to flatten the Rust API into a set of FFI bindings,
conventionally in a [./ffi/](/components/logins/ffi) directory. Use the
[ffi_support](https://docs.rs/ffi-support/0.1.3/ffi_support/) crate to help make
this easier, which will involve implementing some traits in the core rust code.
Consult the crate's documentation for tips and gotchas.
Next, you will need to write Kotlin code that consumes that FFI,
conventionally in a [./android/](/components/logins/android) directory. This
code should use [JNA](https://github.com/java-native-access/jna) to load the
compiled rust code via shared library and expose it as a nice safe ergonomic
Kotlin API.
It seems likely that we could provide a useful template here to get you started.
But we haven't yet.
Finally, you will need to add your package into the
[android-components](https://github.com/mozilla-mobile/android-components) repo
via some undocumented process that starts by asking in the rust-components
channel on slack.
### How should I name the resulting package?
Published packages should be named `org.mozilla.appservices.$NAME` where `$NAME`
is the name of your component, such as `logins`. The Java namespace in which
your package defines its classes etc should be `mozilla.appservices.$NAME.*`.
### How do I publish the resulting package?
Great question! We should write or link to an answer here.
### How do I know what library name to load to access the compiled rust code?
Great question! We should write or link to an answer here.
### Why cant we use cbindgen or an alternative C binding generator?
We could, but it wouldnt save us that much. cbindgen would automate the process
of turning the ffi crate's source into the equivalent C header file, however it
wouldnt help with the kotlin code (although one could imagine something that
automated generation of the kotlin bindings file). In particular, it wouldnt
help us produce the ffi crates code, or the wrappers that make the bindings
safe (this is most of the work). It would be nice to automate the process, but
it hasnt been particularly error prone to update it by hand so far.
Its also worth noting that the ffi crates source doesnt have the ownership
information encoded in it with regards to strings (theyre all pointers to
c_char), which makes generating kotlin bindings harder, since the kotlin
bindings need to take/return Pointer for strings owned by Rust, and String for
strings owned by Kotlin. This is because when we later release the string owned
by rust, it must be the same pointer we were given earlier (If you pass String,
JNA allocates temporary memory for the call, passes you it, and releases it
afterwards).
(The solution to this would likely be something like wasm_bindgen, where you
annotate your Rust source. The tool would then spit out both the FFI crate, and
the bulk of the Kotlin/Swift APIs. This would be a lot of work, but it would be
cool to have someday).
cbindgen doesnt always work seamlessly. Its a standalone parser of rust code,
not a rustc plugin -- it doesnt always understand your rust library like the
compiler does. A number of these issues cropped up when autopush used it: e.g.
it being unable to parse newer rust code or having buggy handling of certain
rust syntax.
### What design principles should inform my component API?
Many, most of which aren't written down yet. This is an incomplete list:
* Avoid callbacks where possible, in favour of simple blocking calls.
* Think of building a "library", not a "framework"; the application should be in
control and calling functions exposed by your component, not providing
functions for your component to call.
### What challenges exist when calling back into Kotlin from Rust?
There are a number of them. The issue boils down to the fact that you need to be
completely certain that a JVM is associated with a given thread in order to call
java code on it. The difficulty is that the JVM can GC its threads and will not
let rust know about it. JNA can work around this for us to some extent, however
there are difficulties.
The approach it takes is essentially to spawn a thread for each callback
invocation. If you are certain youre going to do a lot of callbacks and they
all originate on the same thread, you can tell it to cache these.
Calling back from Rust into Kotlin isnt too bad so long as you ensure the
callback can not be GCed while rust code holds onto it, and you can either
accept the overhead of extra threads being instantiated on each call, or you can
ensure that it only happens from a single thread.
Note that the situation would be somewhat better if we used JNI directly (and
not JNA), but this would cause us to need to write two versions of each ffi
crate, one for iOS, and one for Android.
Ultimately, in any case where you can reasonably move to making something a
blocking call, do so. Its very easy to run such things on a background thread
in Kotlin. This is in line with the Android documentation on JNI usage, and my
own experience. Its vastly simpler and less painful this way.
(Of course, not every case is solvable like this).
### Why are we using JNA rather than JNI, and what tradeoffs does that involve?
We get a couple things from using JNA that we wouldn't with JNI.
1. We are able to write a *single* FFI crate. If we used JNI we'd need to write
one FFI that android calls, and one that iOS calls.
2. JNA provides a mapping of threads to callbacks for us, making callbacks over
the FFI possible. That said, in practice this is still error prone, and easy
to misuse/cause memory safety bugs, but it's required for cases like logging,
among others, and so it is a nontrivial piece of complexity we'd have to
reimplement.
However, it comes with the following downsides:
1. JNA has bugs. In particular, its not safe to use bools with them, it thinks
they are 32 bits, when on most platforms (every platform Rust supports) they
are 8 bits. They've been unwilling to fix the issue due to it breaking
backwards compatibility (which is... somewhat fair, there is a lot of C89
code out there that uses `bool` as a typedef for a 32-bit `int`).
2. JNA makes it really easy to do the wrong thing and have it work but corrupt
memory. Several of the caveats around this are documented in the
[`ffi_support` docs](https://docs.rs/ffi-support/*/ffi_support/), but a
major one is when to use `Pointer` vs `String` (getting this wrong will
often work, but may corrupt memory).
### How do I debug Rust code with the step-debugger in Android Studio
1. Uncomment the `packagingOptions { doNotStrip "**/*.so" }` line from the
build.gradle file of the component you want to debug.
2. In the rust code, either:
1. Cause something to crash where you want the breakpoint. Note: Panics
don't work here, unfortunately. (I have not found a convenient way to
set a breakpoint to rust code, so
`unsafe { std::ptr::write_volatile(0 as *const _, 1u8) }` usually is
what I do).
2. If you manage to get an LLDB prompt, you can set a breakpoint using
`breakpoint set --name foo`, or `breakpoint set --file foo.rs --line 123`.
I don't know how to bring up this prompt reliably, so I often do step 1 to
get it to appear, delete the crashing code, and then set the
breakpoint using the CLI. This is admittedly suboptimal.
3. Click the Debug button in Android Studio, to display the "Select Deployment
Target" window.
4. Make sure the debugger selection is set to "Both". This tends to unset
itself, so make sure.
5. Click "Run", and debug away.

Просмотреть файл

@ -1,6 +0,0 @@
# Exposing Rust to Swift on iOS
* add the new FFI header include to MozillaAppServices.h. Optionally, one can add the new FFI header to the files in the project, but don't add it to the build target.
* add the search path to the HEADER_SEARCH_PATHS in the base.xcconfig for the project to the location of the new FFI header
* update Cargo.toml
* update lib.rs

Двоичные данные
docs/howtos/img/swift-protobuf-build-rule.png

Двоичный файл не отображается.

До

Ширина:  |  Высота:  |  Размер: 1.0 MiB

Двоичные данные
docs/howtos/img/swift-protobuf-framework.png

Двоичный файл не отображается.

До

Ширина:  |  Высота:  |  Размер: 323 KiB

Двоичные данные
docs/howtos/img/swift-protobuf-test-setup.png

Двоичный файл не отображается.

До

Ширина:  |  Высота:  |  Размер: 330 KiB

Двоичные данные
docs/howtos/img/xcode_add_component_1.png Normal file

Двоичный файл не отображается.

После

Ширина:  |  Высота:  |  Размер: 638 KiB

Двоичные данные
docs/howtos/img/xcode_add_component_2.png Normal file

Двоичный файл не отображается.

После

Ширина:  |  Высота:  |  Размер: 787 KiB

Двоичные данные
docs/howtos/img/xcode_add_component_3.png Normal file

Двоичный файл не отображается.

После

Ширина:  |  Высота:  |  Размер: 715 KiB

Двоичные данные
docs/howtos/img/xcode_add_component_4.png Normal file

Двоичный файл не отображается.

После

Ширина:  |  Высота:  |  Размер: 654 KiB

Просмотреть файл

@ -1,539 +0,0 @@
# Using protobuf-encoded data over Rust FFI.
This assumes you already have your FFI mostly set up. If you don't that part
should be covered by another document, which may or may not exist yet (at the time of this writing, it does not).
Most of this is concerned with how to do it the first time as well. If your rust
component already is returning protobuf-encoded data, you probably just need to
follow the examples of the other steps it takes.
## Rust Changes
1. To your main rust crate, add dependencies on the `prost` and `prost-derive` crates.
2. Create a new file named `mylib_msg_types.proto` (well, prefix it with the
actual name of your lib) in your main crate's `src` folder.
Due to annoying details of how the iOS megazord works, the name of the .proto
file must be unique.
1. This file should start with
```
syntax = "proto2";
package mozilla.appservices.mylib.protobuf;
option java_package = "mozilla.appservices.mylib";
option java_outer_classname = "MsgTypes";
option swift_prefix = "MsgTypes_";
option optimize_for = LITE_RUNTIME;
```
The package name is going to determine where the .rs file is output.
2. Fill in your definitions in the rest of the file. See
https://developers.google.com/protocol-buffers/docs/proto for examples.
3. In `tools/protobuf_files.toml`, add your protobuf file definition:
```toml
["mylib_msg_types.proto"]
dir = "../components/mylib/src/"
```
4. Generate your Rust files from the .proto files using:
```console
$ cargo regen-protobufs
```
5. Into your main crate's lib.rs file, add something equivalent to the following:
```rust
pub mod msg_types {
include!("mozilla.appservices.mylib.protobuf.rs");
}
```
This exposes the generated rust file (from the .proto file) as a
rust module.
6. Open your main crates's src/ffi.rs (note: *not* ffi/src/lib.rs! We'll get
there shortly!)
For each type you declare in your .proto file, first decide if you want to
use this as the primary type to represent this data, or if you want to convert
it from a more idiomatic Rust type into the message type when returning.
If it's something that exists solely to return over the FFI, or you may have
a large number of them (or if you need to return them in an array, see the
FAQ question on this) it *may* be best to just use the type from msg_types in your rust code.
We'll what you do in both cases. The parts only relevant if you are converting
between a rust type and the protobuf start with "*(optional unless converting types)*".
Note that if your canonical rust type is defined in another crate, or if it's
something like `Vec<T>`, you will need to use a wrapper. See the FAQ question
on `Vec<T>` about this.
1. *(optional unless converting types)* Define the conversion between the
idiomatic Rust type and the type produced from `msg_types`. This will
likely look something like this:
```rust
impl From<HistoryVisitInfo> for msg_types::HistoryVisitInfo {
fn from(hvi: HistoryVisitInfo) -> Self {
Self {
// convert url::Url to String
url: hvi.url.into_string(),
// Title is already an Option<String>
title: hvi.title,
// Convert Timestamp to i64
timestamp: hvi.title.0 as i64,
// Convert rust enum to i32
visit_type: hvi.visit_type as i32,
}
}
}
```
2. Add a call to:
```rust
ffi_support::implement_into_ffi_by_protobuf!(msg_types::MyType);
```
3. *(optional unless converting types)* Add a call to
```rust
ffi_support::implement_into_ffi_by_delegation!(MyType, msg_types::MyType);
```
If `MyType` is something that you were previously returning via JSON, you need
to remove the call to `implement_into_ffi_by_json!`, and you may also want to
delete `Serialize` from it's `#[derive(...)]` while you're at it, unless you
still need it.
7. In your ffi crate's lib.rs, make the following changes:
1. Any function that conceptually returns a protobuf type must now return
`ffi_support::ByteBuffer` (if it returned via JSON before, this should be
a change of `-> *mut c_char` to `-> ByteBuffer`).
Sometimes you may want to return an *optional* protobuf. Since we are returning the `RustBuffer` struct **by value** in our FFI code, it cannot be "nulled". However this structure has a `data` pointer, which will can get nulled: instead of returning a `Result<T>`, simply return `Result<Option<T>>`. In practice this means [very little changes](https://github.com/mozilla/application-services/blob/2df37a4c8c9d3b9c9159b0f80542303088027618/components/tabs/ffi/src/lib.rs#L85-L95) in Rust code.
We have also outlined in the specific Kotlin/Swift implementations below the small changes to make to your code.
2. You must add a call to
`ffi_support::define_bytebuffer_destructor!(mylib_destroy_bytebuffer)`.
The name you chose for `mylib_destroy_bytebuffer` **must not** collide with the name anybody else uses for this.
## Kotlin Changes
1. Inside your component's build.gradle (e.g.
`components/mything/android/build.gradle`, not the top level one):
1. Add `apply plugin: 'com.google.protobuf'` to the top of the file.
2. Into the `android { ... }` block, add:
```groovy
sourceSets {
main {
proto {
srcDir '../src'
}
}
}
```
3. Add a new top level block:
```groovy
protobuf {
protoc {
artifact = 'com.google.protobuf:protoc:3.0.0'
}
plugins {
javalite {
artifact = 'com.google.protobuf:protoc-gen-javalite:3.0.0'
}
}
generateProtoTasks {
all().each { task ->
task.builtins {
remove java
}
task.plugins {
javalite { }
}
}
}
}
```
4. Add the following to your dependencies:
```groovy
implementation 'com.google.protobuf:protobuf-lite:3.0.0'
implementation project(':native-support')
```
2. In the file where the foreign functions are defined, make sure that the
function returning this type returns a `RustBuffer.ByValue` (`RustBuffer` is
in `mozilla.appservices.support.native`).
Additionally, add a declaration for `mylib_destroy_bytebuffer` (the name must match what was used in the `ffi/src/lib.rs` above). This should look like:
```kotlin
fun mylib_destroy_bytebuffer(v: RustBuffer.ByValue)
```
3. Usage code then looks as follows:
```kotlin
val rustBuffer = rustCall { error ->
MyLibFFI.INSTANCE.call_thing_returning_rustbuffer(...)
}
try {
// Note: if conceptually your function returns Option<ByteBuffer>
// from rust, you should do the following here instead:
//
// val message = rustBuffer.asCodedInputStream()?.let { stream ->
// MsgTypes.SomeMessageData.parseFrom(stream)
// }
val message = MsgTypes.SomeMessageData.parseFrom(
infoBuffer.asCodedInputStream()!!)
// use `message` to produce the higher level type you want to return.
} finally {
LibPlacesFFI.INSTANCE.mylib_destroy_bytebuffer(infoBuffer)
}
```
If you make any changes to the `.proto` file and want regenerate your local kotlin files, you can use:
```
./gradlew generateDebugProto
```
While being in the root directory of `application-services`.
## Swift
1. You need to install `carthage` and `swift-protobuf`, as well as `protobuf`
(if you don't have it already). The following should do the trick if you
use homebrew:
```
brew install carthage swift-protobuf protobuf
```
2. Run `carthage bootstrap` from the root of this repository. This will produce a
(gitignored) Carthage directory in this repository.
3. In Xcode, you need to add `Carthage/Build/iOS/SwiftProtobuf.framework`
to your target's "Linked frameworks and libraries" list.
- You'll need to click "Add Other" for choosing a path to be available.
- This was hard for me to find, so see [this screenshot](./img/swift-protobuf-framework.png)
if you are having trouble locating it.
4. Add `FRAMEWORK_SEARCH_PATHS = "../../../Carthage/Build/iOS"` (Note: you may need a different number of `..`s) to the `base.xcconfig`
5. Add your msg_types.proto file to the xcode project.
- Do not check "copy items as needed".
6. Under "Build Rules", add a new build rule:
- Choose "Source files with names matching:" and enter `*.proto`
- Choose "Using custom script" and paste the following into the textarea.
```
protoc --proto_path=$INPUT_FILE_DIR --swift_out=$DERIVED_FILE_DIR $INPUT_FILE_PATH
```
- Under "Output Files", add `$(DERIVED_FILE_DIR)/$(INPUT_FILE_BASE).pb.swift`
(with no compiler flags).
This should look like [this screenshot](./img/swift-protobuf-build-rule.png)
(which should also help indicate where to find the build rules page) after
you finish.
7. If you have a test target for your framework (and you should, but not all of
our iOS code does), you need to do the following as well:
1. In the test target's Build phases, add SwiftProtobuf.framework to the
"Link Binary with Libraries" phase (As in step 3, you need to "Add
Other", and choose
`<repo-root>/Carthage/Build/iOS/SwiftProtobuf.framework`).
2. Add a new "Copy Files" build phase to the test target. Select the
SwiftProtobuf.framework, and set the destination to Frameworks.
See [this screenshot](./img/swift-protobuf-test-setup.png) for what it should look
like when done.
8. Add a declaration for the rust buffer type to your .h file:
Note that the name must be unique, hence the `MyLib` prefix (use your actual
lib name, and not MyLib, of course).
```c
typedef struct MyLibRustBuffer {
int64_t len;
uint8_t *_Nullable data;
} MyLibRustBuffer;
// Note: this must be the same name you called
// `ffi_support::define_bytebuffer_destructor!` with in
// Rust setup step 8.
void mylib_destroy_bytebuffer(MyLibRustBuffer bb);
```
Then, your functions that return `ffi_support::ByteBuffers` in rust should
return `MyLibRustBuffer` in the version exposed by the header, for example (from places)
```c
PlacesRustBuffer bookmarks_get_by_guid(PlacesConnectionHandle handle,
char const *_Nonnull guid,
PlacesRustError *_Nonnull out_err);
```
9. Add the following somewhere in your swift code: (TODO: Eventually we should
figure out a way to share this)
```swift
extension Data {
init(mylibRustBuffer: MyLibRustBuffer) {
self.init(bytes: mylibRustBuffer.data!, count: Int(mylibRustBuffer.len))
}
}
```
10. Usage code then looks something like:
```swift
let buffer = try MylibError.unwrap { error in
mylib_function_returning_buffer(self.handle, error)
}
// Note: if conceptually your function returns Option<ByteBuffer>
// from rust, you should do the following here:
//
// if buffer.data == nil {
// return nil
// }
defer { mylib_destroy_bytebuffer(buffer) }
let msg = try! MsgTypes_MyMessageType(serializedData: Data(mylibRustBuffer: buffer))
// use msg...
```
Note: If Xcode has trouble locating the types generated from the .proto file,
even though the build works, restarting Xcode may fix the issue.
# Using protobuf to pass data *into* Rust code
Don't pass `ffi_support::ByteBuffer`/`RustBuffer` into rust.
It is a type for going in the other direction.
Instead, you should pass the data and length separately.
The Rust side of this looks something like this:
```rust
#[no_mangle]
pub unsafe extern "C" fn rust_fun_taking_protobuf(
data *const u8,
len: i32,
error: &mut ExternError,
) {
// Or another call_with_blah function as needed
ffi_support::call_with_result(error, || {
// TODO: We should find a way to share some of this boilerplate
assert!(len >= 0, "Bad buffer len: {}", len);
let bytes = if len == 0 {
// This will still fail, but as a bad protobuf format.
&[]
} else {
assert!(!data.is_null(), "Unexpected null data pointer");
std::slice::from_raw_parts(data, len as usize)
};
let my_thing: MyMsgType = prost::Message::decode(bytes)?;
// Do stuff with my_thing...
Ok(())
})
}
```
## Kotlin/Android
There are two ways of passing the data/length pairs on android. You can use
either a `Array<Byte>` or a `Pointer`, which you can get from a "direct"
`java.nio.ByteBuffer`. We recommend the latter, as it avoids an additional copy,
which can be done as follows (using the `toNioDirectBuffer` our kotlin support
library provides):
In Kotlin:
```kotlin
// In the com.sun.jna.Library
fun rust_fun_taking_protobuf(data: Pointer, len: Int, out: RustError.ByReference)
// In some your wrapper (note: `toNioDirectBuffer` is defined by our
// support library)
val (len, nioBuf) = theProtobufType.toNioDirectBuffer()
rustCall { err ->
val ptr = Native.getDirectBufferPointer(nioBuf)
MyLib.INSTANCE.rust_fun_taking_protobuf(ptr, len, err)
}
```
Note that the `toNioDirectBuffer` helper can't return the Pointer directly, as
it is only valid until the NIO buffer is garbage collected, and if the pointer
were returned it would not be reachable.
## Swift
1. Make sure you've done the first 7 steps (up until you need to modify .h
files) of the swift setup for returning protobuf data.
2. The function taking the protobuf should look like this in the header file:
```c
void rust_fun_taking_protobuf(uint8_t const *_Nonnull data,
int32_t len,
MylibRustError *_Nonnull out_err);
```
3. Then, the usage code from Swift would look like:
```swift
var msg = MsgTypes_MyThing()
// populate `msg` with whatever you need to send...
// Note: the `try!` here only fails if you failed to
// populate a required field.
let data = try! msg.serializedData()
let size = Int32(data.count)
try data.withUnsafeBytes { (bytes: UnsafePointer<UInt8>) in
try MyLibError.unwrap { error in
rust_fun_taking_protobuf(bytes, size, error)
}
}
```
# FAQ
### What are the downsides of using types from `msg_types.proto` heavily?
1. It doesn't lead to particularly idiomatic Rust code.
2. We loose the ability to enforce many type invariants that we'd like. For
example, we cannot declare that a field holds a `Url`, and must use a
`String` instead.
### I'd like to expose a function returning a `Vec<T>`.
If T is a type from your mylib_msg_types.proto, then this is fairly easy:
Don't, instead add a new msg_type that contains a repeated T field, and make
that rust function return that.
Then, make so long as the new msg_type has `implement_into_ffi_by_protobuf!` and the ffi function returns a ByteBuffer, things should "Just Work".
---
Unfortunately, if T is merely *convertable* to something from mylib_msg_types.proto,
this adds a bunch of boilerplate.
Say we have the following mylib_msg_types.proto:
```proto
message HistoryVisitInfo {
required string url = 1;
optional string title = 2;
required int64 timestamp = 3;
required int32 visit_type = 4;
}
message HistoryVisitInfos {
repeated HistoryVisitInfo infos = 1;
}
```
in src/ffi.rs, we then need
```rust
// Convert from idiomatic rust HistoryVisitInfo to msg_type HistoryVisitInfo
impl From<HistoryVisitInfo> for msg_types::HistoryVisitInfo {
fn from(hvi: HistoryVisitInfo) -> Self {
Self {
url: hvi.url,
title: hvi.title,
timestamp: hvi.title.0 as i64,
visit_type: hvi.visit_type as i32,
}
}
}
// Declare a type that exists to wrap the vec (see the next question about
// why this is needed)
pub struct HistoryVisitInfos(pub Vec<HistoryVisitInfo>);
// Define the conversion between said wrapper and the protobuf
// HistoryVisitInfos
impl From<HistoryVisitInfos> for msg_types::HistoryVisitInfos {
fn from(hvis: HistoryVisitInfos) -> Self {
Self {
infos: hvis.0
.into_iter()
.map(msg_types::HistoryVisitInfo::from)
.collect()
}
}
}
// generate the IntoFfi for msg_types::HistoryVisitInfos
implement_into_ffi_by_protobuf!(msg_types::HistoryVisitInfos);
// Use it to implement it for HistoryVisitInfos
implement_into_ffi_by_delegation!(HistoryVisitInfos, msg_types::HistoryVisitInfos);
```
Then, in `ffi/src/lib.rs`, where you currently return the Vec, you need to
change it to return wrap that in main_crate::ffi::HistoryVisitInfos, something like
```rust
CONNECTIONS.call_with_result(error, handle, |conn| -> places::Result<_> {
Ok(HistoryVisitInfos(storage::history::get_visit_infos(
conn,
places::Timestamp(start_date.max(0) as u64),
places::Timestamp(end_date.max(0) as u64),
)?))
})
```
### Why is that so painful?
Yep. There are a few reasons for this.
`ffi_support` is the only one who is in a position to decide how a `Vec<T>` is
returned over the FFI. Rust has a rule that either the trait (in this case
`IntoFfi`) or the type (in this case `Vec`) must be implemented in the crate
where the `impl` block happens. This is known as the orphan rule.
Additionally, until rust gains support for
[specialization](https://github.com/rust-lang/rust/issues/31844), we have very
little flexibility with how this works. We can't implement it one way for some
kinds of T's, and another way for others (however, we can, and do, make it
opt-in, but that's unrelated).
This means ffi_support is in the position of deciding how `Vec<T>` goes over the
FFI for all T. At one point, the reasonable choice seemed to be JSON. This is
still used fairly heavily for returning arrays of things, and so until we move
*everything* to use protobufs, we don't really want to take that out.
Unfortunately even we no longer use JSON for this, the conversion between
`Vec<T>` and the ByteBuffer has to happen through an intermediate type, due to
the way protobuf messages work (you can't have a message that's an array, but
you *can* have one that is a single item type which contains a repeated array),
and it isn't clear how to make this work (it can't be an argument to a macro, as
that would violate the orphan rule).
The only thing that would work is if we use the types generated by prost for more
than just returning things over the FFI. e.g. the rust `get_visit_infos()` call would return `HistoryVisitInfos` struct that is generated from a `.proto` file.
#### Could this be worked around by using length-delimited protobuf messages?
Yes, possibly. Looking into this is something we may do in the future.
### Why is the module produced from .proto `msg_types` and not `ffi_types`?
We use `msg_types` and not e.g. `ffi_types`, since in some cases (see the next
FAQ about returning arrays, for example) it can reduce boilerplate a lot to use
these for returning the data to rust code directly (particularly when the rust
API exists almost exclusively to be called from the FFI).
Using a name like `ffi_types`, while possibly intuitive, gives the impression
that these types should not be used outside the FFI, and that it may even be
unsafe to do so.

Просмотреть файл

@ -4,6 +4,9 @@ All names in this project should adhere to the guidelines outlined in this docum
## Rust Code
TL;DR: do what Rust's builtin warnings and clippy lints tell you
(and CI will fail if there are any unresolved warnings or clippy lints).
### Overview
- All variable names, function names, module names, and macros in Rust code should follow typical `snake_case` conventions.