Audits some of our docs to keep them up to date (#4671)

This commit is contained in:
Tarik Eshaq 2021-11-25 11:39:43 -08:00 коммит произвёл GitHub
Родитель 59c049af40
Коммит a876d349f5
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
14 изменённых файлов: 151 добавлений и 206 удалений

Просмотреть файл

@ -1,11 +1,13 @@
# application-services Rust Components
# Application Services Rust Components
Application Services is collection of Rust Components. The components are used to enable Firefox, and related applications to integrate with Firefox accounts, sync and enable experimentation. Each component is built using a core of shared code written in Rust, wrapped with native language bindings for different platforms.
Application Services (a-s) is collection of Rust Components that are used to enable Firefox applications to integrate with Firefox accounts, sync, experimentation, etc. Each component is built using a core of shared code written in Rust, wrapped with native language bindings for different platforms.
## Contact us
To contact us you can:
To contact the Application Services team you can:
- Find us in the chat [#rust-components:mozilla.org](https://chat.mozilla.org/#/room/#rust-components:mozilla.org) ([How to connect](https://wiki.mozilla.org/Matrix#Connect_to_Matrix))
- To report issues or request changes, file a bug in [Bugzilla for Firefox :: Sync](https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=Sync)
- To report issues with sync on **Firefox Desktop**, file a bug in [Bugzilla for Firefox :: Sync](https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=Sync)
- To report issues with our components, file an issue in [the GitHub issue tracker](https://github.com/mozilla/application-services/issues)
The source code is available [on GitHub](https://github.com/mozilla/application-services/).

Просмотреть файл

@ -1,11 +1,11 @@
[application-services Rust Components](README.md)
[Application Services Rust Components](README.md)
- [Contributing](contributing.md)
- [Building](building.md)
- [How to use the local development autopublish flow for Fenix](howtos/locally-published-components-in-fenix.md)
- [How to use the local development flow for Firefox iOS](howtos/locally-published-components-in-ios.md)
- [How to use the local development flow for Focus for iOS](howtos/locally-published-spm-in-ios.md)
- [How to locally build JNA](howtos/locally-building-jna.md)
- [How we test Rust Components](howtos/testing-a-rust-component.md)
- [How to test Rust Components](howtos/testing-a-rust-component.md)
- [How to integration (smoke) test application-services](howtos/smoke-testing-app-services.md)
- [Writing efficient tests](design/test-faster.md)
- [Dependency management](dependency-management.md)
@ -30,6 +30,7 @@
- [Shipping Rust Components as Swift Packages](design/swift-package-manager.md)
- [Rust Component's Strategy](design/components-strategy.md)
- [Metrics - (Glean Telemetry)](design/metrics.md)
- [Rust Version Policy](design/rust-versions.md)
- [How to cut a new release](howtos/cut-a-new-release.md)
- [CI Publishing tools and flow](build-and-publish-pipeline.md)
- [How to upgrade NSS](howtos/upgrading-nss-guide.md)

Просмотреть файл

@ -1,12 +1,12 @@
# Building Application Services
When working on application-services Rust Components it's important to set up your environment for both building the application-services library, but also for Android or iOS as appropriate to test your changes in our clients.
When working on Application Services, it's important to set up your environment for building the Rust code and the Android or iOS code needed by the application.
## First time builds
Building for the first time is more complicated than a typical Rust project.
To build for an end-to-end experience that enables you to test changes in our
client applications like Fenix and Firefox iOS, there are a number of build
To build for an end-to-end experience that enables you to test changes in
client applications like **Firefox for Android (Fenix)** and **Firefox iOS**, there are a number of build
systems required for all the dependencies. The initial setup is likely to take
a number of hours to complete.
@ -18,90 +18,85 @@ a number of hours to complete.
```shell
$ git clone https://github.com/mozilla/application-services # (or use the ssh link)
$ cd application-services
$ git submodule init
$ git submodule update --recursive
$ git submodule update --init --recursive
```
2. Install Rust: install [via rustup](https://www.rust-lang.org/tools/install)
3. Install your system dependencies: - install via the instructions below for [Linux](building.md#linux), [MacOS](building.md#macos) or [Windows](building.md#windows)
4. Check dependencies, environment variables and test
1. Run: `./libs/verify-desktop-environment.sh`
1. Run: `cargo test`
3. Install your system dependencies:
Once you have successfully run `./libs/verify-desktop-environment.sh` and `cargo test` you can move to the **Building for Fenix** and **Building for iOS** sections below to setup your local environment for testing with our client applications.
#### Linux
1. Install the system dependencies required for building NSS
1. Install gyp: `apt install gyp` (required for NSS)
1. Install ninja-build: `apt install ninja-build`
1. Install python3 (at least 3.6): `apt install python3`
1. Install zlib: `apt install zlib1g-dev`
1. Install the system dependencies required for SQLcipher
1. Install tcl: `apt install tclsh` (required for SQLcipher)
#### MacOS
1. Install Xcode: check the [ci config](../.circleci/config.yml) for the correct
version.
1. Install Xcode tools: `xcode-select --install`
1. Install homebrew: [via homebrew](https://brew.sh/) (it's what we use for ci)
1. Install the system dependencies required for building NSS
1. Install ninja: `brew install ninja`
1. Install gyp (via https://github.com/mogemimi/pomdog/wiki/How-to-Install-GYP)
1. Install swift-protobuf: `brew install swift-protobuf`
#### Windows
*Install windows build tools*
> Why [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/about)?
>
> It's currently tricky to get some of these builds working on Windows, primarily due to our use of SQLcipher. By using WSL it is possible to get builds working, but still have them published to your "native" local maven cache so it's available for use by a "native" Android Studio.
#### Linux
1. Install the system dependencies required for building NSS
1. Install gyp: `apt install gyp` (required for NSS)
1. Install ninja-build: [via package for distribution](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages#package-managers)
1. Install python3: [3.6 via python.org](https://docs.python.org/3/using/unix.html)
1. Install zlib: `apt install zlib1g-dev`
1. Install the system dependencies required for SQLcipher
1. Install tcl: `apt install tclsh` (required for SQLcipher)
1. Install [WSL](https://docs.microsoft.com/en-us/windows/wsl/about) (recommended over native tooling)
1. Install unzip: `sudo apt install unzip`
1. Install python3: `sudo apt install python3` *Note: must be python 3.6 or later*
1. Install system build tools: `sudo apt install build-essential`
1. Install zlib: `sudo apt-get install zlib1g-dev`
1. Install tcl: `sudo apt install tcl-dev`
4. Check dependencies and environment variables by running: `./libs/verify-desktop-environment.sh`
> Note that this script might instruct you to set some environment variables, set those by adding them to your
`.zshrc` or `.bashrc` so they are set by default on your terminal
6. Run cargo test: `cargo test`
#### MacOS
1. Install Xcode: check the [ci config](../.circleci/config.yml) for the correct
version.
1. Install Xcode tools: `xcode-select --install`
1. Install homebrew: [via homebrew](https://brew.sh/) (its what we use for ci)
1. Install the system dependencies required for building NSS
1. Install ninja: `brew install ninja`
1. Install gyp (via https://github.com/mogemimi/pomdog/wiki/How-to-Install-GYP)
1. Install swift-protobuf: `brew install swift-protobuf`
#### Windows
*Install windows build tools*
> Why [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/about)?
>
> It's currently tricky to get some of these builds working on Windows, primarily due to our use of SQLcipher. By using WSL it is possible to get builds working, but still have them published to your "native" local maven cache so it's available for use by a "native" Android Studio.
1. Install [WSL](https://docs.microsoft.com/en-us/windows/wsl/about) (recommended over native tooling)
1. Install unzip: `sudo apt install unzip`
1. Install python3: `sudo apt install python3` *Note: must be python 3.6*
1. Install system build tools: `sudo apt install build-essential`
1. Install zlib: `sudo apt-get install zlib1g-dev`
1. Install tcl: `sudo apt install tcl-dev`
Once you have successfully run `./libs/verify-desktop-environment.sh` and `cargo test` you can move to the [**Building for Fenix**](building.md#building-for-fenix) and [**Building for iOS**](building.md#building-for-firefox-ios) sections below to setup your local environment for testing with our client applications.
---
## Building for Fenix
The instructions here assume that you are building for Fenix in order test your changes in Fenix and want to take advantage of the
[Fenix Auto-publication workflow for android-components and application-services](https://github.com/mozilla-mobile/fenix/#auto-publication-workflow-for-android-components-and-application-services)
The following instructions assume that you are building `application-services` for Fenix, and want to take advantage of the
[Fenix Auto-publication workflow for android-components and application-services](howtos/locally-published-components-in-fenix.md).
1. Install Android SDK, JAVA, NDK and set required env vars
1. Clone the [Fenix](https://github.com/mozilla-mobile/fenix/) repository (not in a-s)
1. Clone the [android-components](https://github.com/mozilla-mobile/android-components/) repository (not in a-s)
1. Install [Java **11**] for your system
1. Clone the [Fenix](https://github.com/mozilla-mobile/fenix/) repository (**not** inside the Application Service repository).
1. Clone the [android-components](https://github.com/mozilla-mobile/android-components/) repository (**not** inside the Application Service repository).
1. Install [Java **11**](https://www.oracle.com/java/technologies/downloads/#java11) for your system
1. Set `JAVA_HOME` to point to the JDK 11 installation directory.
1. Download and install [Android Studio](https://developer.android.com/studio/#downloads)
1. Set `ANDROID_SDK_ROOT` and `ANDROID_HOME` to the Android Studio sdk location and add it to your rc file.
1. Download and install [Android Studio](https://developer.android.com/studio/#downloads).
1. Set `ANDROID_SDK_ROOT` and `ANDROID_HOME` to the Android Studio sdk location and add it to your rc file (either `.zshrc` or `.bashrc` depending on the shell you use for your terminal).
1. Configure the required versions of NDK
`Configure menu > System Settings > Android SDK > SDK Tools > NDK > Show Package Details > NDK (Side by side)`
- 21.4.7075529 (required by Fenix; note: a specific NDK version isn't configured, this maps to default [NDK version](https://developer.android.com/studio/projects/install-ndk#default-ndk-per-agp) for the [AGP version](https://github.com/mozilla-mobile/fenix/blob/main/buildSrc/src/main/java/Dependencies.kt#L11))
- 21.3.6528147 (required by a-s, [as configured](https://github.com/mozilla/application-services/blob/main/build.gradle#L30))
1. If you are on Windows using WSL - drop to the section below, Windows setup
for Android (WSL) before proceeding.
1. Check dependencies, environment variables and test
- 21.3.6528147 (required by Application Services, [as configured](https://github.com/mozilla/application-services/blob/main/build.gradle#L30))
1. If you are on Windows using WSL - drop to the section below, [Windows setup
for Android (WSL)](building.md#windows-setup-for-android-via-wsl) before proceeding.
1. Check dependencies, environment variables
1. Run `./libs/verify-android-environment.sh`
2. Follow instructions and rerun until it is successful.
### Windows setup for Android (via WSL)
Note: For non-Ubuntu linux versions, it may be necessary to execute `$ANDROID_HOME/tools/bin/sdkmanager "build-tools;26.0.2" "platform-tools" "platforms;android-26" "tools"`. See also [this gist](https://gist.github.com/fdmnio/fd42caec2e5a7e93e12943376373b7d0) for additional info.
Note: For non-Ubuntu linux versions, it may be necessary to execute `$ANDROID_HOME/tools/bin/sdkmanager "build-tools;26.0.2" "platform-tools" "platforms;android-26" "tools"`. See also [this gist](https://gist.github.com/fdmnio/fd42caec2e5a7e93e12943376373b7d0) for additional information.
#### Configure Maven
Configure maven to use the native windows maven repository - then, when doing ./gradlew install from WSL, it ends up in the Windows maven repo. This means we can do a number of things with Android Studio in "native" windows and have then work correctly with stuff we built in WSL.
Configure maven to use the native windows maven repository - then, when doing `./gradlew install` from WSL, it ends up in the Windows maven repo. This means we can do a number of things with Android Studio in "native" windows and have then work correctly with stuff we built in WSL.
1. Install maven: `sudo apt install maven`
1. Confirm existence of (or create) a `~/.m2` folder
1. In the `~/.m2` create a file called `settings.xml`
1. Add the content below replacing `{username}` with your username:
```
```xml
<settings>
<localRepository>/mnt/c/Users/{username}/.m2/repository</localRepository>
</settings>
@ -115,6 +110,6 @@ Configure maven to use the native windows maven repository - then, when doing ./
1. Run `./libs/verify-ios-environment.sh` to check your setup and environment
variables.
1. Make any corrections recommended by the script and re-run.
1. Follow the guide for [using local a-s builds in iOS](https://github.com/mozilla/application-services/blob/main/docs/howtos/locally-published-components-in-ios.md#using-locally-published-components-in-firefox-for-ios)
1. Follow the guide for [using local a-s builds in iOS](howtos/locally-published-components-in-ios.md)
> Note: The built Xcode project is located at `megazords/ios/MozillaAppServices.xcodeproj`.

Просмотреть файл

@ -1,30 +1,26 @@
# Contributing to Application Services
Anyone is welcome to help with the Application Services project. Feel free to get in touch with other community members on Matrix, the mailing list or through issues here on GitHub.
- Matrix: [#sync:mozilla.org](https://chat.mozilla.org/#/room/#sync:mozilla.org)
- Mailing list: <https://mail.mozilla.org/listinfo/sync-dev>
- and of course, [the issues list](https://github.com/mozilla/application-services/issues)
Anyone is welcome to help with the [Application Services](index.md) project. Feel free to get in touch with [other community members on Matrix or through issues on GitHub.](./index.md#contact-us)
Participation in this project is governed by the
[Mozilla Community Participation Guidelines](https://www.mozilla.org/en-US/about/governance/policies/participation/).
## Bug Reports ##
You can file issues here on GitHub. Please try to include as much information as you can and under what conditions
You can file issues on [GitHub](https://github.com/mozilla/application-services/issues). Please try to include as much information as you can and under what conditions
you saw the issue.
## Building the project ##
Build instructions are available [here](building.md). Do not hesitate to let us know which pain-points you had with setting up your environment!
Build instructions are available in the [`building`](building.md) page. Please let us know if you encounter any pain-points setting up your environment.
## Finding issues ##
Below are a few different queries you can use to find appropriate issues to work on. Feel free to reach out if you need any additional clarification before picking up an issue.
Below are a few different queries you can use to find appropriate issues to work on. Feel free to reach out if you need any additional clarification before picking up an issue.
- **[good first issues](https://github.com/mozilla/application-services/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue)** - If you are a new contributor, search for issues labeled `good-first-issue`
- **[good second issues](https://github.com/mozilla/application-services/labels/good-second-issue)** Once you've got that first PR approved and you are looking for something a little more challenging, we are keeping a list of next-level issues. Search for the `good-second-issue` label.
- **[papercuts](https://github.com/mozilla/application-services/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+%22Epic%3A+papercuts%22+)** A collection of smaller sized issues that may be a bit more advanced than a first or second issue.
- **[good second issues](https://github.com/mozilla/application-services/labels/good-second-issue)** - Once you've got that first PR approved and you are looking for something a little more challenging, we are keeping a list of next-level issues. Search for the `good-second-issue` label.
- **[papercuts](https://github.com/mozilla/application-services/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+%22Epic%3A+papercuts%22+)** - A collection of smaller sized issues that may be a bit more advanced than a first or second issue.
- **[important, but not urgent](https://github.com/mozilla/application-services/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+%22Epic%3A+important+not+urgent%22)** - For more advanced contributors, we have a collection of issues that we consider important and would like to resolve sooner, but work isn't currently prioritized by the core team.
@ -36,19 +32,18 @@ Before submitting a PR:
- Your patch should include new tests that cover your changes, or be accompanied by explanation for why it doesn't need any. It is your and your reviewer's responsibility to ensure your patch includes adequate tests.
- Consult the [testing guide](./howtos/testing-a-rust-component.md) for some tips on writing effective tests.
- Your code should pass all the automated tests before you submit your PR for review.
- Before pushing your changes, run `./automation/tests.py changes`. This will calculate which components were changed and run test suites against them, as well as linters and formatters. Because it runs a limited set of tests, it should execute in a fairly reasonable amount of time.
- After you open a PR, our CI system will run a full test suite. It's possible that this step will result in errors not caught with changes mode, so make sure to check the results.
- "Work in progress" pull requests are welcome, but should be clearly labeled as such and should not be merged until all tests pass and the code has been reviewed.
- You can label pull requests as "Work in progress" by using the Github PR UI to indicate this PR is a draft ([learn more about draft PRs](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests#draft-pull-requests)).
- Run `cargo fmt` to ensure your Rust code is correctly formatted. You should run this command after running tests and before pushing changes so that any fixes for failed tests are included.
- If you have modified any Swift code, also run `swiftformat --swiftversion 4` on the modified code.
- Your patch should include a changelog entry in [CHANGES_UNRELEASED.md](../CHANGES_UNRELEASED.md) or an explanation of why
it does not need one. Any breaking changes to Swift or Kotlin binding APIs should be noted explicitly
- Before pushing your changes, run `./automation/tests.py changes`. The script will calculate which components were changed and run test suites, linters and formatters against those components. Because the script runs a limited set of tests, the script should execute in a fairly reasonable amount of time.
- If you have modified any Swift code, also run `swiftformat --swiftversion 5` on the modified code.
- Your patch should include a changelog entry in [CHANGES_UNRELEASED.md](https://github.com/mozilla/application-services/blob/main/CHANGES_UNRELEASED.md) or an explanation of why
it does not need one. Any breaking changes to Swift or Kotlin binding APIs should be noted explicitly.
- If your patch adds new dependencies, they must follow our [dependency management guidelines](./dependency-management.md).
Please include a summary of the due dilligence applied in selecting new dependencies.
Please include a summary of the due diligence applied in selecting new dependencies.
- After you open a PR, our Continuous Integration system will run a full test suite. It's possible that this step will result in errors not caught with the script so make sure to check the results.
- "Work in progress" pull requests are welcome, but should be clearly labeled as such and should not be merged until all tests pass and the code has been reviewed.
- You can label pull requests as "Work in progress" by using the Github PR UI to indicate this PR is a draft ([learn more about draft PRs](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-requests#draft-pull-requests)).
When submitting a PR:
- You agree to license your code under the project's open source license ([MPL 2.0](/LICENSE)).
- You agree to license your code under the project's open source license ([MPL 2.0](https://github.com/mozilla/application-services/blob/main/LICENSE)).
- Base your branch off the current `main` branch.
- Add both your code and new tests if relevant.
- Please do not include merge commits in pull requests; include only commits with the new relevant code.

Просмотреть файл

@ -1,6 +1,6 @@
# Dependency Management Guidelines
This repo uses third-party code from a variety of sources, so we need to be mindful
This repository uses third-party code from a variety of sources, so we need to be mindful
of how these dependencies will affect our consumers. Considerations include:
* General code quality.
@ -14,7 +14,7 @@ guidelines we've developed so far.
## Rust Code
Unlike [Firefox](https://firefox-source-docs.mozilla.org/build/buildsystem/rust.html),
we do not vendor third-party source code directly into the repo. Instead we rely on
we do not vendor third-party source code directly into the repository. Instead we rely on
`Cargo.lock` and its hash validation to ensure that each build uses an identical copy
of all third-party crates. These are the measures we use for ongoing maintence of our
existing dependencies:
@ -22,11 +22,10 @@ existing dependencies:
* Check `Cargo.lock` into the repository.
* Generate built artifacts using the `--locked` flag to `cargo build`, as an additional
assurance that the existing `Cargo.lock` will be respected.
* TODO: how to actually make this happen via rust-android-gradle plugin?
* Regularly run [cargo-audit](https://github.com/RustSec/cargo-audit) in CI to alert us to
security problems in our dependencies.
* It runs on every PR, and once per hour as a scheduled job with failures reported to slack.
* Use [a home-grown tool](../tools/dependency_summary.py) to generate a summary of dependency licenses
* It runs on every PR, and once per hour on the `main` branch
* Use [a home-grown tool](https://github.com/mozilla/application-services/blob/main/tools/dependency_summary.py) to generate a summary of dependency licenses
and to check them for compatibility with MPL-2.0.
* Check these summaries into the repository and have CI alert on unexpected changes,
to guard against pulling in new versions of a dependency under a different license.
@ -50,7 +49,7 @@ We try to balance this responsibility against the many benefits of using existin
but should still be given some thought.
* There is still the potential for supply-chain compromise with dev dependencies!
* As part of the PR that introduces the new dependency:
* Regenerate dependency summary files using the [regenerate_dependency_summaries.sh](../tools/regenerate_dependency_summaries.sh).
* Regenerate dependency summary files using the [regenerate_dependency_summaries.sh](https://github.com/mozilla/application-services/blob/main/tools/regenerate_dependency_summaries.sh).
* Explicitly describe your consideration of the above points.
Updating to new versions of existing dependencies is a normal part of software development

Просмотреть файл

@ -5,4 +5,5 @@
* [Shipping Rust Components as Swift Packages](swift-package-manager.md) - High level design of how use the Swift Package Manager to distribute our Rust components to iOS
* [Sync overview](sync-overview.md) - High level overview of how Firefox sync works
* [Rust Component's Strategy](components-strategy.md) - High level description of our Rust components strategy
- [Metrics - (Glean Telemetry)](metrics.md)
* [Metrics - (Glean Telemetry)](metrics.md)
* [Rust Version Policy](rust-versions.md)

Просмотреть файл

Просмотреть файл

@ -5,7 +5,7 @@
We'd like to keep `cargo test`, `cargo build`, `cargo check`, ... reasonably
fast, and we'd *really* like to keep them fast if you pass `-p` for a specific
project. Unfortunately, there are a few ways this can become unexpectedly slow.
The easiest of these for us to combat at the moment is probably unfortunate
The easiest of these problems for us to combat at the moment is the unfortunate
placement of dev-dependencies in our build graph.
If you perform a `cargo test -p foo`, all dev-dependencies of `foo` must be
@ -17,9 +17,7 @@ dependencies it needs for those tests, instead of waiting for your benchmark
suite, or the arg-parser your examples use, or etc.
Unfortunately, all cargo knows is that these are `dev-dependencies`, and not
which targets actually use them. (Aside: Now that per-target feature selection
is stable we actually could fix this, however it would be extremely tedious, far
worse than the approach outlined in this document).
which targets actually use them.
Additionally, unqualified invocations of cargo (that is, without `-p`) might
have an even worse time if we aren't careful. If I run, `cargo test`, cargo
@ -28,16 +26,15 @@ dependencies, if `places` depends on `fxa-client`, all of `fxa-clients`
dev-dependencies must be compiled, ready, and linked in at least to the `lib`
target before we can even think about starting on `places`.
This should all sound somewhat obvious, and I guess it is. We have not been
careful about what shape the dependency graph ends up as when example code is
We have not been careful about what shape the dependency graph ends up as when example code is
taken into consideration (as it is by cargo during certain builds), and as a
result, we have this problem. The problem is that this isn't really a problem we
result, we have this problem. Which isn't really a problem we
want to fix: Example code can and should depend on several different components,
and use them together in interesting ways.
So, because we don't really want to change the things our examples do, or make
So, because we don't want to change what our examples do, or make
major architectural changes of the non-test code for something like this, we
need to do something.
need to do something else.
## The Solution
@ -54,9 +51,7 @@ crates so that:
caches.
4. ...
There's no firm rule for this, but the following guidelines seem easy to follow:
Basically, some rules of thumb for when / when not to do this:
Some rules of thumb for when / when not to do this:
- All rust examples should be put in `examples/*`.
@ -75,7 +70,7 @@ To be clear, this is way more important for benchmarks (which always compile as
release and have a costly link phase).
Anyway, say you have a directory structure like the following:
Say you have a directory structure like the following:
```
mycrate

Просмотреть файл

@ -1,6 +1,6 @@
# Adding a new component to Application Services
Each component in the Application Services repo has three parts (the Rust code,
Each component in the Application Services repository has three parts (the Rust code,
the Kotlin wrapper, and the Swift wrapper) so there are quite a few moving
parts involved in adding a new component. This is a rapid-fire list of all
the things you'll need to do if adding a new component from scratch.
@ -20,14 +20,14 @@ Use [UniFFI](https://mozilla.github.io/uniffi-rs/) to define how your crate's
API will get exposed to foreign-language bindings. By convention, put the interface
definition file at `./components/<your_crate_name>/<your_crate_name>.udl`. Use
the `builtin-bindgen` feature of UniFFI to simplify the build process, by
putting this in your `Cargo.toml`:
putting the following in your `Cargo.toml`:
```
[build-dependencies]
uniffi_build = { version = "<latest version here>", features=["builtin-bindgen"] }
```
Include your new crate in the application-services workspace, by adding
Include your new crate in the `application-services` workspace, by adding
it to the `members` and `default-members` lists in the `Cargo.toml` at
the root of the repository.
@ -36,8 +36,8 @@ In order to be published to consumers, your crate must be included in the
* For Android, add it as a dependency in `./megazords/full/Cargo.toml` and
add a `pub use <your_crate_name>` to `./megazords/full/src/lib.rs`.
* For iOS, add it as a dependency in `./megazords/ios/rust/Cargo.toml` and
add a `pub use <your_crate_name>` to `./megazords/ios/rust/src/lib.rs`.
* For iOS, add it as a dependency in `./megazords/ios-rust/rust/Cargo.toml` and
add a `pub use <your_crate_name>` to `./megazords/ios-rust/src/lib.rs`.
Run `cargo check -p <your_crate_name>` in the repository root to confirm that
things are configured properly. This will also have the side-effect of updating
@ -57,7 +57,7 @@ your own component's directory, and edit it to replace the references to
Create a file `./components/<your_crate_name>/uniffi.toml` with the
following contents:
```
```toml
[bindings.kotlin]
package_name = "mozilla.appservices.<your_crate_name>"
cdylib_name = "megazord"
@ -66,7 +66,7 @@ cdylib_name = "megazord"
Create a file `./components/<your_crate_name>/android/src/main/AndroidManifest.xml`
with the following contents:
```
```xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="org.mozilla.appservices.<your_crate_name>" />
```

Просмотреть файл

@ -1,9 +1,7 @@
# Guide to Building a Syncable Rust Component
This is a guide to creating a new Syncable Rust Component like many of the
components in this repo. If you are looking for information how to build (ie,
compile, etc) the existing components, you are looking for
[our build documentation](https://github.com/mozilla/application-services/blob/main/docs/building.md)
> This is a guide to creating a new Syncable Rust Component like many of the components in this repo. If you are looking for information how to build (ie,compile, etc) the existing components, you are looking for [our build documentation](../building.md)
Welcome!
@ -36,13 +34,13 @@ think it should be used.]
The "Store" is the entry-point for the consuming application - it provides the
core functionality exposed by the component and manages your databases and other
singletons. The responsibilities of this will include things like creating the
singletons. The responsibilities of the "Store" will include things like creating the
DB if it doesn't exist, doing schema upgrades etc.
The functionality exposed by the "Store" will depend on the complexity of the
API being exposed. For example, for `webext-storage`, where there are only a
handful of simple public functions, it just directly exposes all the
functionalty of the component. However, for Places, which has a much more
functionality of the component. However, for Places, which has a much more
complex API, the (logical) Store instead supplies "Connection" instances which
expose the actual functionality.
@ -62,7 +60,7 @@ We typically have a "DB" abstraction which manages the database itself - the
logic for handling schema upgrades etc and enforcing the "only 1 writer" rule
is done by this.
However, this is just a convenience - the DB abstrations aren't really passed
However, this is just a convenience - the DB abstractions aren't really passed
around - we just pass raw connections (or transactions) around. For example, if
there's a utility function that reads from the DB, it will just have a Rusqlite
connection passed. (Again, older components don't really do this well, but
@ -150,12 +148,12 @@ data itself - often in a `meta` table.
All logic for knowing which records need to be sync must be part of the
application logic, and will often be implemented using `triggers`. It's quite
common for components to use a "change counter" strategy, which can be
summaried as:
summarized as:
* Every table which defines the "top level" items being synced will have a
column called something like 'sync_change_counter' - the app will probably
track this counter manually instead of using a trigger, because sync itself
will need different behaviour when it updates the records.
will need different behavior when it updates the records.
* At sync time, items with a non-zero change counter are candidates for syncing.
@ -221,4 +219,4 @@ Here are some earlier blog posts on the topic which might be helpful:
* [Building and Deploying a Rust library on iOS](https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-06-rust-on-ios.html)
* [Blog post re: lessons in binding to Rust code from iOS](https://discourse.mozilla.org/t/dear-diary-turns-out-x-platform-is-hard/25348)
The above are likely to be superceded by uniffi docs, but for now, good luck!
The above are likely to be superseded by uniffi docs, but for now, good luck!

Просмотреть файл

@ -1,6 +1,6 @@
# Using locally-published components in Fenix
# Using locally published components in Fenix
It's often important to test work-in-progress changes to this repo against a real-world
It's often important to test work-in-progress changes to Application Services components against a real-world
consumer project. The most reliable method of performing such testing is to publish your
components to a local Maven repository, and adjust the consuming project to install them
from there.
@ -25,19 +25,18 @@ in their build. The workflow is:
1. Build the consuming project following its usual build procedure, e.g. via `./gradlew assembleDebug` or `./gradlew
test`.
If all goes well, this should automatically build your checkout of application-servies, publish it
to a local maven repository, and configure the consuming project to install from there instead of
If all goes well, this should automatically build your checkout of `application-services`, publish it
to a local maven repository, and configure the consuming project to install it from there instead of
from our published releases.
## Using a manual workflow
Note: This is a bit tedious, and you should first try the auto-publishing workflow described
above. But if the auto-publishing workflow bitrots then it's important to know how to do it
by hand. Since most consuming apps get their copy of application-services via a dependency
on android-components, this procedure involves three separate repos:
above. But if the auto-publishing workflow fails then it's important to know how to do the publishing process manually. Since most consuming apps get their copy of `application-services` via a dependency
on `android-components`, this procedure involves three separate repos:
1. Inside the `application-services` repository root:
1. In [`.buildconfig-android.yml`](app-services-yaml), change
1. In [`.buildconfig-android.yml`](https://github.com/mozilla/application-services/blob/main/.buildconfig-android.yml), change
`libraryVersion` to end in `-TESTING$N` <sup><a href="#note1">1</a></sup>,
where `$N` is some number that you haven't used for this before.
@ -50,18 +49,18 @@ on android-components, this procedure involves three separate repos:
3. Run `./gradlew publishToMavenLocal`. This may take between 5 and 10 minutes.
2. Inside the `android-components` repository root:
1. In [`.buildconfig.yml`](android-components-yaml), change
1. In [`.buildconfig.yml`](https://github.com/mozilla-mobile/android-components/blob/main/.buildconfig.yml), change
`componentsVersion` to end in `-TESTING$N` <sup><a href="#note1">1</a></sup>,
where `$N` is some number that you haven't used for this before.
Example: `componentsVersion: 0.51.0-TESTING3`
2. Inside [`buildSrc/src/main/java/Dependencies.kt`](android-components-deps),
2. Inside [`buildSrc/src/main/java/Dependencies.kt`](https://github.com/mozilla-mobile/android-components/blob/main/buildSrc/src/main/java/Dependencies.kt),
change `mozilla_appservices` to reference the `libraryVersion` you
published in step 1 part 1.
Example: `const val mozilla_appservices = "0.27.0-TESTING3"`
3. Inside [`build.gradle`](android-components-build-gradle), add
3. Inside [`build.gradle`](https://github.com/mozilla-mobile/android-components/blob/main/build.gradle), add
`mavenLocal()` inside `allprojects { repositories { <here> } }`.
4. Inside the android-components `local.properties` file, ensure
@ -70,34 +69,32 @@ on android-components, this procedure involves three separate repos:
5. Run `./gradlew publishToMavenLocal`.
3. Inside the consuming project repository root:
1. Inside [`build.gradle`](fenix-build-gradle-1), add
1. Inside [`build.gradle`](https://github.com/mozilla-mobile/fenix/blob/main/build.gradle), add
`mavenLocal()` inside `allprojects { repositories { <here> } }`.
2. Ensure that `local.properties` does not contain any configuration to
related to auto-publishing the application-services repo.
3. Inside [`buildSrc/src/main/java/Dependencies.kt`](fenix-deps), change the
version numbers for android-components and/or application-services to
3. Inside [`buildSrc/src/main/java/AndroidComponents.kt`](https://github.com/mozilla-mobile/fenix/blob/main/buildSrc/src/main/java/AndroidComponents.kt), change the
version numbers for android-components to
match the new versions you defined above.
Example: `const val mozilla_android_components = "0.51.0-TESTING3"`
Example: `const val mozilla_appservices = "0.27.0-TESTING3"`
Example: `const val VERSION = "0.51.0-TESTING3"`
You should now be able to build and run the consuming application (assuming you could
do so before all this).
### Caveats
1. This assumes you have followed the [android/rust build setup](./setup-android-build-environment.md)
1. This assumes you have followed the [build instructions for Fenix](../building.md#building-for-fenix)
2. Make sure you're fully up to date in all repos, unless you know you need to
not be.
3. This omits the steps if changes needed because, e.g. application-services
made a breaking change to an API used in android-components. These should be
3. This omits the steps if changes needed because, e.g. `application-services`
made a breaking change to an API used in `android-components`. These should be
understandable to fix, you usually should be able to find a PR with the fixes
somewhere in the android-component's list of pending PRs (or, failing that, a
description of what to do in the application-services changelog).
4. Ask in #sync if you get stuck.
4. [Contact us](../README.md#contact-us) if you get stuck.
## Adding support for the auto-publish workflow
@ -106,14 +103,14 @@ If you had to use the manual workflow above and found it incredibly tedious, you
try adding support for the auto-publish workflow to the consuming project! The details will differ
depending on the specifics of the project's build setup, but at a high level you will need to:
1. In your [settings.gradle](fenix-settings), locate (or add) the code for parsing the `local.properties` file,
1. In your [settings.gradle](https://github.com/mozilla-mobile/fenix/blob/main/settings.gradle), locate (or add) the code for parsing the `local.properties` file,
and add support for loading a directory path from the property `autoPublish.application-services.dir`.
If this property is present, spawn a subprocess to run `./gradlew autoPublishForLocalDevelopment`
in the specified directory. This automates step (1) of the manual workflow above, publishing your
changes to application-services into a local maven repository under a unique version number.
changes to `application-services` into a local maven repository under a unique version number.
1. In your [build.gradle]([fenix-build-gradle-1]), if the `autoPublish.application-services.dir` property
1. In your [build.gradle](https://github.com/mozilla-mobile/fenix/blob/main/build.gradle), if the `autoPublish.application-services.dir` property
is present, have each project apply the build script from `./build-scripts/substitute-local-appservices.gradle`
in the specified directory.
@ -159,12 +156,3 @@ avoid it. Additionally, while the `$N` we have used in our running example has
matched (e.g. all of the identifiers ended in `-TESTING3`, this is not required,
so long as you match everything up correctly at the end. This can be tricky, so
I always try to use the same number).
[app-services-yaml]: https://github.com/mozilla/application-services/blob/594f4e3f6c190bc5a6732f64afc573c09020038a/.buildconfig-android.yml#L1
[android-components-yaml]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/.buildconfig.yml#L1
[android-components-deps]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/buildSrc/src/main/java/Dependencies.kt#L37
[android-components-build-gradle]: https://github.com/mozilla-mobile/android-components/blob/b98206cf8de818499bdc87c00de942a41f8aa2fb/build.gradle#L28
[fenix-build-gradle-1]: https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/build.gradle#L26
[fenix-deps]:
https://github.com/mozilla-mobile/fenix/blob/f897c2e295cd1b97d4024c7a9cb45dceb7a2fa89/buildSrc/src/main/java/Dependencies.kt#L28
[fenix-settings]: https://github.com/mozilla-mobile/fenix/blob/0d398f7d44f877a61cd243ee9fac587a9d5c0a1f/settings.gradle#L31

Просмотреть файл

@ -1,7 +1,7 @@
# Smoke testing Application Services against end-user apps
This is a great way of finding integration bugs with application services.
It can be done manually using substitution scripts, but we also have scripts that will do all of these for you.
This is a great way of finding integration bugs with `application-services`.
The testing can be done manually using substitution scripts, but we also have scripts that will do the smoke-testing for you.
## Firefox iOS
@ -12,10 +12,10 @@ Add the `-h` argument to discover all of the script's exciting options!
## Android Components
The `automation/smoke-test-android-components.py` script will clone (or use a local version) of
android-components and run a subset of its tests against the current application-services worktree.
It tries to only run tests that might be relevant to application-services functionality.
android-components and run a subset of its tests against the current `application-services` worktree.
It tries to only run tests that might be relevant to `application-services` functionality.
## Fenix
The `automation/smoke-test-fenix.py` script will clone (or use a local version) of Fenix and
run tests against the current application-services worktree.
run tests against the current `application-services` worktree.

Просмотреть файл

@ -1,20 +1,20 @@
# Guide to Testing a Rust Component
This document gives a high-level overview of how we test components in application-services.
This document gives a high-level overview of how we test components in `application-services`.
It will be useful to you if you're adding a new component, or working on increasing the test
coverage of an existing component.
If you are only interested in running the existing test suite, please consult the
[contributor docs](../contributing.md) and the [tests.py](../../automation/tests.py) script.
[contributor docs](../contributing.md) and the [tests.py](https://github.com/mozilla/application-services/blob/main/automation/tests.py) script.
## Unit and Functional Tests
### Rust code
Since the core implementation of our components lives in rust, so does the core of our testing strategy.
Since the core implementations of our components live in rust, so does the core of our testing strategy.
Each rust component should be accompanied by a suite of unittests, following the [guidelines for writing
Each rust component should be accompanied by a suite of unit tests, following the [guidelines for writing
tests](https://doc.rust-lang.org/book/ch11-00-testing.html) from the [Rust
Book](https://doc.rust-lang.org/book/title-page.html).
Some additional tips:
@ -22,34 +22,27 @@ Some additional tips:
* Where possible, it's better use use the Rust typesystem to make bugs impossible than to write
tests to assert that they don't occur in practice. But given that the ultimate consumers of our
code are not in Rust, that's sometimes not possible. The best idiomatic Rust API for a feature
is not necessarily the best API for consuming it over an FFI boundary!
is not necessarily the best API for consuming it over an FFI boundary.
* Rust's builtin assertion macros are pretty spartan; we use the [more_asserts](https://crates.io/crates/more_asserts)
* Rust's builtin assertion macros are sparse; we use the [more_asserts](https://crates.io/crates/more_asserts)
for some additional helpers.
* Rust's strict typing can make test mocks difficult. If there's something you need to mock out in tests,
make it a Trait and use the [mockiato](https://crates.io/crates/mockiato) crate to mock it out.
make it a Trait and use the [mockiato](https://crates.io/crates/mockiato) crate to mock it.
The Rust tests for a component should be runnable via `cargo test`.
### FFI Layer code
We currently do not test the FFI-layer Rust code for our components, since it's generally a very thin
wrapper around the underlying (and in theory well-tested!) Rust component code. If you find yourself
adding a particularly complex bit of code in an FFI-layer crate, add unittests in the same style as
for other Rust code.
(Editor's note: I remain hopeful that one day we'll autogenerate most of the FFI-layer code, and in
such a world we don't need to invest in tests for it.)
We are currently using [`uniffi`](converting-a-component-to-uniffi.md) to generate most ((and soon all!) of our FFI code and thus the FFI code itself does not need to be extensively tested.
### Kotlin code
The Kotlin wrapper code for a component should have its own test suite, which should follow the general guidelines for
[testing Android code in Mozilla projects](https://github.com/mozilla-mobile/shared-docs/blob/master/android/testing.md#jvm-testing).
[testing Android code in Mozilla projects](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#jvm-testing).
In practice that means we use
[JUnit](https://github.com/mozilla-mobile/shared-docs/blob/master/android/testing.md#junit-testing-framework)
[JUnit](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#junit-testing-framework)
as the test framework and
[Robolectric](https://github.com/mozilla-mobile/shared-docs/blob/master/android/testing.md#robolectric-android-api-shadows)
[Robolectric](https://github.com/mozilla-mobile/shared-docs/blob/main/android/testing.md#robolectric-android-api-shadows)
to provide implementations of Android-specific APIs.
The Kotlin tests for a component should be runnable via `./gradlew <component>:test`.
@ -65,12 +58,7 @@ tests need to ensure that an appropriate version of JNA and of the compiled Rust
their library search path at runtime. Our `build.gradle` files contain a collection of hackery that ensures
this, which should be copied into any new components.
(Editor's note: I remain hopeful that one day we'll autogenerate most of the Kotlin binding code, and in
such a world we don't need to invest in tests for it.)
XXX TODO: talk about proguard? I don't really understand it...
XXX TODO: any additional tips here, such as mocking out storage etc?
The majority of our Kotlin bindings are autogenerated using [`uniffi`](converting-a-component-to-uniffi.md) and do not need extensive testing.
### Swift code
@ -88,16 +76,13 @@ and should not repeat tests for functionality that is already well tested at the
But given that the Swift bindings involve a non-trivial amount of hand-written boilerplate code,
it's important to exercise that code throughly.
(Editor's note: I remain hopeful that one day we'll autogenerate most of the Swift binding code, and in
such a world we don't need to invest in tests for it.)
XXX TODO: any additional tips here, such as mocking out storage etc?
The majority of our Swift bindings are autogenerated using [`uniffi`](converting-a-component-to-uniffi.md) and do not need extensive testing.
## Integration tests
### End-to-end Sync Tests
The [`testing/sync-test`](../../testing/sync-test) directory contains a test harness for running sync-related
The [`testing/sync-test`](https://github.com/mozilla/application-services/tree/main/testing/sync-test) directory contains a test harness for running sync-related
Rust components against a live Firefox Sync infrastructure, so that we can verifying the functionality
end-to-end.
@ -109,12 +94,12 @@ Each component that implements a sync engine should have a corresponding suite o
### Android Components Test Suite
It's important that changes in application-services are tested against upstream consumer code in the
It's important that changes in `application-services` are tested against upstream consumer code in the
[android-components](https://github.com/mozilla-mobile/android-components/) repo. This is currently
a manual process involving:
* Configuring your local checkout of android-components to [use your local application-services
build](./working-with-reference-browser.md).
build](./locally-published-components-in-fenix.md).
* Running the android-components test suite via `./gradle test`.
* Manually building and running the android-components sample apps to verify that they're still working.
@ -122,17 +107,7 @@ Ideally some or all of this would be automated and run in CI, but we have not ye
## Test Coverage
Lamentably, we do not measure or report on code test coverage.
See [this github issue](https://github.com/mozilla/application-services/issues/1745) for some early explorations.
The rust ecosystem for code coverage is still maturing, with [cargo-tarpaulin](https://github.com/xd009642/tarpaulin)
appearing to be a promising candidate. However, such tools will only report code that is exercised by the rust
unittests, not code that is exercised by the Kotlin or Swift tests or the end-to-end integration test suite.
For code coverage to be useful to us, we need to either:
* Commit to ensuring high coverage via rust-level tests alone, or
* Figure out how to measure it for code being driven by non-rust test suites.
We currently have code coverage reporting on Github using [codecov](https://github.com/mozilla/application-services/blob/main/codecov.yml). However, our code coverage does not tell us how much more coverage is caused by our consumers' tests.
## Ideas for Improvement

Просмотреть файл

@ -13,9 +13,7 @@ TL;DR: do what Rust's builtin warnings and clippy lints tell you
- All Rust types, traits, structs, and enum variants must follow `UpperCamelCase`.
- Static and constant variables should be written in `SCREAMING_SNAKE_CASE`.
- The functions of all component `ffi/src/lib.rs` files should also follow `snake_case`, but with an additional prefix based on the library of that function.
- Static and constant variables should be written in `SCREAMING_SNAKE_CASE`. s
For more in-depth Rust conventions, see the [Rust Style Guide](https://doc.rust-lang.org/1.0.0/style/style/naming/README.html).
@ -34,8 +32,6 @@ const COMMON_SQL
- All other uses are `lowerCamelCase`.
- Concerning this project, when dealing with Swift code, use Rust convention for FFI binding files (e.g. `RustFxAFFI.h`).
For more in-depth Swift conventions, check out the [Swift API Design Guidelines](https://swift.org/documentation/api-design-guidelines/).
### Examples: