Allow our a-s documentation to be published via mdbook

- creates the infra required for mdbook (as stolen from glean)
 - removes unfinished, unmaintained sync-storage-handbook and synconomicon
 - includes a build script for mdbook (tools/build-book.sh - modified from glean)
 - adds a SUMMARY.md file - which provides the mdbook TOC
This commit is contained in:
Janet Dragojevic 2021-04-02 11:39:37 -04:00
Родитель f3f0cf6e33
Коммит 803c2e2833
31 изменённых файлов: 791 добавлений и 297 удалений

4
.gitignore поставляемый
Просмотреть файл

@ -43,3 +43,7 @@ automation/symbols-generation/bin/
__pycache__
*.py[cod]
.Python
# Generated mdbook documentation files
build/docs/*
docs/book

Просмотреть файл

@ -1,41 +1,14 @@
## Firefox Application Services Docs
# application-services Rust Components
This directory is the documentation root for all the products managed by the
Firefox Application Services Team.
Application Services (a-s) is collection of Rust Components that are used to enable Firefox applications to integrate with Firefox accounts, sync, experimentation, etc. Each component is built using a core of shared code written in Rust, wrapped with native language bindings for different platforms.
The [./product-portal/](product-portal) directory contains the source docs for
the consumer-facing product portal website, which can be viewed here:
To contact us you can:
- Find us in the chat [#rust-components:mozilla.org](https://chat.mozilla.org/#/room/#rust-components:mozilla.org) ([How to connect](https://wiki.mozilla.org/Matrix#Connect_to_Matrix))
- To report issues or request changes, file a bug in [Bugzilla for Firefox :: Sync](https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=Sync)
* https://mozilla.github.io/application-services/
Everything else in this directory is contributor-facing documentation to help
you work on app-services projects. We have:
The source code is available [on GitHub](https://github.com/mozilla/application-services/).
* A high-level guide to [Contributing to Application Services](./contributing.md)
* A description of the [metrics](./metrics/README.md) gathered by each component
(via the [glean](https://mozilla.github.io/glean/) framework).
* The [Synconomicon](https://mozilla.github.io/application-services/synconomicon/), a deep dive into the internals of sync and storage for Firefox Applications.
* The [Sync and Storage Handbook](https://mozilla.github.io/application-services/sync-storage-handbook/index.html), which provides a higher-level view of our sync and storage components.
* Docs for various infrastructure pieces:
* Our [Dependency Management Policies](./dependency-management.md)
* Our [Build and Publish Pipeline](./build-and-publish-pipeline.md)
* Architectural design docs:
* How [megazording](./design/megazords.md) works, and why we do it.
* The motivation and design of the [sync manager](./design/sync-manager.md).
* Howtos for specific coding activities:
* Code and architecture guidelines:
* [Guide to Building a Rust Component](./howtos/building-a-rust-component.md)
* [Guide to Testing a Rust Component](./howtos/testing-a-rust-component.md)
* [Guide to adding a new Component to our build/publish setup](./howtos/adding-a-new-component.md)
* [Guide to converting component with hand-written bindings to use UniFFI](./howtos/converting-a-component-to-uniffi.md)
* [How to pass data cross the FFI boundary](./howtos/when-to-use-what-in-the-ffi.md)
* Development Tooling:
* [How to try out local changes in Fenix](./howtos/locally-published-components-in-fenix.md)
* [How to try out local changes in Firefox for iOS](./howtos/locally-published-components-in-ios.md)
* [How to access logs for debugging](./logging.md)
* [How to build and use a locally-modified version of JNA](./howtos/locally-building-jna.md)
* Process guidelines:
* [How to cut a new release](./howtos/cut-a-new-release.md)
* For consumers:
* [Guide to Consuming Rust Components on Android](./howtos/consuming-rust-components-on-android.md)
* [Guide to Consuming Rust Components on iOS](./howtos/consuming-rust-components-on-ios.md)
## License
The Application Services Source Code is subject to the terms of the Mozilla Public License v2.0.
You can obtain a copy of the MPL at <https://mozilla.org/MPL/2.0/>.

26
docs/SUMMARY.md Normal file
Просмотреть файл

@ -0,0 +1,26 @@
[application-services Rust Components](README.md)
- [Contributing](contributing.md)
- [Building](building.md)
- [How to use the local development autopublish flow for Fenix](howtos/locally-published-components-in-fenix.md)
- [How to use the local development flow for Firefox iOS](howtos/locally-published-components-in-ios.md)
- [How to locally build JNA](howtos/locally-building-jna.md)
- [How we test Rust Components](howtos/testing-a-rust-component.md)
- [How to integration (smoke) test application-services](howtos/smoke-testing-app-services.md)
- [Writing efficient tests](design/test-faster.md)
- [Dependency management](dependency-management.md)
- [How to add a new component](howtos/adding-a-new-component.md)
- [How to build a new syncable component](howtos/building-a-rust-component.md)
- [Naming Conventions](naming-conventions.md)
- [How to convert a Rust Component to Uniffi](howtos/converting-a-component-to-uniffi.md)
- [How to know what to use when in the FFI](howtos/when-to-use-what-in-the-ffi.md)
- [How to use Rust Components in Android](android-faqs.md))
- [How to use Rust Components on iOS](howtos/consuming-rust-components-on-ios.md)
- [Logging](logging.md)
- [How to cut a new release](howtos/cut-a-new-release.md)
- [How we distribute the application-services library](design/megazords.md)
- [CI Publishing tools and flow](build-and-publish-pipeline.md)
- [How to upgrade NSS](howtos/upgrading-nss-guide.md)
- [Metrics - (Glean Telemetry)](metrics/README.md)
- [Logins](metrics/logins/metrics.md)
- [Places](metrics/places/metrics.md)
- [Adding to these documents](adding-docs.md)

23
docs/adding-docs.md Normal file
Просмотреть файл

@ -0,0 +1,23 @@
# Developing documentation
The documentation in this repository pertains to the application-services library, primarily the sync and storage componets, firefox account client and the nimbus-sdk, experimentation client.
The markdown is converted to static HTML using [mdbook](https://rust-lang.github.io/mdBook/). To add a new document, you need to add it to the SUMMARY.md file which produces the sidebar table of contents.
## Building documentation
### Building the narrative (book) documentation
The `mdbook` crate is required in order to build the documentation:
```sh
cargo install mdbook mdbook-mermaid mdbook-open-on-gh
```
The repository documents are be built with:
```sh
./tools/build.docs.sh
```
The built documentation is saved in `build/docs/book`.

21
docs/book.toml Normal file
Просмотреть файл

@ -0,0 +1,21 @@
[book]
title = "Cross-platform Rust Components"
authors = ["Firefox SACI Team"]
src = "."
[build]
build-dir = "book"
create-missing = false
[output.html]
additional-css = ["shared/a-s.css", "shared/mermaid.css"]
additional-js = ["shared/tabs.js", "shared/mermaid.min.js", "shared/mermaid-init.js"]
git-repository-url = "https://github.com/mozilla/application-services"
git-branch = "main"
mathjax-support = true
[preprocessor.open-on-gh]
command = "mdbook-open-on-gh"
[preprocessor.mermaid]
command = "mdbook-mermaid"

169
docs/shared/a-s.css Normal file
Просмотреть файл

@ -0,0 +1,169 @@
/* Style the tab */
.tabbar {
overflow: hidden;
border: 1px solid #ccc;
background-color: #f1f1f1;
}
/* Style the buttons that are used to open the tab content */
.tabbar button {
background-color: inherit;
float: left;
border: none;
outline: none;
cursor: pointer;
padding: 14px 16px;
transition: 0.3s;
}
/* Change background color of buttons on hover */
.tabbar button:hover {
background-color: #ddd;
}
/* Create an active/current tablink class */
.tabbar button.active {
background-color: #ccc;
}
/* The container that holds all of the tab contents */
.tabcontents {
display: flex;
}
/* The container for each individual language */
.tab {
display: none;
width: 100%;
border: 1px solid #ccc;
border-top: none;
padding: 6px 12px;
}
.tab.active {
display: block;
}
/* The footer with the "Open on GitHub" link */
footer#open-on-gh {
font-size: 0.8em;
text-align: center;
border-top: 1px solid black;
padding: 5px 0;
}
/* Distribution simulator styles */
#simulator-container {
display: flex;
flex-wrap: wrap;
overflow: hidden;
}
#simulator-container h3 {
margin-top: 20px;
}
#simulator-container .input-group {
width: 100%;
}
#simulator-container .input-group label {
display: block;
font-weight: bold;
font-size: 14px;
margin-bottom: 7px;
}
#simulator-container .input-group input,
#simulator-container .input-group select,
#custom-data-modal textarea {
display: block;
width: 100%;
padding: 5px;
margin-bottom: 10px;
border-radius: 3px;
border: 1px solid #e0e0e0;
box-sizing: border-box;
}
#histogram-props,
#data-options {
width: 50%;
box-sizing: border-box;
}
#data-options {
padding-right: 50px;
}
#data-options .input-group {
margin-bottom: 10px;
}
#data-options .input-group:first-of-type {
margin-top: 20px;
}
#data-options .input-group:last-of-type {
margin-bottom: 0;
}
#data-options .input-group label {
display: inline-block;
}
#data-options .input-group input {
display: inline;
width: auto;
}
#histogram-chart-container {
width: 100%;
padding: 30px;
border: 1px solid #e0e0e0;
margin: 30px 0;
overflow: hidden;
position: relative;
}
#histogram-chart {
margin-top: 50px;
width: 100%;
}
#histogram-chart-legend {
font-size: 14px;
text-align: center;
width: 100%;
}
#histogram-functional-props,
#histogram-non-functional-props {
display: none;
}
#custom-data-modal-overlay {
background-color: rgba(0, 0, 0, .5);
position: fixed;
width: 100vw;
height: 100vh;
top: 0;
left: 0;
z-index: 999;
display: none;
}
#custom-data-modal {
width: 50%;
background-color: white;
border-radius: 5px;
position: relative;
top: 15%;
left: 25%;
padding: 50px;
}
.hide {
display: none !important;
}

Просмотреть файл

@ -0,0 +1,4 @@
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
mermaid.initialize({startOnLoad:true});

356
docs/shared/mermaid.css Normal file
Просмотреть файл

@ -0,0 +1,356 @@
/*
Licensed under the MIT License (MIT).
Copyright (c) 2014 - 2018 Knut Sveidqvist
Full license: https://github.com/mermaid-js/mermaid/blob/develop/LICENSE
*/
/* Flowchart variables */
/* Sequence Diagram variables */
/* Gantt chart variables */
.mermaid .mermaid .label {
color: #333;
}
.mermaid .node rect,
.mermaid .node circle,
.mermaid .node ellipse,
.mermaid .node polygon {
fill: #ECECFF;
stroke: #CCCCFF;
stroke-width: 1px;
}
.mermaid .arrowheadPath {
fill: #333333;
}
.mermaid .edgePath .path {
stroke: #333333;
}
.mermaid .edgeLabel {
background-color: #e8e8e8;
}
.mermaid .cluster rect {
fill: #ffffde !important;
rx: 4 !important;
stroke: #aaaa33 !important;
stroke-width: 1px !important;
}
.mermaid .cluster text {
fill: #333;
}
.mermaid .actor {
stroke: #CCCCFF;
fill: #ECECFF;
}
.mermaid text.actor {
fill: black;
stroke: none;
}
.mermaid .actor-line {
stroke: grey;
}
.mermaid .messageLine0 {
stroke-width: 1.5;
stroke-dasharray: "2 2";
marker-end: "url(#arrowhead)";
stroke: #333;
}
.mermaid .messageLine1 {
stroke-width: 1.5;
stroke-dasharray: "2 2";
stroke: #333;
}
.mermaid #arrowhead {
fill: #333;
}
.mermaid #crosshead path {
fill: #333 !important;
stroke: #333 !important;
}
.mermaid .messageText {
fill: #333;
stroke: none;
}
.mermaid .labelBox {
stroke: #CCCCFF;
fill: #ECECFF;
}
.mermaid .labelText {
fill: black;
stroke: none;
}
.mermaid .loopText {
fill: black;
stroke: none;
}
.mermaid .loopLine {
stroke-width: 2;
stroke-dasharray: "2 2";
marker-end: "url(#arrowhead)";
stroke: #CCCCFF;
}
.mermaid .note {
stroke: #aaaa33;
fill: #fff5ad;
}
.mermaid .noteText {
fill: black;
stroke: none;
font-family: 'trebuchet ms', verdana, arial;
font-size: 14px;
}
/** Section styling */
.mermaid .section {
stroke: none;
opacity: 0.2;
}
.mermaid .section0 {
fill: rgba(102, 102, 255, 0.49);
}
.mermaid .section2 {
fill: #fff400;
}
.mermaid .section1,
.mermaid .section3 {
fill: white;
opacity: 0.2;
}
.mermaid .sectionTitle0 {
fill: #333;
}
.mermaid .sectionTitle1 {
fill: #333;
}
.mermaid .sectionTitle2 {
fill: #333;
}
.mermaid .sectionTitle3 {
fill: #333;
}
.mermaid .sectionTitle {
text-anchor: start;
font-size: 11px;
text-height: 14px;
}
/* Grid and axis */
.mermaid .grid .tick {
stroke: lightgrey;
opacity: 0.3;
shape-rendering: crispEdges;
}
.mermaid .grid path {
stroke-width: 0;
}
/* Today line */
.mermaid .today {
fill: none;
stroke: red;
stroke-width: 2px;
}
/* Task styling */
/* Default task */
.mermaid .task {
stroke-width: 2;
}
.mermaid .taskText {
text-anchor: middle;
font-size: 11px;
}
.mermaid .taskTextOutsideRight {
fill: black;
text-anchor: start;
font-size: 11px;
}
.mermaid .taskTextOutsideLeft {
fill: black;
text-anchor: end;
font-size: 11px;
}
/* Specific task settings for the sections*/
.mermaid .taskText0,
.mermaid .taskText1,
.mermaid .taskText2,
.mermaid .taskText3 {
fill: white;
}
.mermaid .task0,
.mermaid .task1,
.mermaid .task2,
.mermaid .task3 {
fill: #8a90dd;
stroke: #534fbc;
}
.mermaid .taskTextOutside0,
.mermaid .taskTextOutside2 {
fill: black;
}
.mermaid .taskTextOutside1,
.mermaid .taskTextOutside3 {
fill: black;
}
/* Active task */
.mermaid .active0,
.mermaid .active1,
.mermaid .active2,
.mermaid .active3 {
fill: #bfc7ff;
stroke: #534fbc;
}
.mermaid .activeText0,
.mermaid .activeText1,
.mermaid .activeText2,
.mermaid .activeText3 {
fill: black !important;
}
/* Completed task */
.mermaid .done0,
.mermaid .done1,
.mermaid .done2,
.mermaid .done3 {
stroke: grey;
fill: lightgrey;
stroke-width: 2;
}
.mermaid .doneText0,
.mermaid .doneText1,
.mermaid .doneText2,
.mermaid .doneText3 {
fill: black !important;
}
/* Tasks on the critical line */
.mermaid .crit0,
.mermaid .crit1,
.mermaid .crit2,
.mermaid .crit3 {
stroke: #ff8888;
fill: red;
stroke-width: 2;
}
.mermaid .activeCrit0,
.mermaid .activeCrit1,
.mermaid .activeCrit2,
.mermaid .activeCrit3 {
stroke: #ff8888;
fill: #bfc7ff;
stroke-width: 2;
}
.mermaid .doneCrit0,
.mermaid .doneCrit1,
.mermaid .doneCrit2,
.mermaid .doneCrit3 {
stroke: #ff8888;
fill: lightgrey;
stroke-width: 2;
cursor: pointer;
shape-rendering: crispEdges;
}
.mermaid .doneCritText0,
.mermaid .doneCritText1,
.mermaid .doneCritText2,
.mermaid .doneCritText3 {
fill: black !important;
}
.mermaid .activeCritText0,
.mermaid .activeCritText1,
.mermaid .activeCritText2,
.mermaid .activeCritText3 {
fill: black !important;
}
.mermaid .titleText {
text-anchor: middle;
font-size: 18px;
fill: black;
}
.mermaid g.classGroup text {
fill: #9370DB;
stroke: none;
font-family: 'trebuchet ms', verdana, arial;
font-size: 10px;
}
.mermaid g.classGroup rect {
fill: #ECECFF;
stroke: #9370DB;
}
.mermaid g.classGroup line {
stroke: #9370DB;
stroke-width: 1;
}
.mermaid svg .classLabel .box {
stroke: none;
stroke-width: 0;
fill: #ECECFF;
opacity: 0.5;
}
.mermaid svg .classLabel .label {
fill: #9370DB;
font-size: 10px;
}
.mermaid .relation {
stroke: #9370DB;
stroke-width: 1;
fill: none;
}
.mermaid .composition {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #compositionStart {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #compositionEnd {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid .aggregation {
fill: #ECECFF;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #aggregationStart {
fill: #ECECFF;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #aggregationEnd {
fill: #ECECFF;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #dependencyStart {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #dependencyEnd {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #extensionStart {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid #extensionEnd {
fill: #9370DB;
stroke: #9370DB;
stroke-width: 1;
}
.mermaid .node text {
font-family: 'trebuchet ms', verdana, arial;
font-size: 14px;
}
.mermaid div.mermaidTooltip {
position: absolute;
text-align: center;
max-width: 200px;
padding: 2px;
font-family: 'trebuchet ms', verdana, arial;
font-size: 12px;
background: #ffffde;
border: 1px solid #aaaa33;
border-radius: 2px;
pointer-events: none;
z-index: 100;
}

49
docs/shared/mermaid.min.js поставляемый Normal file

Различия файлов скрыты, потому что одна или несколько строк слишком длинны

Просмотреть файл

@ -0,0 +1,2 @@
</div>
</div>

Просмотреть файл

@ -0,0 +1,3 @@
<div class="tabs">
<div class="tabbar"></div>
<div class="tabcontents">

97
docs/shared/tabs.js Normal file
Просмотреть файл

@ -0,0 +1,97 @@
/**
* Returns true if browser supports HTML5 localStorage.
*/
function supportsHTML5Storage() {
try {
return 'localStorage' in window && window['localStorage'] !== null;
} catch (e) {
return false;
}
}
/**
* Event handler for when a tab is clicked.
*/
function onClickTab(event) {
let target = event.currentTarget;
let language = target.dataset.lang;
const initialTargetOffset = target.offsetTop;
const initialScrollPosition = window.scrollY;
switchAllTabs(language);
// Keep initial perceived scroll position after resizing
// that may happen due to activation of multiple tabs in the same page.
const finalTargetOffset = target.offsetTop;
window.scrollTo({
top: initialScrollPosition + (finalTargetOffset - initialTargetOffset)
});
}
/**
* Switches the displayed tab for the given container.
*
* :param container: The div containing both the tab bar and the individual tabs
* as direct children.
* :param language: The language to switch to.
*/
function switchTab(container, language) {
const previouslyActiveTab = container.querySelector(".tabcontents .active");
previouslyActiveTab && previouslyActiveTab.classList.remove("active");
let tab = container.querySelector(`.tabcontents [data-lang="${language}"]`);
tab && tab.classList.add("active");
const previouslyActiveButton = container.querySelector(".tabbar .active");
previouslyActiveButton && previouslyActiveButton.classList.remove("active");
let button = container.querySelector(`.tabbar [data-lang="${language}"]`);
button && button.classList.add("active");
}
/**
* Switches all tabs on the page to the given language.
*
* :param language: The language to switch to.
*/
function switchAllTabs(language) {
const containers = document.getElementsByClassName("tabs");
for (let i = 0; i < containers.length; ++i) {
switchTab(containers[i], language);
}
if (supportsHTML5Storage()) {
localStorage.setItem("glean-preferred-language", language);
}
}
/**
* Opens all tabs on the page to the given language on page load.
*/
function openTabs() {
if (!supportsHTML5Storage()) {
return;
}
let containers = document.getElementsByClassName("tabs");
for (let i = 0; i < containers.length; ++i) {
// Create tabs for every language that has content
let tabs = containers[i].children[0];
let tabcontents = containers[i].children[1];
for (let tabcontent of tabcontents.children) {
let button = document.createElement("button");
button.dataset.lang = tabcontent.dataset.lang;
button.className = "tablinks";
button.onclick = onClickTab;
button.innerText = tabcontent.dataset.lang;
tabs.appendChild(button);
}
}
var language = localStorage.getItem("glean-preferred-language");
if (language == null) {
language = "Kotlin";
}
switchAllTabs(language);
}
openTabs()

1
docs/sync-storage-handbook/.gitignore поставляемый
Просмотреть файл

@ -1 +0,0 @@
book/**

Просмотреть файл

@ -1,8 +0,0 @@
[book]
title = "Sync and Storage Handbook"
author = "Mozilla Application Services Team"
description = "An overview of the Application Services sync and storage libraries."
[build]
build-dir = "book"
create-missing = true

Просмотреть файл

@ -1,23 +0,0 @@
# Welcome!
The Application Services libraries provide cross-platform components for storing and syncing user data within the Firefox ecosystem. Firefox manages _lots_ of data: a typical user profile houses over forty different stores, including your history, bookmarks, and saved logins.
As we make Firefox available on more platforms—and ship products that aren't web browsers—it's become clear that each project wants to access existing Firefox data, and make its own data available to others. The goal of the sync and storage components is to expose a uniform, flexible, and high-level way to do this.
## What?
**High-level** means your application thinks in terms of _what_ it wants to do, not _how_. Adding a bookmark, storing a page visit, updating a saved password, and syncing history are all examples of the former. Meanwhile, the component takes care of the details: defining the database schema, handling migrations, optimizing queries, downloading and uploading Sync records, and resolving merge conflicts.
**Uniform** means one way to access data everywhere. The same building blocks are used for each component; once you know how one works, you can understand the others. They are also consistent across products and platforms, so you don't need to change three different codebases.
**Flexible** means it's easy for your application to add new fields and data types, and experiment with new ways to represent data.
## Why?
Historically, each product had to build its own storage and sync system. They often started with similar data models, but then evolved based on immediate product needs. Changes had to be backported to each platform, often across languages: Firefox Desktop was written in a mix of JavaScript and C++, Firefox for Android in Java, and Firefox for iOS in Swift.
Beyond the language barrier, there was little commonality between the implementations. All platforms used [SQLite](https://sqlite.org/) for storage, with a similarly-shaped schema, but the similarities ended there. Some concepts didn't translate well, if at all, and coordinating changes across platforms was hard. Syncing was often bolted on, and required lots of low-level integration work. This made for an inconsistent, brittle experience for developers and users.
The new components build on our experience shipping sync and storage on three platforms. The cross-platform parts are written in [Rust](https://www.rust-lang.org/), with bindings for [Kotlin](https://kotlinlang.org/) and [Swift](https://swift.org/).
In the next section, we'll look at how a component works.

Просмотреть файл

@ -1,8 +0,0 @@
# Summary
* [Introduction](README.md)
* [Components](ch01-components.md)
- [The database](ch01.1-db.md)
- [The syncable store](ch01.2-store.md)
- [FFI](ch01.3-ffi.md)
* [Telemetry](ch02-telemetry.md)

Просмотреть файл

@ -1,18 +0,0 @@
# Components
We currently provide two components.
**Logins** manage saved usernames and passwords. This component exposes APIs for creating and updating logins, and supports "unlocking" and "locking" a store by opening or closing its database connection. Internally, it tracks metadata like the use count, last use date, and last change date. This component is currently used in [Lockbox](https://mozilla-lockbox.github.io/) and [Firefox for iOS](https://github.com/mozilla-mobile/firefox-ios/).
**Places** manages bookmarks and history. This component provides high-level APIs for common use cases, like recording history visits, autocompleting visited URLs, clearing and expiring history, and managing bookmarks. Places is based on the Firefox Desktop implementation, and uses a mostly backward-compatible storage schema. It can either be consumed directly, as in Firefox for iOS, or through a layer like [Mozilla Android Components](https://mozac.org/). It's used in the [Android Reference Browser](https://github.com/mozilla-mobile/reference-browser/) and [Fenix](https://github.com/mozilla-mobile/fenix/).
## Anatomy of a Component
All components share a similar architecture.
* The **database** layer persists data to disk.
* The **domain** layer defines data types and the operations on them.
* The **FFI layer** is the glue between the application and storage.
* The **binding layer** exposes an idiomatic, platform-specific API for the application.
We'll take a closer look at each of the layers in the following sections.

Просмотреть файл

@ -1,31 +0,0 @@
# The database
Our database of choice is SQLite. Logins, bookmarks, and history are relational data, and Places in particular makes heavy use of joins, indexes, constraints, and triggers. Each component defines a schema with tables for local and synced data. The sync tables also stage incoming and outgoing items, and help with conflict resolution. Finally, this layer manages SQLite connections, and provides conveniences for executing SQL statements.
## Connection management
A database can have many read-only connections, but only two read-write connections: a main writer that's exposed to the application, and an internal Sync connection. Each component provides a way to open a new reader, or get the existing writer.
## Schema migrations
Opening the first connection to an existing database runs migrations to bring the database up to date. The latest schema and version are always baked in to the component. For the logins component, you can find the schema in `components/logins/src/schema.rs`. For Places, take a look at `components/places/sql` and `components/places/src/db/schema.rs`.
Opening a connection to a file that doesn't exist creates and initializes an empty database. Opening a database with a newer schema version is supported, but the version will be rolled back. As long as the schema changes are backward-compatible, this lets different release channels work with the same database.
Read-write connections can also set up temporary in-memory tables and triggers. These are used for internal operations, like history expiration, frecency recalculation, and merging synced bookmarks in Places. Read-only connections don't have these.
## WAL
All connections use SQLite's [WAL](https://sqlite.org/wal.html) mode. This is faster than the default rollback journal, and allows concurrent reads and writes. However, this does mean that read-only connections might not immediately see changes from read-write or Sync connections.
Reading occasionally stale data is perfect for performance-sensitive use cases, like fetching URL bar autocomplete matches. For cases where consistency is more important, like the history, bookmarks, and saved logins list views, it's better to always read using the read-write connection. That way, reads after writes always return the just-written data.
WAL mode doesn't allow multiple concurrent writes. This means that, for example, adding a bookmark or navigating to a page during a sync may cause one of the writes to fail. The Places component handles this, and prioritizes writes from the main connection over the Sync connection.
## Maintenance
Maintenance runs a tune up on the database, rebuilding indexes and checking for schema coherence. Currently, only the Places component exposes a maintenance operation, for vacuuming the database.
## Encryption
Components can use SQLCipher to encrypt data on disk. This is currently enabled by default for logins, and disabled for Places.

Просмотреть файл

@ -1,15 +0,0 @@
# Syncable store
The syncable store, part of the domain layer, is the heart of each component. It exposes high-level operations for each data type, like recording a history visit, adding a bookmark, changing a password, or syncing with the server. The store is written in safe, idiomatic Rust, and shared across platforms, so you only need to implement operations once.
The store lives in the `src` subdirectory of each component.
# Syncing
Syncing is tightly integrated into storage. This isn't an accident—we've found that pushing concerns like change tracking and conflict resolution down into the store is far less error-prone than managing them separately.
## Merging
Conflict resolution is an important part of syncing. Each store implements a strategy for reconciling local changes with changes from other devices.
Logins uses three-way merging to resolve sync conflicts, where changing different fields of the same record on both sides carries both changes forward. Places uses a two-way [tree merge](https://mozilla.github.io/dogear/) for bookmarks, and takes the complete record based on its timestamp.

Просмотреть файл

@ -1,5 +0,0 @@
# FFI
All components expose a C ABI-compatible [foreign function interface](https://doc.rust-lang.org/nomicon/ffi.html) for platform-specific bindings. These functions are unsafe by necessity, as the FFI supports only a limited set of types. They also take care of managing calls from multiple threads, handing out safe references to Rust structures, and serializing and deserializing arguments using Protobufs.
The [ffi-support](https://docs.rs/ffi-support/0.1.3/ffi_support/) crate extracts some of the common patterns we've adopted for our FFIs.

Просмотреть файл

@ -1,3 +0,0 @@
# Bindings
The binding layer is an ergonomic interface for the application that's consuming the component. It's written in Kotlin for Android, and Swift for iOS. These bindings live in `components/{component}/android` and `components/{component}/ios`, respectively. They're meant to be as thin as possible; the bulk of the implementation lives in shared Rust code.

Просмотреть файл

@ -1,3 +0,0 @@
# Telemetry
Most Application Services consumers already collect and submit telemetry. For this reason, the sync and storage components record telemetry, then pass it to the application, instead of submitting it themselves.

1
docs/synconomicon/.gitignore поставляемый
Просмотреть файл

@ -1 +0,0 @@
book/**

Просмотреть файл

@ -1,8 +0,0 @@
[book]
title = "The Synconomicon"
author = "Mozilla Application Services Team"
description = "A deep dive into the internals of sync and storage."
[build]
build-dir = "book"
create-missing = true

Просмотреть файл

@ -1,3 +0,0 @@
# Synconomicon
This is a deep dive into the internals of sync and storage.

Просмотреть файл

@ -1,7 +0,0 @@
# Summary
* [Introduction](README.md)
* [Sync data model](ch01-data-model.md)
- [Logins](ch01.1-logins.md)
* [Life of a sync](ch02-life-of-a-sync.md)
* [Builds](ch03-builds.md)

Просмотреть файл

@ -1,37 +0,0 @@
# Sync data model
All synced data types fit into one of three broad categories: trees, logs, and documents.
## Trees
Bookmarks are an example of a tree. Trees are hierarchical, and the most complex type to sync. Records in tree collections are interdependent, where parents contain pointers to their children, and children back to their parents.
Tree-structured data presents a problem for syncing, because some changes may span multiple records. These changes must be uploaded in lockstep, and applied in order. Missing or incomplete records lead to problems like orphans and parent-child disagreements. Clients must also be prepared to handle and fix inconsistent structure on the server.
## Logs
History is an example of a log. Logs are append-only, independent, and can be synced in any order. Since all entries are distinct, there are no conflicts: two visits to the same page on two different devices are still two distinct visits. These records are independent, and can be synced in any order.
Log data is the easiest to sync, but the problem is _volume_. We currently limit history to the last 20 visits per page, cap initial syncs to 5,000 pages in the last 30 days, and expire records that aren't updated on the server after 60 days. These limitations are for efficiency as much as for historical reasons. Clients don't need to process thousands of entries on each sync, and the server avoids bloated indexes for large collections. Unfortunately, this is also a form of data loss, as the server never has the complete history.
## Documents
Logins, addresses, and credit cards are examples of semistructured document data. Like logs, documents are independent, and can be synced in any order relative to each other. However, they _can_ conflict if two clients change the same record.
Engines that implement three-way merging support per-field conflict resolution, since they store the value that was last uploaded to the server. However, engines that only support two-way merging resolve conflicts at the record level, based on which side is newer.
Documents _can_ refer to other records. For example, credit cards have an "address hint" that points to a potential address record for the card. However, these identifiers aren't stable, and can't be enforced by the server or other clients. Each client must expect and handle stale and nonexistent references.
There's a [proposal to support generic syncing](https://github.com/mozilla/application-services/pull/658) for document types. We expect most new data types to fall into this category.
## Change tracking
Each client must track changes to synced data, so that it knows what to upload during the next sync. How this is done depends on the client, the data type, and the underlying store.
## Server record format
On the server, each piece of synced data is stored as a Basic Storage Object, or BSO. A BSO is a JSON envelope that contains an encrypted blob, which is itself a JSON string when decrypted. BSOs are grouped together in buckets called collections, where each BSO belonging to the same collection has the same structure.
BSOs are typically referenced as `collection/id`. For example, `meta/global` means "the BSO `global` in the `meta` collection." Most BSOs have a random, globally unique identifier, like `bookmarks/fvcXYVP3pY2w`. Others, like `meta/global` and `crypto/keys`, have well-known names.
BSOs are encrypted on the client side before upload, and decrypted during download. This means the server can't see the contents of any Sync records, except for their collection names, IDs, and last modified timestamps.

Просмотреть файл

@ -1,50 +0,0 @@
# Logins
The logins SQL schema is based on the original [Firefox for iOS implementation](https://github.com/mozilla-mobile/firefox-ios/blob/faa6a2839abf4da2c54ff1b3291174b50b31ab2c/Storage/SQL/SQLiteLogins.swift).
Logins used to be called "passwords", so you might still see references to the latter. The Sync server, for instance, stores login records in the "passwords" collection.
## Record format
There are two kinds of saved logins: forms, and HTTP authentication credentials.
Forms with username and password fields are the most common type. Here's an example of a form record:
```json
{
"hostname": "https://example.com",
"formSubmitURL": "https://example.com",
"httpRealm": null,
"username": "hi@example.com",
"password": "S3kr3tz",
"usernameField": "inputEmail",
"passwordField": "inputPassword",
"timeCreated": 1554854025151,
"timePasswordChanged": 1554854025151,
}
```
* `hostname` is the _origin_ of the site: scheme, hostname, and port. (It's called the "hostname" for historical reasons, even though it includes the scheme and port, too).
* `formSubmitURL` is the _origin_ of the `<form>` element's `action` attribute, or the origin of the site if the `<form>` for this login doesn't have an `action`.
* `httpRealm` is always `null`.
* `usernameField` and `passwordField` are the `name`s of the `<input>` fields containing the username and password.
* `timeCreated` and `timePasswordChanged` are when the login was first saved, and when the password was last changed, respectively. Both are in milliseconds.
HTTP authentication credentials aren't as common. These come from the "Authentication Required" dialog in Firefox Desktop. Here's an example record:
```json
{
"hostname": "https://example.com",
"formSubmitURL": null,
"httpRealm": "",
"username": "user",
"password": "S3kr3tz",
"usernameField": "",
"passwordField": "",
"timeCreated": 1554854025151,
"timePasswordChanged": 1554854025151,
}
```
* Unlike in forms, `formSubmitURL` is always `null`, and `usernameField` and `passwordField` are always empty.
* `httpRealm` comes from the `WWW-Authenticate` HTTP header. It's what Firefox shows you in the dialog, after "The site says:".

Просмотреть файл

@ -1,36 +0,0 @@
# Life of a sync
Each sync goes through a number of steps, implemented as a state machine. These states handle authentication, fetching encryption keys, pulling new changes from the server and local store, merging, and updating the server and store.
## 1. Authentication
### 1.1 Get an OAuth token
The first step authenticates with [Firefox Accounts](https://mozilla.github.io/application-services/docs/accounts/welcome.html) to obtain an **OAuth token** and **Sync encryption keys**. If we already have a token and keys, we can skip ahead to 1.2, and exchange it for a tokenserver token. The authentication and encryption scheme is described in [this wiki page](https://github.com/mozilla/fxa-auth-server/wiki/onepw-protocol).
*Legacy clients also support authentication with signed BrowserID assertions, but this flow is deprecated and intentionally undocumented.*
### 1.2 Exchange the OAuth token for a tokenserver token
The [tokenserver](https://mozilla-services.readthedocs.io/en/latest/token/index.html) handles **node assignment**, so we know which storage node to talk to, and **token generation**, so we can authenticate to that node.
## 2. Setup
At this point, we have our tokenserver token, and can make authenticated requests to our storage node.
### 1.1 Fetch `info/collections`
The `info/collections` endpoint returns last-modified times for all collections.
### 1.2 Fetch or upload `meta/global`
The `meta/global` record holds sync IDs, storage versions, and collections that we declined to sync.
### 1.3 Fetch or upload `crypto/keys`
The `crypto/keys` record holds collection encryption keys. This key is encrypted with kB.
## 3. Sync
Now that we know what we're syncing, and have our keys, we can download and upload records.

Просмотреть файл

@ -1,3 +0,0 @@
# Builds
Each sync and storage component can be consumed either individually, or as part of a package called a _megazord_. Megazording centralizes global state management, and ensures that build artifacts only contain one copy of each library.

27
tools/build-book.sh Executable file
Просмотреть файл

@ -0,0 +1,27 @@
#!/bin/bash
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# Build all docs with one command
# Documentation will be placed in `build/docs`.
set -xe
# Build the development book
output=$(mdbook build docs 2>&1)
if echo "$output" | grep -q "\[ERROR\]" ; then
exit 1
fi
# copy the output files to the publising directory
rm -rf build/docs
mkdir -p build/docs
echo '<meta http-equiv=refresh content=0;url=book/index.html>' > build/docs/index.html
mkdir -p build/docs
cp -a docs/book/. build/docs/book
mkdir -p build/docs/shared
cp -a docs/shared/. build/docs/shared