From 977d5b8a5790567cc50c89f17ac9b6e4a5040142 Mon Sep 17 00:00:00 2001 From: Nicholas Nethercote Date: Tue, 7 Jul 2020 07:38:27 +0000 Subject: [PATCH] Bug 1648348 - Create "Writing Rust Code" docs. r=froydnj,zbraniecki,lina. This patch includes content from the following places. - Lina's "Getting Rusty: How to ship an XPCOM component in Firefox" slide deck. - Zibi's "Rust <--> C/C++ FFI for newbies" gist. It also links to Emilio's "FFI patterns #1 - Complex Rust data structures exposed seamlessly to C++" blog post. I was going to include that content, but it's very long, so I have omitted it for now. Differential Revision: https://phabricator.services.mozilla.com/D81963 --- build/docs/rust.rst | 9 +- docs/config.yml | 1 + docs/testing-rust-code/index.md | 6 + docs/writing-rust-code/basics.md | 83 +++++++++++ docs/writing-rust-code/ffi.md | 240 +++++++++++++++++++++++++++++++ docs/writing-rust-code/index.md | 17 +++ docs/writing-rust-code/xpcom.md | 123 ++++++++++++++++ moz.build | 2 + 8 files changed, 478 insertions(+), 3 deletions(-) create mode 100644 docs/writing-rust-code/basics.md create mode 100644 docs/writing-rust-code/ffi.md create mode 100644 docs/writing-rust-code/index.md create mode 100644 docs/writing-rust-code/xpcom.md diff --git a/build/docs/rust.rst b/build/docs/rust.rst index 699e50654691..c3d377d16070 100644 --- a/build/docs/rust.rst +++ b/build/docs/rust.rst @@ -4,9 +4,12 @@ Including Rust Code in Firefox ============================== -The build system has support for building, linking, and vendoring Rust crates. -It is straightforward to take an existing Rust crate and integrate it into -Firefox. +This page explains how to add, build, link, and vendor Rust crates. + +The `code documentation <../../writing-rust-code>`_ explains how to write and +work with Rust code in Firefox. The +`test documentation <../../testing-rust-code>`_ explains how to test and debug +Rust code in Firefox. Linking Rust crates into libxul =============================== diff --git a/docs/config.yml b/docs/config.yml index 4b8b538f67eb..c5ea1e180eba 100644 --- a/docs/config.yml +++ b/docs/config.yml @@ -24,6 +24,7 @@ categories: - uriloader - widget/cocoa - code-quality + - writing-rust-code build_doc: - mach - tools/try diff --git a/docs/testing-rust-code/index.md b/docs/testing-rust-code/index.md index da1035796fdc..d7fb87ada9ac 100644 --- a/docs/testing-rust-code/index.md +++ b/docs/testing-rust-code/index.md @@ -1,5 +1,11 @@ # Testing & Debugging Rust Code +This page explains how to test and debug Rust code in Firefox. + +The [build documentation](../build/buildsystem/rust.html) explains how to add +new Rust code to Firefox. The [code documentation](../writing-rust-code) +explains how to write and work with Rust code in Firefox. + ## Testing Mozilla crates Rust code will naturally be tested as part of system tests such as Mochitests. diff --git a/docs/writing-rust-code/basics.md b/docs/writing-rust-code/basics.md new file mode 100644 index 000000000000..1d3732de4ed1 --- /dev/null +++ b/docs/writing-rust-code/basics.md @@ -0,0 +1,83 @@ +# Basics + +## Formatting Rust code + +To format all the Rust code within a directory `$DIR`, run: +``` +./mach lint -l rustfmt --fix $DIR +``` + +## Using Cargo + +Many Cargo commands can be run on individual crates. Change into the directory +containing the crate's `Cargo.toml` file, and then run the command with +`MOZ_TOPOBJDIR` set appropriately. For example, to generate and view rustdocs +for the `xpcom` crate, run these commands: + +``` +cd xpcom/rust/xpcom +MOZ_TOPOBJDIR=$OBJDIR cargo doc +cd - +firefox target/doc/xpcom/index.html +``` +where `$OBJDIR` is the path to the object directory. + +## Using static prefs + +Static boolean/integer prefs can be easily accessed from Rust code. Add a +`rust: true` field to the pref definition in +[modules/libpref/init/StaticPrefList.yaml](https://searchfox.org/mozilla-central/source/modules/libpref/init/StaticPrefList.yaml), +like this: +```yaml +- name: my.lucky.pref + type: RelaxedAtomicBool + value: true + mirror: always + rust: true +``` +The pref can then be accessed via the `pref!` macro, like this: +``` +let my_lucky_pref = static_prefs::pref!("my.lucky.pref"); +``` + +## Helper crates + +The following in-tree helper crates provide idiomatic support for some common patterns. +- [nserror](https://searchfox.org/mozilla-central/source/xpcom/rust/nserror/src/lib.rs) +reflects `nsresult` codes into Rust. +- [nsstring](https://searchfox.org/mozilla-central/source/xpcom/rust/nsstring/src/lib.rs) + exposes bindings for XPCOM string types. You can use the same `ns{A,C}String` + types as C++ for owned strings and pass them back and forth over the + boundary. There is also `ns{A,C}Str` for dependent or borrowed strings. +- [xpcom](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/src) + provides multiple building blocks for a component's implementation. + - The `RefPtr` type is for managing reference-counted pointers. + - XPCOM service getters are generated by + [xpcom/build/Services.py](https://searchfox.org/mozilla-central/source/xpcom/build/Services.py) + and can be called like this: + ``` + let pref_service = xpcom::services::get_PrefService(); + ``` + - There is also a `get_service` function that works like `do_GetService` in + C++, as an alternative. + - A set of `derive` macros help with declaring interface implementations. The + [docs](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/xpcom_macros/src/lib.rs) + have details and examples. +- [moz_task](https://searchfox.org/mozilla-central/source/xpcom/rust/moz_task/src/lib.rs) + wraps XPCOM's threading functions in order to make it easy and safe to write + threaded code. It has helpers for getting and creating threads, dispatching + async runnables, and thread-safe handles. +- [storage](https://searchfox.org/mozilla-central/source/storage/rust/src/lib.rs) + is an interface to mozStorage, our wrapper for SQLite. It can wrap an + existing storage connection, and prepare and execute statements. This crate + wraps the synchronous connection API, and lets you execute statements + asynchronously via `moz_task`. +- [storage_variant](https://searchfox.org/mozilla-central/source/storage/variant/src/lib.rs) + is for working with variants. It also provides a `HashPropertyBag` type + that's useful for passing hash maps over XPCOM to JS. + +Unfortunately, rustdocs are [not yet generated and +hosted](https://bugzilla.mozilla.org/show_bug.cgi?id=1428139) for crates within +mozilla-central. Therefore, the crate links shown above link to files +containing the relevant rustdocs source where possible. However, you can +generate docs locally using the `cargo doc` command described above. diff --git a/docs/writing-rust-code/ffi.md b/docs/writing-rust-code/ffi.md new file mode 100644 index 000000000000..f10fad422198 --- /dev/null +++ b/docs/writing-rust-code/ffi.md @@ -0,0 +1,240 @@ +# Rust/C++ interop + +This document describes how to use FFI in Firefox to get Rust code and C++ code to interoperate. + +## Transferable types + +Generally speaking, the more complicated is the data you want to transfer, the +harder it'll be to transfer across the FFI boundary. + +Booleans, integers, and pointers cause little trouble. +- C++ `bool` matches Rust `bool` +- C++ `uint8_t` matches Rust `u8`, `int32_t` matches Rust `i32`, etc. +- C++ `const T*` matches Rust `*const T`, `T*` matches Rust `*mut T`. + +Lists are handled by C++ `nsTArray` and Rust `ThinVec`. + +For strings, it is best to use the `nsstring` helper crate. Using a raw pointer +plus length is also possible for strings, but more error-prone. + +If you need a hashmap, you'll likely want to decompose it into two lists (keys +and values) and transfer them separately. + +Other types can be handled with tools that generate bindings, as the following +sections describe. + +## Accessing C++ code and data from Rust + +To call a C++ function from Rust requires adding a function declaration to Rust. +For example, for this C++ function: + +``` +extern "C" { +bool UniquelyNamedFunction(const nsCString* aInput, nsCString* aRetVal) { + return true; +} +} +``` +add this declaration to the Rust code: +```rust +extern "C" { + pub fn UniquelyNamedFunction(input: &nsCString, ret_val: &mut nsCString) -> bool; +} +``` + +Rust code can now call `UniquelyNamedFunction()` within an `unsafe` block. Note +that if the declarations do not match (e.g. because the C++ function signature +changes without the Rust declaration being updated) crashes are likely. (Hence +the `unsafe` block.) + +Because of this unsafety, for non-trivial interfaces (in particular when C++ +structs and classes must be accessed from Rust code) it's common to use +[rust-bindgen](https://github.com/rust-lang/rust-bindgen), which generates Rust +bindings. The documentation is +[here](https://rust-lang.github.io/rust-bindgen/). + +## Accessing Rust code and data from C++ + +A common option for accessing Rust code and data from C++ is to use +[cbindgen](https://github.com/eqrion/cbindgen), which generates C++ header +files. for Rust crates that expose a public C API. cbindgen is a very powerful +tool, and this section only covers some basic uses of it. + +### Basics + +First, add suitable definitions to your Rust. `#[no_mangle]` and `extern "C"` +are required. + +```rust +#[no_mangle] +pub unsafe extern "C" fn unic_langid_canonicalize( + langid: &nsCString, + ret_val: &mut nsCString +) -> bool { + ret_val.assign("new value"); + true +} +``` + +Then, add a `cbindgen.toml` file in the root of your crate. It may look like this: + +```toml +header = """/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */""" +autogen_warning = """/* DO NOT MODIFY THIS MANUALLY! This file was generated using cbindgen. See RunCbindgen.py */ +#ifndef mozilla_intl_locale_MozLocaleBindings_h +#error "Don't include this file directly, instead include MozLocaleBindings.h" +#endif +""" +include_version = true +braces = "SameLine" +line_length = 100 +tab_width = 2 +language = "C++" +# Put FFI calls in the `mozilla::intl::ffi` namespace. +namespaces = ["mozilla", "intl", "ffi"] + +# Export `ThinVec` references as `nsTArray`. +[export.rename] +"ThinVec" = "nsTArray" +``` + +Next, extend the relevant `moz.build` file to invoke cbindgen. + +```python +if CONFIG['COMPILE_ENVIRONMENT']: + CbindgenHeader('unic_langid_ffi_generated.h', + inputs=['/intl/locale/rust/unic-langid-ffi']) + + EXPORTS.mozilla.intl += [ + '!unic_langid_ffi_generated.h', + ] +``` + +This tells the build system to run cbindgen on +`intl/locale/rust/unic-langid-ffi` to generate `unic_langid_ffi_generated.h`, +which will be placed in `$OBJDIR/dist/include/mozilla/intl/`. + +Finally, include the generated header into a C++ file and call the function. + +```c++ +#include "mozilla/intl/unic_langid_ffi_generated.h" + +using namespace mozilla::intl::ffi; + +void Locale::MyFunction(nsCString& aInput) const { + nsCString result; + unic_langid_canonicalize(aInput, &result); +} +``` + +### Complex types + +Many complex Rust types can be exposed to C++, and cbindgen will generate +appropriate bindings for all `pub` types. For example: + +```rust +#[repr(C)] +pub enum FluentPlatform { + Linux, + Windows, + Macos, + Android, + Other, +} + +extern "C" { + pub fn FluentBuiltInGetPlatform() -> FluentPlatform; +} +``` + +```c++ +ffi::FluentPlatform FluentBuiltInGetPlatform() { + return ffi::FluentPlatform::Linux; +} +``` + +For an example using cbindgen to expose much more complex Rust types to C++, +see [this blog post]. + +[this blog post]: https://crisal.io/words/2020/02/28/C++-rust-ffi-patterns-1-complex-data-structures.html + +### Instances + +If you need to create and destroy a Rust struct from C++ code, the following +example may be helpful. + +First, define constructor, destructor and getter functions in Rust. (C++ +declarations for these will be generated by cbindgen.) + +```rust +#[no_mangle] +pub unsafe extern "C" fn unic_langid_new() -> *mut LanguageIdentifier { + let langid = LanguageIdentifier::default(); + Box::into_raw(Box::new(langid)) +} + +#[no_mangle] +pub unsafe extern "C" fn unic_langid_destroy(langid: *mut LanguageIdentifier) { + drop(Box::from_raw(langid)); +} + +#[no_mangle] +pub unsafe extern "C" fn unic_langid_as_string( + langid: &mut LanguageIdentifier, + ret_val: &mut nsACString, +) { + ret_val.assign(&langid.to_string()); +} +``` + +Next, in a C++ header define a destructor via `DefaultDelete`. + +```c++ +#include "mozilla/intl/unic_langid_ffi_generated.h" +#include "mozilla/UniquePtr.h" + +namespace mozilla { + +template <> +class DefaultDelete { + public: + void operator()(intl::ffi::LanguageIdentifier* aPtr) const { + unic_langid_destroy(aPtr); + } +}; + +} // namespace mozilla +``` + +(This definition must be visible any place where +`UniquePtr` is used, otherwise C++ will try to +free the code, which might lead to strange behaviour!) + +Finally, implement the class. + +```c++ +class Locale { +public: + explicit Locale(const nsACString& aLocale) + : mRaw(unic_langid_new()) {} + + const nsCString Locale::AsString() const { + nsCString tag; + unic_langid_as_string(mRaw.get(), &tag); + return tag; + } + +private: + UniquePtr mRaw; +} +``` + +This makes it possible to instantiate a `Locale` object and call `AsString()`, +all from C++ code. + +## Other examples + +For a detailed explanation of an interface in Firefox that doesn't use cbindgen +or rust-bindgen, see [this blog post](https://hsivonen.fi/modern-cpp-in-rust/). diff --git a/docs/writing-rust-code/index.md b/docs/writing-rust-code/index.md new file mode 100644 index 000000000000..f8a6c2ad7426 --- /dev/null +++ b/docs/writing-rust-code/index.md @@ -0,0 +1,17 @@ +# Writing Rust Code + +This page explains how to write and work with Rust code in Firefox, with an +emphasis on interoperation with C++ code. + +The [build documentation](../build/buildsystem/rust.html) explains how to add +new Rust code to Firefox. The [test documentation](../testing-rust-code) +explains how to test and debug Rust code in Firefox. + +```eval_rst +.. toctree:: + :titlesonly: + :maxdepth: 1 + :glob: + + * +``` diff --git a/docs/writing-rust-code/xpcom.md b/docs/writing-rust-code/xpcom.md new file mode 100644 index 000000000000..bf900e03f80a --- /dev/null +++ b/docs/writing-rust-code/xpcom.md @@ -0,0 +1,123 @@ +# XPCOM components in Rust + +XPCOM components can be written in Rust. + +## A tiny example + +The following example shows a new type that implements `nsIObserver`. + +First, create a new empty crate (e.g. with `cargo init --lib`), and add the +following dependencies in its `Cargo.toml` file. + +```toml +[dependencies] +libc = "0.2" +nserror = { path = "../../../xpcom/rust/nserror" } +nsstring = { path = "../../../xpcom/rust/nsstring" } +xpcom = { path = "../../../xpcom/rust/xpcom" } +``` + +(The number of `../` occurrences will depend on the depth of the crate in the +file hierarchy.) + +Next hook it into the build system according to the [build +documentation](../build/buildsystem/rust.html). + +The Rust code will need to import some basic types. `xpcom::interfaces` +contains all the usual `nsI` interfaces. + +```rust +use libc::c_char; +use nserror::nsresult; +use std::sync::atomic::{AtomicBool, Ordering}; +use xpcom::{interfaces::nsISupports, RefPtr}; +``` + +The next part declares the implementation. + +```rust +#[derive(xpcom)] +#[xpimplements(nsIObserver)] +#[refcnt = "atomic"] +struct InitMyObserver { + ran: AtomicBool, +} +``` + +It defines an initializer struct, prefixed with `Init`, with three attributes. +- Some `derive` magic. +- An `xpimplements` declaration naming the interface(s) being implemented. +- The reference count type. + +Next, all interface methods are declared in the `impl` block as `unsafe` methods. + +```rust +impl MyObserver { + #[allow(non_snake_case)] + unsafe fn Observe( + &self, + _subject: *const nsISupports, + _topic: *const c_char, + _data: *const i16, + ) -> nsresult { + self.ran.store(true, Ordering::SeqCst); + nserror::NS_OK + } +} +``` + +These methods always take `&self`, not `&mut self`, so we need to use interior +mutability: `AtomicBool`, `RefCell`, `Cell`, etc. + +XPCOM methods are unsafe by default, but the +[xpcom_method!](https://searchfox.org/mozilla-central/source/xpcom/rust/xpcom/src/method.rs) +macro can be used to clean this up. It also takes care of null-checking and +hiding pointers behind references, lets you return a `Result` instead of an +`nsresult,` and so on. + +To use this type within Rust code, do something like the following. + +```rust +let observer = MyObserver::allocate(InitMyObserver { + ran: AtomicBool::new(false), +}); +let rv = unsafe { + observer.Observe(x.coerce(), + cstr!("some-topic").as_ptr(), + ptr::null()) +}; +assert!(rv.succeeded()); +``` + +The implementation has an (auto-generated) `allocate` method that takes in an +initialization struct, and returns a `RefPtr` to the instance. + +`coerce` casts any XPCOM object to one of its base interfaces; in this case, +the base interface is `nsISupports`. In C++, this would be handled +automatically through inheritance, but Rust doesn’t have inheritance, so the +conversion must be explicit. + +## Bigger examples + +The following XPCOM components are written in Rust. +- [kvstore](https://searchfox.org/mozilla-central/source/toolkit/components/kvstore), + which exposes the LMDB key-value store (via the [Rkv + library](https://docs.rs/rkv)) The API is asynchronous, using `moz_task` to + schedule all I/O on a background thread, and supports getting, setting, and + iterating over keys. +- [cert_storage](https://searchfox.org/mozilla-central/source/security/manager/ssl/cert_storage), + which stores lists of [revoked intermediate certificates](https://blog.mozilla.org/security/2015/03/03/revoking-intermediate-certificates-introducing-onecrl/). +- [bookmark_sync](https://searchfox.org/mozilla-central/source/toolkit/components/places/bookmark_sync), + which [merges](https://mozilla.github.io/dogear) bookmarks from Firefox Sync + with bookmarks in the Places database. +- [webext_storage_bridge](https://searchfox.org/mozilla-central/source/toolkit/components/extensions/storage/webext_storage_bridge), + which powers the WebExtension storage.sync API. It's a self-contained example + that pulls in a crate from application-services for the heavy lifting, wraps + that up in a Rust XPCOM component, and then wraps the component in a JS + interface. There's also some boilerplate there around adding a + `components.conf` file, and a dummy C++ header that declares the component + constructor. +- [firefox-accounts-bridge](https://searchfox.org/mozilla-central/source/services/fxaccounts/rust-bridge/firefox-accounts-bridge), + which wraps the Rust Firefox Accounts client with which we eventually want to + replace our creaky JS implementation. It has a lot of the same patterns as + webext_storage_bridge. diff --git a/moz.build b/moz.build index 7ab7752a3323..e6c8e82175a1 100644 --- a/moz.build +++ b/moz.build @@ -200,6 +200,8 @@ SPHINX_TREES['code-quality'] = 'docs/code-quality' SPHINX_TREES['testing-rust-code'] = 'docs/testing-rust-code' +SPHINX_TREES['writing-rust-code'] = 'docs/writing-rust-code' + SPHINX_TREES['bug-mgmt'] = 'docs/bug-mgmt' SPHINX_TREES['setup'] = 'docs/setup'