Add EM_ASYNC_JS macro (#14728)

Adds an easy way to declare asynchronous JS functions. With this helper, users can use await directly inside of such snippets, don't need to wrap every such function into Asyncify.handleAsync manually, don't need to deal with propagating return values, and, finally, don't need to manually list all such functions in ASYNCIFY_IMPORTS since we know statically they're asynchronous.

I've also updated docs to merge "returning values" section into the first example, since it's now straightforward with EM_ASYNC_JS, a bit more involved but still easy with EM_JS + Asyncify.handleAsync, and only Asyncify.handleSleep API requires more explanation.

Fixes #9709.

ChangeLog.md

@@ -52,6 +52,9 @@ See docs/process.md for more on how version tagging works.
   impossible to wait for the function to actually finish and retrieve its
   result. Now in those cases it will return a `Promise` instead that will
   resolve with the function's return value upon completion. (#11890)
+- Added `EM_ASYNC_JS` macro - similar to `EM_JS`, but allows using `await`
+  inside the JS block and automatically integrates with Asyncify without
+  the need for listing the declared function in `ASYNCIFY_IMPORTS` (#9709).
 
 2.0.25 - 06/30/2021
 -------------------

emcc.py

@@ -102,7 +102,7 @@ DEFAULT_ASYNCIFY_IMPORTS = [
   'emscripten_scan_registers', 'emscripten_lazy_load_code',
   'emscripten_fiber_swap',
   'wasi_snapshot_preview1.fd_sync', '__wasi_fd_sync', '_emval_await',
-  'dlopen',
+  'dlopen', '__asyncjs__*'
 ]
 
 # Target options

site/source/docs/porting/asyncify.rst

@@ -99,8 +99,10 @@ Making async Web APIs behave as if they were synchronous
 Aside from ``emscripten_sleep`` and the other standard sync APIs Asyncify
 supports, you can also add your own functions. To do so, you must create a JS
 function that is called from wasm (since Emscripten controls pausing and
-resuming the wasm from the JS runtime). One way to do that is with a JS library
-function; another is to use ``EM_JS``, which we'll use in this next example:
+resuming the wasm from the JS runtime).
+
+One way to do that is with a JS library function. Another is to use
+``EM_ASYNC_JS``, which we'll use in this next example:
 
 .. code-block:: cpp
 
@@ -108,13 +110,12 @@ function; another is to use ``EM_JS``, which we'll use in this next example:
     #include <emscripten.h>
     #include <stdio.h>
 
-    EM_JS(void, do_fetch, (), {
-      Asyncify.handleAsync(async () => {
-        out("waiting for a fetch");
-        const response = await fetch("a.html");
-        out("got the fetch response");
-        // (normally you would do something with the fetch here)
-      });
+    EM_ASYNC_JS(int, do_fetch, (), {
+      out("waiting for a fetch");
+      const response = await fetch("a.html");
+      out("got the fetch response");
+      // (normally you would do something with the fetch here)
+      return 42;
     });
 
     int main() {
@@ -123,50 +124,15 @@ function; another is to use ``EM_JS``, which we'll use in this next example:
       puts("after");
     }
 
-If you can't use the modern ``async``-``await`` syntax, there is a variant with an explicit ``wakeUp`` callback too:
-
-.. code-block:: cpp
-
-    // example.c
-    #include <emscripten.h>
-    #include <stdio.h>
-
-    EM_JS(void, do_fetch, (), {
-      Asyncify.handleSleep(wakeUp => {
-        out("waiting for a fetch");
-        fetch("a.html").then(response => {
-          out("got the fetch response");
-          // (normally you would do something with the fetch here)
-          wakeUp();
-        });
-      });
-    });
-
-    int main() {
-      puts("before");
-      do_fetch();
-      puts("after");
-    }
-
-The async operation happens in the ``EM_JS`` function ``do_fetch()``, which
-calls ``Asyncify.handleAsync`` or ``Asyncify.handleSleep``. It gives that
-function the code to be run, and gets a ``wakeUp`` function that it calls in the
-asynchronous future at the right time. After we call ``wakeUp()`` the compiled C
-code resumes normally.
-
 In this example the async operation is a ``fetch``, which means we need to wait
-for a Promise. While that is async, note how the C code in ``main()`` is
-completely synchronous!
+for a Promise. While that operation is async, note how the C code in ``main()``
+is completely synchronous!
 
 To run this example, first compile it with
 
 ::
 
-    emcc example.c -O3 -o a.html -s ASYNCIFY -s 'ASYNCIFY_IMPORTS=["do_fetch"]'
-
-Note that you must tell the compiler that ``do_fetch()`` can do an
-asynchronous operation, using ``ASYNCIFY_IMPORTS``, otherwise it won't
-instrument the code to allow pausing and resuming; see more details later down.
+    emcc example.c -O3 -o a.html -s ASYNCIFY
 
 To run this, you must run a :ref:`local webserver <faq-local-webserver>`
 and then browse to ``http://localhost:8000/a.html``.
@@ -182,80 +148,67 @@ You will see something like this:
 That shows that the C code only continued to execute after the async JS
 completed.
 
+Ways to use async APIs in older engines
+#######################################
+
+If your target JS engine doesn't support the modern ``async/await`` JS
+syntax, you can desugar the above implementation of ``do_fetch`` to use Promises
+directly with ``EM_JS`` and ``Asyncify.handleAsync`` instead:
+
+.. code-block:: cpp
+
+    EM_JS(int, do_fetch, (), {
+      return Asyncify.handleAsync(function () {
+        out("waiting for a fetch");
+        return fetch("a.html").then(function (response) {
+          out("got the fetch response");
+          // (normally you would do something with the fetch here)
+          return 42;
+        });
+      });
+    });
+
+When using this form, the compiler doesn't statically know that ``do_fetch`` is
+asynchronous anymore. Instead, you must tell the compiler that ``do_fetch()``
+can do an asynchronous operation using ``ASYNCIFY_IMPORTS``, otherwise it won't
+instrument the code to allow pausing and resuming (see more details later down):
+
+::
+
+    emcc example.c -O3 -o a.html -s ASYNCIFY -s 'ASYNCIFY_IMPORTS=["do_fetch"]'
+
+Finally, if you can't use Promises either, you can desugar the example to use
+``Asyncify.handleSleep``, which will pass a ``wakeUp`` callback to your
+function implementation. When this ``wakeUp`` callback is invoked, the C/C++
+code will resume:
+
+.. code-block:: cpp
+
+    EM_JS(int, do_fetch, (), {
+      return Asyncify.handleSleep(function (wakeUp) {
+        out("waiting for a fetch");
+        fetch("a.html").then(function (response) {
+          out("got the fetch response");
+          // (normally you would do something with the fetch here)
+          wakeUp(42);
+        });
+      });
+    });
+
+Note that when using this form, you can't return a value from the function itself.
+Instead, you need to pass it as an argument to the ``wakeUp`` callback and
+propagate it by returning the result of ``Asyncify.handleSleep`` in ``do_fetch``
+itself.
+
 More on ``ASYNCIFY_IMPORTS``
 ############################
 
 As in the above example, you can add JS functions that do an async operation but
-look synchronous from the perspective of C. The key thing is to add such methods
-to ``ASYNCIFY_IMPORTS``, regardless of whether the JS function is from a JS
-library or ``EM_JS``. That list of imports is the list of imports to the wasm
-module that the Asyncify instrumentation must be aware of. Giving it that list
-tells it that all other JS calls will **not** do an async operation, which lets
-it not add overhead where it isn't needed.
-
-Returning values
-################
-
-You can also return values from async JS functions. Here is an example:
-
-.. code-block:: cpp
-
-    // example.c
-    #include <emscripten.h>
-    #include <stdio.h>
-
-    EM_JS(int, get_digest_size, (const char* str), {
-      // Note how we return the output of handleAsync() here.
-      return Asyncify.handleAsync(async () => {
-        const text = UTF8ToString(str);
-        const encoder = new TextEncoder();
-        const data = encoder.encode(text);
-        out("ask for digest for " + text);
-        const digestValue = await window.crypto.subtle.digest("SHA-256", data);
-        out("got digest of length " + digestValue.byteLength);
-        // Return the value as you normally would.
-        return digestValue.byteLength;
-      });
-    });
-
-    int main() {
-      const char* silly = "some silly text";
-      printf("%s's digest size is: %d\n", silly, get_digest_size(silly));
-      return 0;
-    }
-
-You can build this with
-
-::
-
-    emcc example.c -s ASYNCIFY=1 -s 'ASYNCIFY_IMPORTS=["get_digest_size"]' -o a.html -O2
-
-This example calls the Promise-returning ``window.crypto.subtle()`` API (the
-example is based off of
-`this MDN example <https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#Basic_example>`_
-).
-
-Note that we must propagate the value returned from ``handleAsync()``. The calling C code then
-gets it normally, after the Promise completes.
-
-If you're using the ``handleSleep`` API, the value needs to be also passed to the ``wakeUp`` callback, instead of being returned from our handler:
-
-.. code-block:: cpp
-
-    // ...
-    return Asyncify.handleSleep(wakeUp => {
-      const text = UTF8ToString(str);
-      const encoder = new TextEncoder();
-      const data = encoder.encode(text);
-      out("ask for digest for " + text);
-      window.crypto.subtle.digest("SHA-256", data).then(digestValue => {
-        out("got digest of length " + digestValue.byteLength);
-        // Return the value by sending it to wakeUp(). It will then be returned
-        // from handleSleep() on the outside.
-        wakeUp(digestValue.byteLength);
-      });
-    });
-    // ...
+look synchronous from the perspective of C. If you don't use ``EM_ASYNC_JS``,
+it's vital to add such methods to ``ASYNCIFY_IMPORTS``. That list of imports is
+the list of imports to the wasm module that the Asyncify instrumentation must be
+aware of. Giving it that list tells it that all other JS calls will **not** do
+an async operation, which lets it not add overhead where it isn't needed.
 
 Usage with Embind
 #################

src/library_async.js

@@ -80,8 +80,9 @@ mergeInto(LibraryManager.library, {
                 // indirect calls.
                 if (Asyncify.state !== originalAsyncifyState &&
                     ASYNCIFY_IMPORTS.indexOf(x) < 0 &&
+                    !x.startsWith('__asyncjs__') &&
                     !(x.startsWith('invoke_') && {{{ !ASYNCIFY_IGNORE_INDIRECT }}})) {
-                  throw 'import ' + x + ' was not in ASYNCIFY_IMPORTS, but changed the state';
+                  throw new Error('import ' + x + ' was not in ASYNCIFY_IMPORTS, but changed the state');
                 }
               }
             }

system/include/emscripten/em_js.h

@@ -57,10 +57,15 @@
 // emJsFuncs metadata is read in emscripten.py's create_em_js, which creates an
 // array of JS function strings to be included in the JS output.
 
-#define EM_JS(ret, name, params, ...)                                                              \
+#define _EM_JS(ret, c_name, js_name, params, code)                                                 \
   _EM_JS_CPP_BEGIN                                                                                 \
-  ret name params EM_IMPORT(name);                                                                 \
+  ret c_name params EM_IMPORT(js_name);                                                            \
   EMSCRIPTEN_KEEPALIVE                                                                             \
-  __attribute__((section("em_js"), aligned(1))) char __em_js__##name[] =                           \
-    #params "<::>" #__VA_ARGS__;                                                                   \
+  __attribute__((section("em_js"), aligned(1))) char __em_js__##js_name[] =                        \
+    #params "<::>" code;                                                                           \
   _EM_JS_CPP_END
+
+#define EM_JS(ret, name, params, ...) _EM_JS(ret, name, name, params, #__VA_ARGS__)
+
+#define EM_ASYNC_JS(ret, name, params, ...) _EM_JS(ret, name, __asyncjs__##name, params,          \
+  "{ return Asyncify.handleAsync(async () => " #__VA_ARGS__ "); }")

tests/core/test_em_async_js.c

@@ -0,0 +1,12 @@
+#include <emscripten.h>
+#include <stdio.h>
+
+EM_ASYNC_JS(double, foo, (int timeout), {
+	await new Promise(resolve => setTimeout(resolve, timeout));
+	return 4.2;
+});
+
+int main() {
+	printf("foo returned: %f\n", foo(10));
+	return 0;
+}

tests/core/test_em_async_js.out

@@ -0,0 +1 @@
+foo returned: 4.200000

tests/test_core.py

@@ -8560,6 +8560,13 @@ NODEFS is no longer included by default; build with -lnodefs.js
     self.emcc_args += args
     self.do_core_test('embind_lib_with_asyncify.cpp')
 
+  @no_asan('asyncify stack operations confuse asan')
+  def test_em_async_js(self):
+    self.uses_es6 = True
+    self.set_setting('ASYNCIFY')
+    self.maybe_closure()
+    self.do_core_test('test_em_async_js.c')
+
 
 # Generate tests for everything
 def make_run(name, emcc_args, settings=None, env=None):

tests/test_other.py

@@ -9900,7 +9900,7 @@ Module.arguments has been replaced with plain arguments_ (the initial value can
   # Compile-test for -s USE_WEBGPU=1 and library_webgpu.js.
   def test_webgpu_compiletest(self):
     for args in [[], ['-s', 'ASSERTIONS'], ['-s', 'MAIN_MODULE=1']]:
-      self.run_process([EMXX, test_file('webgpu_dummy.cpp'), '-s', 'USE_WEBGPU', '-s', 'ASYNCIFY', '-s', 'ASYNCIFY_IMPORTS=["init_js_device"]'] + args)
+      self.run_process([EMXX, test_file('webgpu_dummy.cpp'), '-s', 'USE_WEBGPU', '-s', 'ASYNCIFY'] + args)
 
   def test_signature_mismatch(self):
     create_file('a.c', 'void foo(); int main() { foo(); return 0; }')

tests/webgpu_dummy.cpp

@@ -38,12 +38,10 @@ private:
   int mHandle;
 };
 
-EM_JS(int, init_js_device, (), {
-  return Asyncify.handleAsync(async () => {
-    const adapter = await navigator.gpu.requestAdapter();
-    const device = await adapter.requestDevice();
-    return JsValStore.add(device);
-  });
+EM_ASYNC_JS(int, init_js_device, (), {
+  const adapter = await navigator.gpu.requestAdapter();
+  const device = await adapter.requestDevice();
+  return JsValStore.add(device);
 });
 
 wgpu::Device init_device() {