* Rework WASM compilation options
Necessary to work with newer versions of emscripten that are more picky about which option goes to the compiler, and which to the linker. Also took the opportunity to remove the need for the patching of the bergamot-translation-worker.js file, this can now easily be done through supported apis. Furthermore, I tried to downsize the generated javascript and wasm code a bit.
Initial estimates show that bergamot-translator compiled with emscripten 3.0.0 runs at about 3x the speed of 2.0.9 (when using embedded intgemm). Speed-up when using mozIntGemm is less dramatic.
* Updated marian-dev submodule
* Revert changes specific to patching external gemm modules for wasm
* Better Compilation and Link flags
- Added "-O3" optimization flag for linking as well
- "-g2" only for release and debug builds
- "-g1" for release builds
- Replaced deprecated "--bind" flag with "-lembind"
- Removed redundant link flag
* Upgraded emsdk to 3.1.8
* Enclosed EXPORTED_FUNCTIONS values in a list
* Fixed the remaining 2.0.9 reference in circle ci build script
* Updated README
Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
* Use a more vanilla windows workflow from translateLocally, remove the
complicated lukka/*. Also removes port overrides in the overall upgrade.
* Disable vcpkg binary caching
* Remove PCRE library hacks after upstream ssplit improvements
Fixes the docs workflow which is failing after pip is picking up Jinja 3.20.
We only need >=2.3, this one sets it to 3.0.3 builds were successful last.
Quality scores for HTML translation exposed as <font
x-bergamot-sentence-score=""> and <font x-bergamot-word-score=""> tags
in the HTML output. While this increases the size of the HTML returned,
the resulting rendered HTML can easily be styled to show the scores.
With Javascript or CSS, developers can easily have some interface based
on these extra attributes.
Also includes updates to the test page to show a proof-of-concept
demonstration.
Fixes: #355
Deprecates cacheEnabled parameter to be replaced with cacheSize=0.
Python bindings, Documentation in comments and tests updated to reflect
this change.
Exposes the fields corresponding to cache via embind as a value object.
The equivalent object-based syntax in worker.js allows propagation
from JS.
Fixes: #351
See also: mozilla/firefox-translations#96
- Prefer spreading markup over a full word.
- Ignore certain tags that are unlikely to be supposed to be translated,
such as `<code>` and `<samp>`.
- Never treat `<wbr>` as a space.
- Allow for inconsistent cases in tag names.
- Fix bug where void elements were inserted multiple times.
- Better handling of whitespace around punctuation.
- Ignore parsing `<noscript>` to be compatible with Firefox.
- Improvements to documentation and readability of `HTML` and `Scanner`
classes.
Fixes: #313, #339
* Create github release via circleci only for mozilla fork
- The extension uses mozilla fork for translator artifacts
-- Hence create github release via circleci only when
running in mozilla fork
* Small refactoring in ci script
Hide `cache-mutex-buckets` from the user. Now configured to be equal to number
of workers. Python bindings which had exposed these are modified to reflect
the API change. `std::optional` enabled on cache, constructed only if enabled.
Pointers used are replaced with an equivalent `std::optional.`
Fixes: #317
Fixes memory leak
ifdef for -fno-exceptions including clang-cl
Move spacing back to intgemm upstream
Co-authored-by: Jerin Philip <jerin.philip@research.iiit.ac.in>
Changes signature of BlockingService::{translate,pivot}Multiple
functions to take per input options, so a mix of HTML and plaintext
can be sent from the extension. Templating over testing is adjusted
to allow for continuous evaluations by modifying the test code.
Updates WebAssembly bindings to reflect the change in signature
and the javascript test-page to work with the new bindings.
This change lacks an accompanying test specific to the mixed HTML
and plaintext inputs.
Fixes: #345
See also: mozilla/firefox-translations#94
Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
Changes `ABORT` on non `.bin` model to an additional check for a `.npz`
extension. If `.bin`, the fast load path is activated by returning `AlignedMemory`.
Otherwise, the return of empty `AlignedMemory` causes fallback to
filesystem-based loads.
BRT: A test that checks if translation using `.npz` is approximately similar to
that of default CLI translation is checked in to ensure stability going ahead.
Previously, we only supported `.bin` models' loading via a fast mmap
path. While we had the underlying capability to load non `.bin` models, this
was not exposed, encouraging fast loads. Loading `.npz` models are helpful
for quick debugging and broader coverage of models available, which will
enhance user experience at translateLocally and python bindings.
Fixes#341.
See also: XapaJIaMnu/translateLocally#89
This reverts commit 62ff781ed4.
Sorry I should have realized Jerin was only amending python and
therefore this didn't break WASM.
Apologies to Jerin on this.
Brings in the previously wasm.yml into python.yml and the new file is
renamed as build.yml.
python.yml already had a version and pre-release jobs. These jobs
downloaded artefacts from prior ran jobs (python wheel builds). The
newly attached emscripten build now uploads artefacts of a WebAssembly
binary and javascript file which are fed into the release and
pre-release jobs in addition to the existing python builds.
Enables ccache for emscripten. The configuration uses pyiodide for a
reference (https://github.com/pyodide/pyodide/pull/1805).
Two workflows to run on macOS and Ubuntu, reduced to one on Ubuntu. As
emscripten and the target is cross-platform, also macOS runners being
limited - it makes sense to have this removed.
Upload artefact enabled in preparation for a release action to be
scheduled which will upload the bergamot*.wasm and bergamot*.js for
consumption.
Previously BlockingService pivoting missed preproc and postproc for HTML
leading to issues in WebAssembly API. This change adds fixes for the
same, along with test coverage for the functionality over both async and
blocking services.
Imports python bindings and associated sources incubated in
https://github.com/jerinphilip/lemonade to bergamot-translator. Adds
a pybind11 dependency for python bindings.
Following the import, the python build is integrated into the existing
CMake based build system here. There is a command-line application
provided through python which provides the ability to fetch and prepare
models from model-repositories (like browsermt/students or OPUS).
Wheels built for a few common operating systems are provided via GitHub
releases through automated actions configured to run at tagged semantic
versions and pushes to main.
The documentation for python is also integrated into our existing
documentation setup. Previous documentation GitHub action is now
configured to run behind python builds in Ubuntu 18.04 Python3.7,
in order to pick up the packaged as a wheel bergamot module and the
sphinx documentation using the python module.
Formatting checks of black, isort with profile black and a pytype type
checker is configured for the python component residing in this repository.
Add a way to AsyncService to shut down without finishing the full queue
through `AsyncService::clear()`. The default behaviour is that
`AsyncService::~AsyncService()` will wait for any pending translation
requests to finish.
One can call `AsyncService::clear()` before the calls to the destructor
to ensure there is no work for the service to finish before the workers
can stop and join. Marian batches that are already in progress will not
stop. We are not trying to cause interrupts in threads or something that
complex. However, these single batches often do not take that long to
complete.
Changes:
- Add clear() to AsyncService
- Add clear() to BatchingPool
- Documentation
See also: XapaJIaMnu/translateLocally#80
Provides an additional constructor which takes care of the bundle
loading inside the boundary of the source here, when a configuration
file is supplied from a client like translateLocally or python bindings.
Once the config file is read, we have access to the information required
to construct the MemoryBundle.
- The command-line application supplied from here, app/bergamot is
configured to use the fast-load path now.
- Changes to binary-loading additionally revealed a bug in the
example-run script used in docs and tied to CI and the fix is
included.
- Shortlist is made optional in the memory bundle, making changes to
getModelMemoryFromConfig.
Fixes#304.
Fixes#306.
See also: XapaJIaMnu/translateLocally#82.
Allow the self-closing `/>` end for void tags. For non-void tags these
were already "allowed" due to how the HTML parser works, but for
elements where they actually occur, like `<br/>`, they caused a parse
error. Support for them was not implemented since we only expect valid
HTML5, e.g. the output of Firefox' Element.innerHTML.
Use case: TranslateLocally uses Qt's HTML representation of rich text.
That HTML uses self-closing tags like `<meta .../>` and `<br/>`.
Implementing a string replace operation that would only match these
elements without parsing HTML is tricky. Fixing it in
bergamot-translator is not.
Implementation: Currently `<img>` is marked as a void tag (an element
which cannot have children or text, and therefore treated differently.
Since void tags normally have no close tag, they are treated as
immediately closed. The HTML parser we use reads `<img/>` as
`<img></img>` which thus causes a problem since now we close an element
that was never open, to begin with.
This fix ignores the `TT_TAG_END` token from the parser when the tag
name is that of a void tag.
Translates a text from source-language to target-language through a
pivot-language. Effectively runs models in series, while having the
following additional benefits compared to when `Service::translate(...)`
would be used repeatedly.
1. Consistency in sentences between source and target. Consistent
creation of the alignment matrix for use in downstream tasks like
tag-translation.
2. Efficient sentence-splitting (does not sentence-split twice, creating
inconsistencies).
3. The `Response` generated can be used as if it were coming through
`translate(...)`, eliminating any need for additional code for clients
in JS or python or C++.
`AsyncService::pivot(...)` is provisioned for C++ multi-threaded setting
and `BlockingService::pivotMultiple(...)` provisioned for blocking
use-case targeted at WebAssembly.
# [BRT]: Test additions, accompanying fixes
For `AsyncService` for a test-case involving of en->es, es->en (same
vocabulary, another one might be more coverage but is too much work).
1. Asserts the Alignment generated after pivoting is a probability
distribution over source tokens given target.
2. Outputs the sentences going from en->en, which should stay consistent
over continuous development to ensure nothing breaks.
3. An accuracy minimum of 70% of token matches from source to target
calibrated on the standard bergamot input text is additionally present,
ensuring that the English tokens at start and end match exactly.
# HTML Pipeline
This PR reworks the HTML translation pipeline to be outside
response-construction via callbacks.
* Convert marian-integration markdown to rst
* Convert native run into a script, include in rst
* Check with CI that the native running example works without fail