evgeny pavlov 78408b8837 Optimize alignments (#703) ... * Add fast Moses tokenizer * Tokenize corpus and remap alignments * Use moses tokenizer in taskcluster * Add tests for index mapping * Add packaged to build fast moses tokenizer * Fix an issue with LD_LIBRARY_PATH for fast moses tokenizer * Rename tokenization function * Rename chunking parameter * Relock poetry * Rerun linter		2024-08-29 13:03:01 -07:00
..
Dockerfile	Optimize alignments (#703)	2024-08-29 13:03:01 -07:00