mozilla
/
firefox-translations-models
зеркало из https://github.com/mozilla/firefox-translations-models.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Add English to Czech (#145) 

* Remove old bergamot files

* Remove old evaluations as well

* Add the models

* Update evaluation results [skip ci]

* Update model registry [skip ci]

---------

Co-authored-by: CircleCI evaluation job <ci-models-evaluation@firefox-translations>
This commit is contained in:
Greg Tatum 2024-04-04 04:08:49 -05:00 коммит произвёл GitHub
Родитель 1a85206014
Коммит c923ed0bc0
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: B5690EEEBB952194
64 изменённых файлов: 37289 добавлений и 37298 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

24
evaluation/bleu-results.md
Просмотреть файл

@ -61,12 +61,12 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
| Translator/Dataset | cs-en | tr-en | sk-en | en-et | en-fi | en-it | en-gl | en-hr | fr-en | en-pt | ro-en | hu-en | th-en | en-zh | vi-en | en-ja | ru-en | mt-en | en-sr | en-nb | en-hu | en-el | el-en | fi-en | en-da | en-nl | en-mt | en-sv | lv-en | en-th | en-ru | en-fa | bs-en | et-en | nb-en | en-ca | nl-en | en-lt | en-tr | bg-en | ko-en | uk-en | en-es | fa-en | ca-en | en-uk | en-vi | is-en | en-bg | en-lv | gl-en | zh-en | en-hi | en-bs | en-cs | hr-en | en-ro | sr-en | sv-en | en-ko | hi-en | en-is | id-en | de-en | da-en | it-en | pl-en | sl-en | en-fr | en-pl | en-id | pt-en | en-sk | es-en | en-sl | ja-en | lt-en | en-de |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bergamot | 31.07 | 30.08 | N/A | 25.50 | N/A | 29.77 | N/A | N/A | 35.43 | 49.85 | N/A | 26.53 | N/A | N/A | N/A | N/A | 33.69 | 42.70 | N/A | N/A | 20.32 | N/A | 32.53 | 29.44 | N/A | 27.30 | N/A | N/A | N/A | N/A | 29.44 | 17.30 | N/A | 32.37 | 37.60 | 42.20 | 29.65 | N/A | N/A | 38.50 | N/A | 35.93 | 32.41 | 28.70 | 38.00 | 26.30 | N/A | 23.40 | 42.10 | N/A | N/A | N/A | N/A | N/A | 24.76 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 33.16 | N/A | 32.60 | 27.87 | 29.10 | 36.01 | 22.27 | N/A | 44.87 | N/A | 32.38 | N/A | N/A | 30.53 | 32.01 |
| google | 33.36 (+2.29, +7.36%) | 39.58 (+9.50, +31.58%) | 43.05 | 28.60 (+3.10, +12.16%) | 27.43 | 28.97 (-0.80, -2.69%) | 37.60 | 33.90 | 37.81 (+2.38, +6.70%) | 53.75 (+3.90, +7.82%) | 44.53 | 31.30 (+4.77, +18.00%) | 30.95 | 45.31 | 40.55 | 33.00 | 38.20 (+4.51, +13.38%) | 59.15 (+16.45, +38.52%) | 40.00 | 35.05 | 24.80 (+4.48, +22.02%) | 29.55 | 37.13 (+4.60, +14.14%) | 33.20 (+3.76, +12.76%) | 49.70 | 29.30 (+2.00, +7.33%) | 40.60 | 48.15 | 36.80 | 15.60 | 34.49 (+5.05, +17.15%) | 27.80 (+10.50, +60.69%) | 45.65 | 35.80 (+3.43, +10.61%) | 42.05 (+4.45, +11.84%) | 35.70 (-6.50, -15.40%) | 33.05 (+3.40, +11.47%) | 28.57 | 25.52 | 41.30 (+2.80, +7.27%) | 29.47 | 42.43 (+6.50, +18.09%) | 34.74 (+2.32, +7.17%) | 40.85 (+12.15, +42.33%) | 48.95 (+10.95, +28.82%) | 32.63 (+6.33, +24.08%) | 45.55 | 38.90 (+15.50, +66.24%) | 44.60 (+2.50, +5.94%) | 33.77 | 43.50 | 32.29 | 33.43 | 36.45 | 27.72 (+2.96, +11.95%) | 40.60 | 40.60 | 47.55 | 50.50 | 29.20 | 42.13 | 30.70 | 47.75 | 35.65 (+2.49, +7.52%) | 50.40 | 34.50 (+1.90, +5.83%) | 31.23 (+3.37, +12.08%) | 38.75 (+9.65, +33.16%) | 29.47 (-6.54, -18.15%) | 25.50 (+3.23, +14.52%) | 49.25 | 46.60 (+1.73, +3.86%) | 40.10 | 33.64 (+1.27, +3.91%) | 35.40 | 28.08 | 36.80 (+6.27, +20.52%) | 33.16 (+1.14, +3.58%) |
| microsoft | 34.67 (+3.61, +11.61%) | 39.16 (+9.08, +30.19%) | 41.95 | 28.47 (+2.97, +11.63%) | 29.27 | 32.30 (+2.53, +8.51%) | 35.10 | 32.43 | 39.13 (+3.70, +10.44%) | 50.15 (+0.30, +0.60%) | 45.53 | 31.03 (+4.50, +16.97%) | 31.25 | 43.42 | 39.30 | 30.72 | 38.38 (+4.68, +13.90%) | 56.75 (+14.05, +32.90%) | 1.00 | 34.95 | 26.05 (+5.73, +28.17%) | 29.95 | 37.33 (+4.80, +14.75%) | 35.09 (+5.64, +19.17%) | 49.65 | 28.80 (+1.50, +5.49%) | 44.20 | 50.00 | 32.67 | 15.05 | 33.62 (+4.18, +14.21%) | 20.50 (+3.20, +18.50%) | 42.65 | 36.17 (+3.80, +11.74%) | 42.90 (+5.30, +14.10%) | 45.25 (+3.05, +7.23%) | 32.60 (+2.95, +9.95%) | 26.10 | 32.18 | 41.20 (+2.70, +7.01%) | 29.50 | 42.30 (+6.37, +17.72%) | 33.76 (+1.35, +4.17%) | 36.15 (+7.45, +25.96%) | 46.50 (+8.50, +22.37%) | 32.03 (+5.73, +21.80%) | 44.95 | 38.17 (+14.77, +63.11%) | 38.55 (-3.55, -8.43%) | 33.20 | 40.20 | 32.09 | 33.73 | 32.65 | 28.26 (+3.50, +14.14%) | 38.50 | 42.07 | 1.60 | 51.90 | 29.87 | 40.57 | 30.83 | 47.00 | 37.73 (+4.57, +13.79%) | 51.05 | 34.55 (+1.95, +5.98%) | 31.83 (+3.97, +14.23%) | 34.65 (+5.55, +19.07%) | 36.48 (+0.47, +1.31%) | 24.83 (+2.57, +11.53%) | 50.15 | 46.47 (+1.60, +3.57%) | 39.20 | 32.93 (+0.56, +1.72%) | 33.60 | 26.32 | 32.50 (+1.97, +6.44%) | 33.54 (+1.53, +4.79%) |
| argos | 20.55 (-10.52, -33.86%) | 20.42 (-9.66, -32.11%) | 24.80 | 23.17 (-2.33, -9.15%) | 19.33 | 27.17 (-2.60, -8.73%) | N/A | N/A | 34.04 (-1.39, -3.93%) | 43.80 (-6.05, -12.14%) | N/A | 12.97 (-13.55, -51.08%) | N/A | 29.00 | N/A | 15.70 | 25.76 (-7.93, -23.54%) | N/A | N/A | N/A | 17.40 (-2.93, -14.39%) | 26.25 | 31.83 (-0.70, -2.15%) | 22.14 (-7.30, -24.79%) | 36.80 | 24.90 (-2.40, -8.79%) | N/A | 42.05 | N/A | N/A | 27.89 (-1.55, -5.26%) | 21.30 (+4.00, +23.12%) | N/A | 31.33 (-1.03, -3.19%) | 38.85 (+1.25, +3.32%) | 30.40 (-11.80, -27.96%) | 29.90 (+0.25, +0.84%) | N/A | 16.18 | 36.10 (-2.40, -6.23%) | 12.93 | 26.27 (-9.67, -26.90%) | 29.93 (-2.49, -7.67%) | 27.90 (-0.80, -2.79%) | 42.30 (+4.30, +11.32%) | 15.73 (-10.57, -40.18%) | N/A | N/A | 39.45 (-2.65, -6.29%) | N/A | N/A | 20.41 | 25.47 | N/A | 15.65 (-9.11, -36.80%) | N/A | N/A | N/A | 44.35 | 14.10 | 25.70 | N/A | 29.00 | 30.97 (-2.18, -6.59%) | 38.20 | 31.07 (-1.53, -4.68%) | 27.67 (-0.20, -0.72%) | N/A | 33.44 (-2.57, -7.14%) | 19.67 (-2.60, -11.68%) | 34.45 | 42.77 (-2.10, -4.68%) | 20.35 | 30.70 (-1.68, -5.18%) | N/A | 12.43 | N/A | 26.56 (-5.46, -17.04%) |
| nllb | 22.66 (-8.41, -27.06%) | 22.90 (-7.18, -23.87%) | 26.95 | 17.57 (-7.93, -31.11%) | 17.41 | 28.30 (-1.47, -4.93%) | 31.40 | 22.53 | 34.95 (-0.48, -1.36%) | 45.65 (-4.20, -8.43%) | 31.57 | 15.85 (-10.68, -40.25%) | 25.45 | 27.04 | 31.70 | 15.58 | 33.42 (-0.28, -0.82%) | 26.95 (-15.75, -36.89%) | 25.60 | 28.95 | 16.60 (-3.72, -18.33%) | 23.30 | 32.80 (+0.27, +0.82%) | 13.96 (-15.49, -52.60%) | 40.30 | 24.35 (-2.95, -10.81%) | 28.60 | 38.20 | 15.40 | 7.45 | 27.47 (-1.98, -6.71%) | 18.80 (+1.50, +8.67%) | 26.30 | 11.40 (-20.97, -64.78%) | 27.60 (-10.00, -26.60%) | 35.45 (-6.75, -16.00%) | 25.55 (-4.10, -13.83%) | 16.30 | 19.00 | 36.40 (-2.10, -5.45%) | 22.90 | 33.00 (-2.93, -8.16%) | 32.04 (-0.38, -1.16%) | 29.75 (+1.05, +3.66%) | 37.40 (-0.60, -1.58%) | 21.97 (-4.33, -16.48%) | 37.15 | 17.63 (-5.77, -24.64%) | 34.05 (-8.05, -19.12%) | 14.30 | 35.00 | 21.20 | 27.03 | 24.50 | 21.30 (-3.46, -13.97%) | 23.75 | 32.23 | 33.60 | 31.45 | 22.37 | 35.43 | 18.17 | 38.95 | 25.39 (-7.77, -23.42%) | 32.20 | 30.15 (-2.45, -7.52%) | 22.83 (-5.03, -18.06%) | 17.40 (-11.70, -40.21%) | 35.81 (-0.20, -0.56%) | 18.33 (-3.93, -17.66%) | 41.50 | 40.00 (-4.87, -10.85%) | 26.00 | 31.91 (-0.47, -1.44%) | 23.25 | 16.80 | 16.97 (-13.57, -44.43%) | 28.09 (-3.92, -12.24%) |
| opusmt | 29.60 (-1.47, -4.73%) | 28.18 (-1.90, -6.32%) | 33.90 | 23.53 (-1.97, -7.71%) | 26.70 | 28.50 (-1.27, -4.26%) | 19.35 | N/A | 35.90 (+0.47, +1.32%) | N/A | N/A | 26.15 (-0.38, -1.41%) | 20.65 | 27.44 | 25.75 | N/A | 29.65 (-4.04, -11.99%) | N/A | N/A | N/A | 20.57 (+0.25, +1.23%) | 24.70 | N/A | 28.30 (-1.14, -3.88%) | 43.00 | 25.30 (-2.00, -7.33%) | N/A | 43.75 | 28.00 | N/A | 25.82 (-3.62, -12.28%) | N/A | N/A | 31.50 (-0.87, -2.68%) | N/A | 29.70 (-12.50, -29.62%) | 29.65 (0.00, 0.00%) | N/A | N/A | 36.30 (-2.20, -5.71%) | 20.13 | 27.77 (-8.17, -22.73%) | 32.73 (+0.31, +0.96%) | N/A | 29.75 (-8.25, -21.71%) | 18.80 (-7.50, -28.52%) | 26.35 | 25.37 (+1.97, +8.40%) | 32.50 (-9.60, -22.80%) | N/A | 21.40 | 20.84 | 12.43 | N/A | 23.99 (-0.77, -3.11%) | N/A | 31.37 | N/A | 46.10 | N/A | 17.40 | 17.33 | 34.25 | 34.09 (+0.93, +2.82%) | 45.10 | 32.25 (-0.35, -1.07%) | 26.80 (-1.07, -3.83%) | N/A | 36.31 (+0.30, +0.83%) | N/A | 37.85 | N/A | 30.35 | 32.58 (+0.20, +0.62%) | N/A | 12.93 | N/A | 29.91 (-2.10, -6.56%) |
| bergamot | 31.07 | 30.08 | N/A | 25.50 | N/A | 29.77 | N/A | N/A | 35.43 | 49.85 | N/A | 26.53 | N/A | N/A | N/A | N/A | 33.69 | 42.70 | N/A | N/A | 20.32 | N/A | 32.53 | 29.44 | N/A | 27.30 | N/A | N/A | N/A | N/A | 29.44 | 17.30 | N/A | 32.37 | 37.60 | 42.20 | 29.65 | N/A | N/A | 38.50 | N/A | 35.93 | 32.41 | 28.70 | 38.00 | 26.30 | N/A | 23.40 | 42.10 | N/A | N/A | N/A | N/A | N/A | 25.35 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 33.16 | N/A | 32.60 | 27.87 | 29.10 | 36.01 | 22.27 | N/A | 44.87 | N/A | 32.38 | N/A | N/A | 30.53 | 32.01 |
| google | 33.36 (+2.29, +7.36%) | 39.58 (+9.50, +31.58%) | 43.05 | 28.60 (+3.10, +12.16%) | 27.43 | 28.97 (-0.80, -2.69%) | 37.60 | 33.90 | 37.81 (+2.38, +6.70%) | 53.75 (+3.90, +7.82%) | 44.53 | 31.30 (+4.77, +18.00%) | 30.95 | 45.31 | 40.55 | 33.00 | 38.20 (+4.51, +13.38%) | 59.15 (+16.45, +38.52%) | 40.00 | 35.05 | 24.80 (+4.48, +22.02%) | 29.55 | 37.13 (+4.60, +14.14%) | 33.20 (+3.76, +12.76%) | 49.70 | 29.30 (+2.00, +7.33%) | 40.60 | 48.15 | 36.80 | 15.60 | 34.49 (+5.05, +17.15%) | 27.80 (+10.50, +60.69%) | 45.65 | 35.80 (+3.43, +10.61%) | 42.05 (+4.45, +11.84%) | 35.70 (-6.50, -15.40%) | 33.05 (+3.40, +11.47%) | 28.57 | 25.52 | 41.30 (+2.80, +7.27%) | 29.47 | 42.43 (+6.50, +18.09%) | 34.74 (+2.32, +7.17%) | 40.85 (+12.15, +42.33%) | 48.95 (+10.95, +28.82%) | 32.63 (+6.33, +24.08%) | 45.55 | 38.90 (+15.50, +66.24%) | 44.60 (+2.50, +5.94%) | 33.77 | 43.50 | 32.29 | 33.43 | 36.45 | 27.72 (+2.36, +9.33%) | 40.60 | 40.60 | 47.55 | 50.50 | 29.20 | 42.13 | 30.70 | 47.75 | 35.65 (+2.49, +7.52%) | 50.40 | 34.50 (+1.90, +5.83%) | 31.23 (+3.37, +12.08%) | 38.75 (+9.65, +33.16%) | 29.47 (-6.54, -18.15%) | 25.50 (+3.23, +14.52%) | 49.25 | 46.60 (+1.73, +3.86%) | 40.10 | 33.64 (+1.27, +3.91%) | 35.40 | 28.08 | 36.80 (+6.27, +20.52%) | 33.16 (+1.14, +3.58%) |
| microsoft | 34.67 (+3.61, +11.61%) | 39.16 (+9.08, +30.19%) | 41.95 | 28.47 (+2.97, +11.63%) | 29.27 | 32.30 (+2.53, +8.51%) | 35.10 | 32.43 | 39.13 (+3.70, +10.44%) | 50.15 (+0.30, +0.60%) | 45.53 | 31.03 (+4.50, +16.97%) | 31.25 | 43.42 | 39.30 | 30.72 | 38.38 (+4.68, +13.90%) | 56.75 (+14.05, +32.90%) | 1.00 | 34.95 | 26.05 (+5.73, +28.17%) | 29.95 | 37.33 (+4.80, +14.75%) | 35.09 (+5.64, +19.17%) | 49.65 | 28.80 (+1.50, +5.49%) | 44.20 | 50.00 | 32.67 | 15.05 | 33.62 (+4.18, +14.21%) | 20.50 (+3.20, +18.50%) | 42.65 | 36.17 (+3.80, +11.74%) | 42.90 (+5.30, +14.10%) | 45.25 (+3.05, +7.23%) | 32.60 (+2.95, +9.95%) | 26.10 | 32.18 | 41.20 (+2.70, +7.01%) | 29.50 | 42.30 (+6.37, +17.72%) | 33.76 (+1.35, +4.17%) | 36.15 (+7.45, +25.96%) | 46.50 (+8.50, +22.37%) | 32.03 (+5.73, +21.80%) | 44.95 | 38.17 (+14.77, +63.11%) | 38.55 (-3.55, -8.43%) | 33.20 | 40.20 | 32.09 | 33.73 | 32.65 | 28.26 (+2.91, +11.46%) | 38.50 | 42.07 | 1.60 | 51.90 | 29.87 | 40.57 | 30.83 | 47.00 | 37.73 (+4.57, +13.79%) | 51.05 | 34.55 (+1.95, +5.98%) | 31.83 (+3.97, +14.23%) | 34.65 (+5.55, +19.07%) | 36.48 (+0.47, +1.31%) | 24.83 (+2.57, +11.53%) | 50.15 | 46.47 (+1.60, +3.57%) | 39.20 | 32.93 (+0.56, +1.72%) | 33.60 | 26.32 | 32.50 (+1.97, +6.44%) | 33.54 (+1.53, +4.79%) |
| argos | 20.55 (-10.52, -33.86%) | 20.42 (-9.66, -32.11%) | 24.80 | 23.17 (-2.33, -9.15%) | 19.33 | 27.17 (-2.60, -8.73%) | N/A | N/A | 34.04 (-1.39, -3.93%) | 43.80 (-6.05, -12.14%) | N/A | 12.97 (-13.55, -51.08%) | N/A | 29.00 | N/A | 15.70 | 25.76 (-7.93, -23.54%) | N/A | N/A | N/A | 17.40 (-2.93, -14.39%) | 26.25 | 31.83 (-0.70, -2.15%) | 22.14 (-7.30, -24.79%) | 36.80 | 24.90 (-2.40, -8.79%) | N/A | 42.05 | N/A | N/A | 27.89 (-1.55, -5.26%) | 21.30 (+4.00, +23.12%) | N/A | 31.33 (-1.03, -3.19%) | 38.85 (+1.25, +3.32%) | 30.40 (-11.80, -27.96%) | 29.90 (+0.25, +0.84%) | N/A | 16.18 | 36.10 (-2.40, -6.23%) | 12.93 | 26.27 (-9.67, -26.90%) | 29.93 (-2.49, -7.67%) | 27.90 (-0.80, -2.79%) | 42.30 (+4.30, +11.32%) | 15.73 (-10.57, -40.18%) | N/A | N/A | 39.45 (-2.65, -6.29%) | N/A | N/A | 20.41 | 25.47 | N/A | 15.65 (-9.71, -38.28%) | N/A | N/A | N/A | 44.35 | 14.10 | 25.70 | N/A | 29.00 | 30.97 (-2.18, -6.59%) | 38.20 | 31.07 (-1.53, -4.68%) | 27.67 (-0.20, -0.72%) | N/A | 33.44 (-2.57, -7.14%) | 19.67 (-2.60, -11.68%) | 34.45 | 42.77 (-2.10, -4.68%) | 20.35 | 30.70 (-1.68, -5.18%) | N/A | 12.43 | N/A | 26.56 (-5.46, -17.04%) |
| nllb | 22.66 (-8.41, -27.06%) | 22.90 (-7.18, -23.87%) | 26.95 | 17.57 (-7.93, -31.11%) | 17.41 | 28.30 (-1.47, -4.93%) | 31.40 | 22.53 | 34.95 (-0.48, -1.36%) | 45.65 (-4.20, -8.43%) | 31.57 | 15.85 (-10.68, -40.25%) | 25.45 | 27.04 | 31.70 | 15.58 | 33.42 (-0.28, -0.82%) | 26.95 (-15.75, -36.89%) | 25.60 | 28.95 | 16.60 (-3.72, -18.33%) | 23.30 | 32.80 (+0.27, +0.82%) | 13.96 (-15.49, -52.60%) | 40.30 | 24.35 (-2.95, -10.81%) | 28.60 | 38.20 | 15.40 | 7.45 | 27.47 (-1.98, -6.71%) | 18.80 (+1.50, +8.67%) | 26.30 | 11.40 (-20.97, -64.78%) | 27.60 (-10.00, -26.60%) | 35.45 (-6.75, -16.00%) | 25.55 (-4.10, -13.83%) | 16.30 | 19.00 | 36.40 (-2.10, -5.45%) | 22.90 | 33.00 (-2.93, -8.16%) | 32.04 (-0.38, -1.16%) | 29.75 (+1.05, +3.66%) | 37.40 (-0.60, -1.58%) | 21.97 (-4.33, -16.48%) | 37.15 | 17.63 (-5.77, -24.64%) | 34.05 (-8.05, -19.12%) | 14.30 | 35.00 | 21.20 | 27.03 | 24.50 | 21.30 (-4.05, -15.99%) | 23.75 | 32.23 | 33.60 | 31.45 | 22.37 | 35.43 | 18.17 | 38.95 | 25.39 (-7.77, -23.42%) | 32.20 | 30.15 (-2.45, -7.52%) | 22.83 (-5.03, -18.06%) | 17.40 (-11.70, -40.21%) | 35.81 (-0.20, -0.56%) | 18.33 (-3.93, -17.66%) | 41.50 | 40.00 (-4.87, -10.85%) | 26.00 | 31.91 (-0.47, -1.44%) | 23.25 | 16.80 | 16.97 (-13.57, -44.43%) | 28.09 (-3.92, -12.24%) |
| opusmt | 29.60 (-1.47, -4.73%) | 28.18 (-1.90, -6.32%) | 33.90 | 23.53 (-1.97, -7.71%) | 26.70 | 28.50 (-1.27, -4.26%) | 19.35 | N/A | 35.90 (+0.47, +1.32%) | N/A | N/A | 26.15 (-0.38, -1.41%) | 20.65 | 27.44 | 25.75 | N/A | 29.65 (-4.04, -11.99%) | N/A | N/A | N/A | 20.57 (+0.25, +1.23%) | 24.70 | N/A | 28.30 (-1.14, -3.88%) | 43.00 | 25.30 (-2.00, -7.33%) | N/A | 43.75 | 28.00 | N/A | 25.82 (-3.62, -12.28%) | N/A | N/A | 31.50 (-0.87, -2.68%) | N/A | 29.70 (-12.50, -29.62%) | 29.65 (0.00, 0.00%) | N/A | N/A | 36.30 (-2.20, -5.71%) | 20.13 | 27.77 (-8.17, -22.73%) | 32.73 (+0.31, +0.96%) | N/A | 29.75 (-8.25, -21.71%) | 18.80 (-7.50, -28.52%) | 26.35 | 25.37 (+1.97, +8.40%) | 32.50 (-9.60, -22.80%) | N/A | 21.40 | 20.84 | 12.43 | N/A | 23.99 (-1.36, -5.38%) | N/A | 31.37 | N/A | 46.10 | N/A | 17.40 | 17.33 | 34.25 | 34.09 (+0.93, +2.82%) | 45.10 | 32.25 (-0.35, -1.07%) | 26.80 (-1.07, -3.83%) | N/A | 36.31 (+0.30, +0.83%) | N/A | 37.85 | N/A | 30.35 | 32.58 (+0.20, +0.62%) | N/A | 12.93 | N/A | 29.91 (-2.10, -6.56%) |
![Results](img/avg-bleu.png)
---
@ -831,12 +831,12 @@ Both absolute and relative differences in BLEU scores between Bergamot and other
| Translator/Dataset | wmt09 | wmt12 | wmt20 | wmt19 | flores-dev | wmt21 | wmt08 | wmt14 | wmt17 | wmt15 | wmt22 | wmt11 | wmt13 | wmt10 | wmt18 | wmt16 | flores-test |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bergamot | 20.80 | 18.80 | 32.70 | 27.10 | 30.10 | 19.50 | 19.00 | 28.70 | 23.40 | 25.40 | 31.70 | 20.70 | 23.40 | 20.90 | 22.70 | 25.80 | 30.20 |
| google | 22.60 (+1.80, +8.65%) | 20.70 (+1.90, +10.11%) | 35.50 (+2.80, +8.56%) | 27.20 (+0.10, +0.37%) | 34.10 (+4.00, +13.29%) | 21.80 (+2.30, +11.79%) | 20.50 (+1.50, +7.89%) | 31.20 (+2.50, +8.71%) | 24.70 (+1.30, +5.56%) | 26.80 (+1.40, +5.51%) | 48.40 (+16.70, +52.68%) | 23.00 (+2.30, +11.11%) | 25.20 (+1.80, +7.69%) | 22.40 (+1.50, +7.18%) | 24.40 (+1.70, +7.49%) | 28.30 (+2.50, +9.69%) | 34.40 (+4.20, +13.91%) |
| microsoft | 25.00 (+4.20, +20.19%) | 22.90 (+4.10, +21.81%) | 34.10 (+1.40, +4.28%) | 27.20 (+0.10, +0.37%) | 33.50 (+3.40, +11.30%) | 22.00 (+2.50, +12.82%) | 22.60 (+3.60, +18.95%) | 31.90 (+3.20, +11.15%) | 25.60 (+2.20, +9.40%) | 27.40 (+2.00, +7.87%) | 42.10 (+10.40, +32.81%) | 25.30 (+4.60, +22.22%) | 27.70 (+4.30, +18.38%) | 24.30 (+3.40, +16.27%) | 24.90 (+2.20, +9.69%) | 29.90 (+4.10, +15.89%) | 34.00 (+3.80, +12.58%) |
| argos | 14.30 (-6.50, -31.25%) | 13.70 (-5.10, -27.13%) | 16.50 (-16.20, -49.54%) | 13.70 (-13.40, -49.45%) | 19.90 (-10.20, -33.89%) | 9.90 (-9.60, -49.23%) | 12.70 (-6.30, -33.16%) | 17.10 (-11.60, -40.42%) | 14.00 (-9.40, -40.17%) | 14.40 (-11.00, -43.31%) | 22.90 (-8.80, -27.76%) | 15.80 (-4.90, -23.67%) | 16.00 (-7.40, -31.62%) | 14.70 (-6.20, -29.67%) | 13.30 (-9.40, -41.41%) | 18.00 (-7.80, -30.23%) | 19.10 (-11.10, -36.75%) |
| nllb | 19.50 (-1.30, -6.25%) | 18.60 (-0.20, -1.06%) | 23.80 (-8.90, -27.22%) | 20.60 (-6.50, -23.99%) | 25.20 (-4.90, -16.28%) | 15.40 (-4.10, -21.03%) | 17.90 (-1.10, -5.79%) | 23.50 (-5.20, -18.12%) | 19.80 (-3.60, -15.38%) | 20.50 (-4.90, -19.29%) | 28.40 (-3.30, -10.41%) | 20.50 (-0.20, -0.97%) | 22.00 (-1.40, -5.98%) | 19.80 (-1.10, -5.26%) | 18.50 (-4.20, -18.50%) | 22.50 (-3.30, -12.79%) | 25.60 (-4.60, -15.23%) |
| opusmt | 21.60 (+0.80, +3.85%) | 20.40 (+1.60, +8.51%) | 27.50 (-5.20, -15.90%) | 22.60 (-4.50, -16.61%) | 28.90 (-1.20, -3.99%) | 17.00 (-2.50, -12.82%) | 19.50 (+0.50, +2.63%) | 27.30 (-1.40, -4.88%) | 21.90 (-1.50, -6.41%) | 23.70 (-1.70, -6.69%) | 30.90 (-0.80, -2.52%) | 22.70 (+2.00, +9.66%) | 24.50 (+1.10, +4.70%) | 21.80 (+0.90, +4.31%) | 21.60 (-1.10, -4.85%) | 26.50 (+0.70, +2.71%) | 29.40 (-0.80, -2.65%) |
| bergamot | 21.90 | 20.40 | 32.80 | 24.80 | 30.70 | 18.60 | 19.70 | 28.40 | 23.10 | 24.30 | 33.60 | 23.30 | 25.40 | 22.10 | 22.30 | 27.80 | 31.80 |
| google | 22.60 (+0.70, +3.20%) | 20.70 (+0.30, +1.47%) | 35.50 (+2.70, +8.23%) | 27.20 (+2.40, +9.68%) | 34.10 (+3.40, +11.07%) | 21.80 (+3.20, +17.20%) | 20.50 (+0.80, +4.06%) | 31.20 (+2.80, +9.86%) | 24.70 (+1.60, +6.93%) | 26.80 (+2.50, +10.29%) | 48.40 (+14.80, +44.05%) | 23.00 (-0.30, -1.29%) | 25.20 (-0.20, -0.79%) | 22.40 (+0.30, +1.36%) | 24.40 (+2.10, +9.42%) | 28.30 (+0.50, +1.80%) | 34.40 (+2.60, +8.18%) |
| microsoft | 25.00 (+3.10, +14.16%) | 22.90 (+2.50, +12.25%) | 34.10 (+1.30, +3.96%) | 27.20 (+2.40, +9.68%) | 33.50 (+2.80, +9.12%) | 22.00 (+3.40, +18.28%) | 22.60 (+2.90, +14.72%) | 31.90 (+3.50, +12.32%) | 25.60 (+2.50, +10.82%) | 27.40 (+3.10, +12.76%) | 42.10 (+8.50, +25.30%) | 25.30 (+2.00, +8.58%) | 27.70 (+2.30, +9.06%) | 24.30 (+2.20, +9.95%) | 24.90 (+2.60, +11.66%) | 29.90 (+2.10, +7.55%) | 34.00 (+2.20, +6.92%) |
| argos | 14.30 (-7.60, -34.70%) | 13.70 (-6.70, -32.84%) | 16.50 (-16.30, -49.70%) | 13.70 (-11.10, -44.76%) | 19.90 (-10.80, -35.18%) | 9.90 (-8.70, -46.77%) | 12.70 (-7.00, -35.53%) | 17.10 (-11.30, -39.79%) | 14.00 (-9.10, -39.39%) | 14.40 (-9.90, -40.74%) | 22.90 (-10.70, -31.85%) | 15.80 (-7.50, -32.19%) | 16.00 (-9.40, -37.01%) | 14.70 (-7.40, -33.48%) | 13.30 (-9.00, -40.36%) | 18.00 (-9.80, -35.25%) | 19.10 (-12.70, -39.94%) |
| nllb | 19.50 (-2.40, -10.96%) | 18.60 (-1.80, -8.82%) | 23.80 (-9.00, -27.44%) | 20.60 (-4.20, -16.94%) | 25.20 (-5.50, -17.92%) | 15.40 (-3.20, -17.20%) | 17.90 (-1.80, -9.14%) | 23.50 (-4.90, -17.25%) | 19.80 (-3.30, -14.29%) | 20.50 (-3.80, -15.64%) | 28.40 (-5.20, -15.48%) | 20.50 (-2.80, -12.02%) | 22.00 (-3.40, -13.39%) | 19.80 (-2.30, -10.41%) | 18.50 (-3.80, -17.04%) | 22.50 (-5.30, -19.06%) | 25.60 (-6.20, -19.50%) |
| opusmt | 21.60 (-0.30, -1.37%) | 20.40 (0.00, 0.00%) | 27.50 (-5.30, -16.16%) | 22.60 (-2.20, -8.87%) | 28.90 (-1.80, -5.86%) | 17.00 (-1.60, -8.60%) | 19.50 (-0.20, -1.02%) | 27.30 (-1.10, -3.87%) | 21.90 (-1.20, -5.19%) | 23.70 (-0.60, -2.47%) | 30.90 (-2.70, -8.04%) | 22.70 (-0.60, -2.58%) | 24.50 (-0.90, -3.54%) | 21.80 (-0.30, -1.36%) | 21.60 (-0.70, -3.14%) | 26.50 (-1.30, -4.68%) | 29.40 (-2.40, -7.55%) |
![Results](img/en-cs-bleu.png)
---

24
evaluation/comet-results.md
Просмотреть файл

@ -53,12 +53,12 @@ We also compare the systems using the `comet-compare` tool that calculates the s
| Translator/Dataset | cs-en | tr-en | sk-en | en-et | en-fi | en-it | en-gl | en-hr | fr-en | en-pt | ro-en | hu-en | th-en | en-zh | vi-en | en-ja | ru-en | mt-en | en-sr | en-nb | en-hu | en-el | el-en | fi-en | en-da | en-nl | en-mt | en-sv | lv-en | en-th | en-ru | en-fa | bs-en | et-en | nb-en | en-ca | nl-en | en-lt | en-tr | bg-en | ko-en | uk-en | en-es | fa-en | ca-en | en-uk | en-vi | is-en | en-bg | en-lv | gl-en | zh-en | en-hi | en-bs | en-cs | hr-en | en-ro | sr-en | sv-en | en-ko | hi-en | en-is | id-en | de-en | da-en | it-en | pl-en | sl-en | en-fr | en-pl | en-id | pt-en | en-sk | es-en | en-sl | ja-en | lt-en | en-de |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bergamot | 0.82 | 0.85 | N/A | 0.87 | N/A | 0.86 | N/A | N/A | 0.84 | 0.89 | N/A | 0.84 | N/A | N/A | N/A | N/A | 0.82 | 0.76 | N/A | N/A | 0.80 | N/A | 0.85 | 0.86 | N/A | 0.86 | N/A | N/A | N/A | N/A | 0.84 | 0.79 | N/A | 0.85 | 0.85 | 0.86 | 0.86 | N/A | N/A | 0.87 | N/A | 0.83 | 0.83 | 0.83 | 0.85 | 0.83 | N/A | 0.73 | 0.89 | N/A | N/A | N/A | N/A | N/A | 0.83 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.83 | N/A | 0.85 | 0.83 | 0.84 | 0.82 | 0.86 | N/A | 0.87 | N/A | 0.83 | N/A | N/A | 0.83 | 0.84 |
| google | 0.85 (+0.03, +3.75%) | 0.89 (+0.04, +4.71%) | 0.89 | 0.91 (+0.04, +4.95%) | 0.91 | 0.88 (+0.02, +2.33%) | 0.87 | 0.90 | 0.86 (+0.02, +1.87%) | 0.90 (+0.02, +1.73%) | 0.89 | 0.86 (+0.03, +3.01%) | 0.88 | 0.88 | 0.88 | 0.91 | 0.85 (+0.02, +2.89%) | 0.84 (+0.08, +10.71%) | 0.90 | 0.90 | 0.88 (+0.08, +9.85%) | 0.90 | 0.87 (+0.02, +2.73%) | 0.89 (+0.03, +3.42%) | 0.91 | 0.88 (+0.02, +2.56%) | 0.73 | 0.91 | 0.87 | 0.89 | 0.90 (+0.06, +6.72%) | 0.88 (+0.09, +11.26%) | 0.89 | 0.89 (+0.04, +5.14%) | 0.89 (+0.04, +5.23%) | 0.88 (+0.01, +1.71%) | 0.88 (+0.02, +2.11%) | 0.89 | 0.90 | 0.89 (+0.02, +2.29%) | 0.87 | 0.87 (+0.04, +4.48%) | 0.85 (+0.02, +2.40%) | 0.89 (+0.06, +7.07%) | 0.89 (+0.05, +5.32%) | 0.90 (+0.07, +8.26%) | 0.90 | 0.87 (+0.14, +18.68%) | 0.92 (+0.02, +2.53%) | 0.90 | 0.89 | 0.84 | 0.83 | 0.92 | 0.89 (+0.06, +7.36%) | 0.88 | 0.90 | 0.88 | 0.90 | 0.89 | 0.90 | 0.86 | 0.90 | 0.86 (+0.03, +3.37%) | 0.91 | 0.86 (+0.02, +1.85%) | 0.86 (+0.02, +2.90%) | 0.88 (+0.05, +5.48%) | 0.85 (+0.03, +3.51%) | 0.90 (+0.04, +4.88%) | 0.92 | 0.89 (+0.01, +1.53%) | 0.92 | 0.84 (+0.01, +1.75%) | 0.91 | 0.85 | 0.87 (+0.03, +3.71%) | 0.87 (+0.03, +3.42%) |
| microsoft | 0.85 (+0.03, +4.13%) | 0.89 (+0.04, +4.50%) | 0.89 | 0.91 (+0.04, +4.67%) | 0.92 | 0.88 (+0.02, +2.42%) | 0.86 | 0.89 | 0.86 (+0.02, +2.43%) | 0.90 (+0.01, +1.28%) | 0.89 | 0.87 (+0.03, +3.29%) | 0.88 | 0.86 | 0.88 | 0.90 | 0.85 (+0.03, +3.29%) | 0.83 (+0.07, +9.25%) | 0.75 | 0.90 | 0.89 (+0.08, +10.31%) | 0.90 | 0.87 (+0.02, +2.62%) | 0.89 (+0.03, +3.83%) | 0.91 | 0.88 (+0.02, +2.12%) | 0.74 | 0.92 | 0.86 | 0.88 | 0.89 (+0.05, +5.59%) | 0.82 (+0.02, +3.03%) | 0.89 | 0.88 (+0.04, +4.25%) | 0.89 (+0.04, +5.13%) | 0.88 (+0.02, +2.31%) | 0.87 (+0.02, +1.81%) | 0.89 | 0.90 | 0.88 (+0.01, +1.63%) | 0.87 | 0.86 (+0.03, +3.78%) | 0.86 (+0.02, +2.65%) | 0.87 (+0.04, +4.79%) | 0.89 (+0.04, +4.92%) | 0.89 (+0.06, +7.15%) | 0.89 | 0.86 (+0.13, +17.94%) | 0.91 (+0.02, +1.79%) | 0.89 | 0.87 | 0.84 | 0.81 | 0.91 | 0.90 (+0.07, +7.82%) | 0.88 | 0.90 | 0.68 | 0.91 | 0.89 | 0.89 | 0.85 | 0.90 | 0.86 (+0.03, +4.16%) | 0.91 | 0.86 (+0.02, +1.97%) | 0.86 (+0.03, +3.13%) | 0.87 (+0.03, +4.08%) | 0.86 (+0.03, +4.15%) | 0.89 (+0.04, +4.14%) | 0.92 | 0.89 (+0.01, +1.43%) | 0.91 | 0.85 (+0.02, +2.34%) | 0.89 | 0.84 | 0.86 (+0.02, +2.59%) | 0.87 (+0.03, +3.72%) |
| argos | 0.71 (-0.11, -13.54%) | 0.76 (-0.09, -10.40%) | 0.78 | 0.86 (-0.02, -1.73%) | 0.85 | 0.84 (-0.02, -2.57%) | N/A | N/A | 0.83 (-0.01, -1.00%) | 0.86 (-0.02, -2.62%) | N/A | 0.66 (-0.18, -21.30%) | N/A | 0.80 | N/A | 0.75 | 0.78 (-0.05, -5.92%) | N/A | N/A | N/A | 0.78 (-0.03, -3.35%) | 0.87 | 0.85 (+0.00, +0.25%) | 0.79 (-0.07, -7.61%) | 0.83 | 0.85 (-0.01, -1.73%) | N/A | 0.87 | N/A | N/A | 0.84 (-0.00, -0.21%) | 0.80 (+0.01, +1.43%) | N/A | 0.86 (+0.01, +1.63%) | 0.87 (+0.02, +2.10%) | 0.77 (-0.09, -10.72%) | 0.86 (+0.00, +0.37%) | N/A | 0.78 | 0.85 (-0.01, -1.37%) | 0.72 | 0.76 (-0.07, -8.33%) | 0.82 (-0.01, -1.52%) | 0.82 (-0.01, -0.64%) | 0.87 (+0.02, +2.23%) | 0.70 (-0.14, -16.31%) | N/A | N/A | 0.88 (-0.01, -1.47%) | N/A | N/A | 0.79 | 0.73 | N/A | 0.68 (-0.16, -18.74%) | N/A | N/A | N/A | 0.88 | 0.71 | 0.82 | N/A | 0.81 | 0.82 (-0.01, -1.29%) | 0.84 | 0.83 (-0.01, -1.44%) | 0.84 (+0.00, +0.36%) | N/A | 0.81 (-0.01, -1.47%) | 0.84 (-0.02, -1.84%) | 0.83 | 0.87 (-0.01, -0.79%) | 0.72 | 0.82 (-0.01, -0.93%) | N/A | 0.72 | N/A | 0.79 (-0.05, -5.97%) |
| nllb | 0.73 (-0.09, -11.15%) | 0.77 (-0.08, -9.17%) | 0.76 | 0.83 (-0.04, -4.59%) | 0.85 | 0.86 (+0.00, +0.21%) | 0.86 | 0.84 | 0.83 (-0.01, -0.77%) | 0.88 (-0.01, -0.61%) | 0.80 | 0.70 (-0.14, -16.93%) | 0.85 | 0.80 | 0.84 | 0.85 | 0.83 (+0.01, +0.98%) | 0.62 (-0.14, -17.90%) | 0.83 | 0.87 | 0.83 (+0.02, +2.74%) | 0.87 | 0.85 (+0.00, +0.50%) | 0.65 (-0.20, -23.75%) | 0.88 | 0.85 (-0.01, -0.76%) | 0.69 | 0.88 | 0.67 | 0.80 | 0.86 (+0.02, +1.84%) | 0.84 (+0.05, +5.81%) | 0.75 | 0.59 (-0.25, -30.00%) | 0.76 (-0.09, -10.49%) | 0.85 (-0.01, -1.49%) | 0.81 (-0.05, -5.79%) | 0.81 | 0.86 | 0.86 (-0.01, -1.17%) | 0.84 | 0.82 (-0.02, -1.81%) | 0.84 (+0.01, +0.91%) | 0.83 (+0.00, +0.16%) | 0.83 (-0.02, -2.68%) | 0.83 (-0.00, -0.39%) | 0.86 | 0.63 (-0.10, -13.68%) | 0.88 (-0.01, -1.54%) | 0.73 | 0.83 | 0.79 | 0.79 | 0.87 | 0.84 (+0.01, +1.41%) | 0.73 | 0.87 | 0.81 | 0.77 | 0.86 | 0.88 | 0.79 | 0.86 | 0.75 (-0.08, -9.91%) | 0.78 | 0.82 (-0.03, -3.55%) | 0.77 (-0.06, -7.70%) | 0.68 (-0.15, -18.21%) | 0.84 (+0.01, +1.81%) | 0.84 (-0.01, -1.50%) | 0.90 | 0.84 (-0.04, -4.16%) | 0.86 | 0.83 (-0.00, -0.10%) | 0.84 | 0.77 | 0.67 (-0.17, -19.89%) | 0.83 (-0.01, -1.09%) |
| opusmt | 0.82 (+0.00, +0.21%) | 0.85 (-0.00, -0.05%) | 0.86 | 0.86 (-0.01, -1.30%) | 0.91 | 0.85 (-0.01, -0.77%) | 0.66 | N/A | 0.85 (+0.01, +0.72%) | N/A | N/A | 0.84 (+0.00, +0.58%) | 0.82 | 0.79 | 0.82 | N/A | 0.82 (-0.01, -1.13%) | N/A | N/A | N/A | 0.84 (+0.04, +4.68%) | 0.86 | N/A | 0.86 (+0.01, +0.63%) | 0.88 | 0.85 (-0.01, -1.22%) | N/A | 0.89 | 0.84 | N/A | 0.84 (+0.00, +0.10%) | N/A | N/A | 0.86 (+0.02, +2.05%) | N/A | 0.77 (-0.10, -11.04%) | 0.86 (+0.00, +0.43%) | N/A | N/A | 0.86 (-0.00, -0.52%) | 0.82 | 0.80 (-0.03, -3.99%) | 0.84 (+0.01, +1.12%) | N/A | 0.78 (-0.06, -7.63%) | 0.79 (-0.04, -4.37%) | 0.76 | 0.78 (+0.05, +7.09%) | 0.82 (-0.07, -7.86%) | N/A | 0.72 | 0.79 | 0.60 | N/A | 0.84 (+0.01, +1.36%) | N/A | 0.86 | N/A | 0.89 | N/A | 0.75 | 0.74 | 0.86 | 0.84 (+0.02, +1.83%) | 0.89 | 0.85 (+0.00, +0.27%) | 0.83 (-0.00, -0.23%) | N/A | 0.83 (+0.01, +1.43%) | N/A | 0.88 | N/A | 0.86 | 0.84 (+0.01, +0.97%) | N/A | 0.72 | N/A | 0.83 (-0.01, -1.31%) |
| bergamot | 0.82 | 0.85 | N/A | 0.87 | N/A | 0.86 | N/A | N/A | 0.84 | 0.89 | N/A | 0.84 | N/A | N/A | N/A | N/A | 0.82 | 0.76 | N/A | N/A | 0.80 | N/A | 0.85 | 0.86 | N/A | 0.86 | N/A | N/A | N/A | N/A | 0.84 | 0.79 | N/A | 0.85 | 0.85 | 0.86 | 0.86 | N/A | N/A | 0.87 | N/A | 0.83 | 0.83 | 0.83 | 0.85 | 0.83 | N/A | 0.73 | 0.89 | N/A | N/A | N/A | N/A | N/A | 0.84 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 0.83 | N/A | 0.85 | 0.83 | 0.84 | 0.82 | 0.86 | N/A | 0.87 | N/A | 0.83 | N/A | N/A | 0.83 | 0.84 |
| google | 0.85 (+0.03, +3.75%) | 0.89 (+0.04, +4.71%) | 0.89 | 0.91 (+0.04, +4.95%) | 0.91 | 0.88 (+0.02, +2.33%) | 0.87 | 0.90 | 0.86 (+0.02, +1.87%) | 0.90 (+0.02, +1.73%) | 0.89 | 0.86 (+0.03, +3.01%) | 0.88 | 0.88 | 0.88 | 0.91 | 0.85 (+0.02, +2.89%) | 0.84 (+0.08, +10.71%) | 0.90 | 0.90 | 0.88 (+0.08, +9.85%) | 0.90 | 0.87 (+0.02, +2.73%) | 0.89 (+0.03, +3.42%) | 0.91 | 0.88 (+0.02, +2.56%) | 0.73 | 0.91 | 0.87 | 0.89 | 0.90 (+0.06, +6.72%) | 0.88 (+0.09, +11.26%) | 0.89 | 0.89 (+0.04, +5.14%) | 0.89 (+0.04, +5.23%) | 0.88 (+0.01, +1.71%) | 0.88 (+0.02, +2.11%) | 0.89 | 0.90 | 0.89 (+0.02, +2.29%) | 0.87 | 0.87 (+0.04, +4.48%) | 0.85 (+0.02, +2.40%) | 0.89 (+0.06, +7.07%) | 0.89 (+0.05, +5.32%) | 0.90 (+0.07, +8.26%) | 0.90 | 0.87 (+0.14, +18.68%) | 0.92 (+0.02, +2.53%) | 0.90 | 0.89 | 0.84 | 0.83 | 0.92 | 0.89 (+0.05, +5.68%) | 0.88 | 0.90 | 0.88 | 0.90 | 0.89 | 0.90 | 0.86 | 0.90 | 0.86 (+0.03, +3.37%) | 0.91 | 0.86 (+0.02, +1.85%) | 0.86 (+0.02, +2.90%) | 0.88 (+0.05, +5.48%) | 0.85 (+0.03, +3.51%) | 0.90 (+0.04, +4.88%) | 0.92 | 0.89 (+0.01, +1.53%) | 0.92 | 0.84 (+0.01, +1.75%) | 0.91 | 0.85 | 0.87 (+0.03, +3.71%) | 0.87 (+0.03, +3.42%) |
| microsoft | 0.85 (+0.03, +4.13%) | 0.89 (+0.04, +4.50%) | 0.89 | 0.91 (+0.04, +4.67%) | 0.92 | 0.88 (+0.02, +2.42%) | 0.86 | 0.89 | 0.86 (+0.02, +2.43%) | 0.90 (+0.01, +1.28%) | 0.89 | 0.87 (+0.03, +3.29%) | 0.88 | 0.86 | 0.88 | 0.90 | 0.85 (+0.03, +3.29%) | 0.83 (+0.07, +9.25%) | 0.75 | 0.90 | 0.89 (+0.08, +10.31%) | 0.90 | 0.87 (+0.02, +2.62%) | 0.89 (+0.03, +3.83%) | 0.91 | 0.88 (+0.02, +2.12%) | 0.74 | 0.92 | 0.86 | 0.88 | 0.89 (+0.05, +5.59%) | 0.82 (+0.02, +3.03%) | 0.89 | 0.88 (+0.04, +4.25%) | 0.89 (+0.04, +5.13%) | 0.88 (+0.02, +2.31%) | 0.87 (+0.02, +1.81%) | 0.89 | 0.90 | 0.88 (+0.01, +1.63%) | 0.87 | 0.86 (+0.03, +3.78%) | 0.86 (+0.02, +2.65%) | 0.87 (+0.04, +4.79%) | 0.89 (+0.04, +4.92%) | 0.89 (+0.06, +7.15%) | 0.89 | 0.86 (+0.13, +17.94%) | 0.91 (+0.02, +1.79%) | 0.89 | 0.87 | 0.84 | 0.81 | 0.91 | 0.90 (+0.05, +6.13%) | 0.88 | 0.90 | 0.68 | 0.91 | 0.89 | 0.89 | 0.85 | 0.90 | 0.86 (+0.03, +4.16%) | 0.91 | 0.86 (+0.02, +1.97%) | 0.86 (+0.03, +3.13%) | 0.87 (+0.03, +4.08%) | 0.86 (+0.03, +4.15%) | 0.89 (+0.04, +4.14%) | 0.92 | 0.89 (+0.01, +1.43%) | 0.91 | 0.85 (+0.02, +2.34%) | 0.89 | 0.84 | 0.86 (+0.02, +2.59%) | 0.87 (+0.03, +3.72%) |
| argos | 0.71 (-0.11, -13.54%) | 0.76 (-0.09, -10.40%) | 0.78 | 0.86 (-0.02, -1.73%) | 0.85 | 0.84 (-0.02, -2.57%) | N/A | N/A | 0.83 (-0.01, -1.00%) | 0.86 (-0.02, -2.62%) | N/A | 0.66 (-0.18, -21.30%) | N/A | 0.80 | N/A | 0.75 | 0.78 (-0.05, -5.92%) | N/A | N/A | N/A | 0.78 (-0.03, -3.35%) | 0.87 | 0.85 (+0.00, +0.25%) | 0.79 (-0.07, -7.61%) | 0.83 | 0.85 (-0.01, -1.73%) | N/A | 0.87 | N/A | N/A | 0.84 (-0.00, -0.21%) | 0.80 (+0.01, +1.43%) | N/A | 0.86 (+0.01, +1.63%) | 0.87 (+0.02, +2.10%) | 0.77 (-0.09, -10.72%) | 0.86 (+0.00, +0.37%) | N/A | 0.78 | 0.85 (-0.01, -1.37%) | 0.72 | 0.76 (-0.07, -8.33%) | 0.82 (-0.01, -1.52%) | 0.82 (-0.01, -0.64%) | 0.87 (+0.02, +2.23%) | 0.70 (-0.14, -16.31%) | N/A | N/A | 0.88 (-0.01, -1.47%) | N/A | N/A | 0.79 | 0.73 | N/A | 0.68 (-0.17, -20.02%) | N/A | N/A | N/A | 0.88 | 0.71 | 0.82 | N/A | 0.81 | 0.82 (-0.01, -1.29%) | 0.84 | 0.83 (-0.01, -1.44%) | 0.84 (+0.00, +0.36%) | N/A | 0.81 (-0.01, -1.47%) | 0.84 (-0.02, -1.84%) | 0.83 | 0.87 (-0.01, -0.79%) | 0.72 | 0.82 (-0.01, -0.93%) | N/A | 0.72 | N/A | 0.79 (-0.05, -5.97%) |
| nllb | 0.73 (-0.09, -11.15%) | 0.77 (-0.08, -9.17%) | 0.76 | 0.83 (-0.04, -4.59%) | 0.85 | 0.86 (+0.00, +0.21%) | 0.86 | 0.84 | 0.83 (-0.01, -0.77%) | 0.88 (-0.01, -0.61%) | 0.80 | 0.70 (-0.14, -16.93%) | 0.85 | 0.80 | 0.84 | 0.85 | 0.83 (+0.01, +0.98%) | 0.62 (-0.14, -17.90%) | 0.83 | 0.87 | 0.83 (+0.02, +2.74%) | 0.87 | 0.85 (+0.00, +0.50%) | 0.65 (-0.20, -23.75%) | 0.88 | 0.85 (-0.01, -0.76%) | 0.69 | 0.88 | 0.67 | 0.80 | 0.86 (+0.02, +1.84%) | 0.84 (+0.05, +5.81%) | 0.75 | 0.59 (-0.25, -30.00%) | 0.76 (-0.09, -10.49%) | 0.85 (-0.01, -1.49%) | 0.81 (-0.05, -5.79%) | 0.81 | 0.86 | 0.86 (-0.01, -1.17%) | 0.84 | 0.82 (-0.02, -1.81%) | 0.84 (+0.01, +0.91%) | 0.83 (+0.00, +0.16%) | 0.83 (-0.02, -2.68%) | 0.83 (-0.00, -0.39%) | 0.86 | 0.63 (-0.10, -13.68%) | 0.88 (-0.01, -1.54%) | 0.73 | 0.83 | 0.79 | 0.79 | 0.87 | 0.84 (-0.00, -0.18%) | 0.73 | 0.87 | 0.81 | 0.77 | 0.86 | 0.88 | 0.79 | 0.86 | 0.75 (-0.08, -9.91%) | 0.78 | 0.82 (-0.03, -3.55%) | 0.77 (-0.06, -7.70%) | 0.68 (-0.15, -18.21%) | 0.84 (+0.01, +1.81%) | 0.84 (-0.01, -1.50%) | 0.90 | 0.84 (-0.04, -4.16%) | 0.86 | 0.83 (-0.00, -0.10%) | 0.84 | 0.77 | 0.67 (-0.17, -19.89%) | 0.83 (-0.01, -1.09%) |
| opusmt | 0.82 (+0.00, +0.21%) | 0.85 (-0.00, -0.05%) | 0.86 | 0.86 (-0.01, -1.30%) | 0.91 | 0.85 (-0.01, -0.77%) | 0.66 | N/A | 0.85 (+0.01, +0.72%) | N/A | N/A | 0.84 (+0.00, +0.58%) | 0.82 | 0.79 | 0.82 | N/A | 0.82 (-0.01, -1.13%) | N/A | N/A | N/A | 0.84 (+0.04, +4.68%) | 0.86 | N/A | 0.86 (+0.01, +0.63%) | 0.88 | 0.85 (-0.01, -1.22%) | N/A | 0.89 | 0.84 | N/A | 0.84 (+0.00, +0.10%) | N/A | N/A | 0.86 (+0.02, +2.05%) | N/A | 0.77 (-0.10, -11.04%) | 0.86 (+0.00, +0.43%) | N/A | N/A | 0.86 (-0.00, -0.52%) | 0.82 | 0.80 (-0.03, -3.99%) | 0.84 (+0.01, +1.12%) | N/A | 0.78 (-0.06, -7.63%) | 0.79 (-0.04, -4.37%) | 0.76 | 0.78 (+0.05, +7.09%) | 0.82 (-0.07, -7.86%) | N/A | 0.72 | 0.79 | 0.60 | N/A | 0.84 (-0.00, -0.23%) | N/A | 0.86 | N/A | 0.89 | N/A | 0.75 | 0.74 | 0.86 | 0.84 (+0.02, +1.83%) | 0.89 | 0.85 (+0.00, +0.27%) | 0.83 (-0.00, -0.23%) | N/A | 0.83 (+0.01, +1.43%) | N/A | 0.88 | N/A | 0.86 | 0.84 (+0.01, +0.97%) | N/A | 0.72 | N/A | 0.83 (-0.01, -1.31%) |
![Results](img/avg-comet.png)
---
@ -1686,12 +1686,12 @@ We also compare the systems using the `comet-compare` tool that calculates the s
| Translator/Dataset | wmt11 | wmt20 | wmt13 | flores-test | wmt08 | wmt21 | wmt14 | wmt17 | wmt22 | wmt12 | wmt18 | wmt15 | wmt10 | flores-dev | wmt16 | wmt19 | wmt09 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bergamot | 0.82 | 0.82 | 0.85 | 0.85 | 0.82 | 0.80 | 0.86 | 0.84 | 0.83 | 0.82 | 0.84 | 0.84 | 0.83 | 0.85 | 0.84 | 0.81 | 0.83 |
| google | 0.88 (+0.06, +7.56%) | 0.89 (+0.07, +8.36%) | 0.90 (+0.05, +5.65%) | 0.91 (+0.06, +7.37%) | 0.88 (+0.07, +8.33%) | 0.86 (+0.06, +7.72%) | 0.92 (+0.05, +6.37%) | 0.89 (+0.05, +6.41%) | 0.91 (+0.09, +10.46%) | 0.88 (+0.06, +7.43%) | 0.89 (+0.05, +6.25%) | 0.90 (+0.06, +6.65%) | 0.89 (+0.06, +7.25%) | 0.92 (+0.07, +7.80%) | 0.89 (+0.06, +6.76%) | 0.87 (+0.06, +7.57%) | 0.89 (+0.06, +7.40%) |
| microsoft | 0.89 (+0.07, +8.48%) | 0.90 (+0.08, +9.47%) | 0.90 (+0.05, +6.23%) | 0.91 (+0.06, +7.21%) | 0.89 (+0.07, +8.63%) | 0.87 (+0.07, +8.81%) | 0.92 (+0.06, +6.99%) | 0.89 (+0.06, +6.82%) | 0.91 (+0.08, +9.58%) | 0.89 (+0.07, +8.12%) | 0.90 (+0.06, +6.65%) | 0.90 (+0.06, +6.84%) | 0.89 (+0.06, +7.65%) | 0.92 (+0.07, +7.76%) | 0.90 (+0.06, +7.13%) | 0.89 (+0.08, +9.25%) | 0.89 (+0.06, +7.64%) |
| argos | 0.69 (-0.13, -15.65%) | 0.59 (-0.23, -28.19%) | 0.72 (-0.13, -15.75%) | 0.71 (-0.14, -16.52%) | 0.66 (-0.15, -18.62%) | 0.59 (-0.21, -26.15%) | 0.69 (-0.17, -19.58%) | 0.68 (-0.16, -18.56%) | 0.74 (-0.09, -10.71%) | 0.67 (-0.15, -18.11%) | 0.68 (-0.16, -19.19%) | 0.68 (-0.16, -18.90%) | 0.68 (-0.15, -17.82%) | 0.72 (-0.13, -15.26%) | 0.68 (-0.15, -18.12%) | 0.60 (-0.21, -25.60%) | 0.69 (-0.14, -16.45%) |
| nllb | 0.84 (+0.02, +2.84%) | 0.83 (+0.01, +0.92%) | 0.86 (+0.01, +1.11%) | 0.87 (+0.01, +1.74%) | 0.84 (+0.02, +2.64%) | 0.81 (+0.01, +1.10%) | 0.87 (+0.01, +0.66%) | 0.84 (+0.01, +0.69%) | 0.85 (+0.02, +2.21%) | 0.84 (+0.02, +1.96%) | 0.85 (+0.01, +0.79%) | 0.85 (+0.00, +0.56%) | 0.85 (+0.02, +2.07%) | 0.86 (+0.01, +1.75%) | 0.84 (+0.01, +0.80%) | 0.82 (+0.01, +0.70%) | 0.84 (+0.01, +1.50%) |
| opusmt | 0.84 (+0.03, +3.41%) | 0.81 (-0.01, -0.89%) | 0.87 (+0.02, +1.99%) | 0.86 (+0.01, +0.89%) | 0.83 (+0.02, +2.11%) | 0.80 (-0.01, -0.69%) | 0.87 (+0.01, +1.26%) | 0.85 (+0.01, +1.25%) | 0.85 (+0.02, +2.95%) | 0.84 (+0.02, +2.44%) | 0.85 (+0.01, +1.29%) | 0.86 (+0.01, +1.57%) | 0.85 (+0.02, +2.20%) | 0.86 (+0.01, +1.26%) | 0.85 (+0.01, +1.21%) | 0.80 (-0.01, -1.33%) | 0.85 (+0.02, +2.19%) |
| bergamot | 0.84 | 0.84 | 0.86 | 0.87 | 0.82 | 0.81 | 0.87 | 0.85 | 0.85 | 0.84 | 0.85 | 0.86 | 0.85 | 0.87 | 0.85 | 0.82 | 0.84 |
| google | 0.88 (+0.04, +4.99%) | 0.89 (+0.05, +5.86%) | 0.90 (+0.03, +4.05%) | 0.91 (+0.05, +5.56%) | 0.88 (+0.07, +8.15%) | 0.86 (+0.05, +6.62%) | 0.92 (+0.04, +4.68%) | 0.89 (+0.05, +5.43%) | 0.91 (+0.07, +7.84%) | 0.88 (+0.04, +5.20%) | 0.89 (+0.04, +4.78%) | 0.90 (+0.04, +5.00%) | 0.89 (+0.04, +4.93%) | 0.92 (+0.05, +5.59%) | 0.89 (+0.04, +5.25%) | 0.87 (+0.06, +7.17%) | 0.89 (+0.05, +5.67%) |
| microsoft | 0.89 (+0.05, +5.89%) | 0.90 (+0.06, +6.95%) | 0.90 (+0.04, +4.62%) | 0.91 (+0.05, +5.40%) | 0.89 (+0.07, +8.46%) | 0.87 (+0.06, +7.69%) | 0.92 (+0.05, +5.28%) | 0.89 (+0.05, +5.83%) | 0.91 (+0.06, +6.98%) | 0.89 (+0.05, +5.88%) | 0.90 (+0.04, +5.17%) | 0.90 (+0.04, +5.18%) | 0.89 (+0.05, +5.32%) | 0.92 (+0.05, +5.55%) | 0.90 (+0.05, +5.62%) | 0.89 (+0.07, +8.85%) | 0.89 (+0.05, +5.91%) |
| argos | 0.69 (-0.15, -17.67%) | 0.59 (-0.25, -29.84%) | 0.72 (-0.15, -17.03%) | 0.71 (-0.16, -17.92%) | 0.66 (-0.15, -18.75%) | 0.59 (-0.22, -26.91%) | 0.69 (-0.18, -20.86%) | 0.68 (-0.16, -19.31%) | 0.74 (-0.11, -12.83%) | 0.67 (-0.17, -19.82%) | 0.68 (-0.17, -20.31%) | 0.68 (-0.17, -20.16%) | 0.68 (-0.17, -19.60%) | 0.72 (-0.15, -17.00%) | 0.68 (-0.16, -19.28%) | 0.60 (-0.21, -25.88%) | 0.69 (-0.15, -17.79%) |
| nllb | 0.84 (+0.00, +0.38%) | 0.83 (-0.01, -1.41%) | 0.86 (-0.00, -0.43%) | 0.87 (+0.00, +0.02%) | 0.84 (+0.02, +2.47%) | 0.81 (+0.00, +0.06%) | 0.87 (-0.01, -0.94%) | 0.84 (-0.00, -0.24%) | 0.85 (-0.00, -0.21%) | 0.84 (-0.00, -0.16%) | 0.85 (-0.01, -0.61%) | 0.85 (-0.01, -1.00%) | 0.85 (-0.00, -0.14%) | 0.86 (-0.00, -0.33%) | 0.84 (-0.01, -0.63%) | 0.82 (+0.00, +0.33%) | 0.84 (-0.00, -0.13%) |
| opusmt | 0.84 (+0.01, +0.93%) | 0.81 (-0.03, -3.18%) | 0.87 (+0.00, +0.44%) | 0.86 (-0.01, -0.81%) | 0.83 (+0.02, +1.95%) | 0.80 (-0.01, -1.70%) | 0.87 (-0.00, -0.35%) | 0.85 (+0.00, +0.32%) | 0.85 (+0.00, +0.51%) | 0.84 (+0.00, +0.31%) | 0.85 (-0.00, -0.12%) | 0.86 (-0.00, -0.01%) | 0.85 (-0.00, -0.01%) | 0.86 (-0.01, -0.82%) | 0.85 (-0.00, -0.22%) | 0.80 (-0.01, -1.69%) | 0.85 (+0.00, +0.55%) |
![Results](img/en-cs-comet.png)
### Comparisons between systems

1934
evaluation/en-cs/flores-dev.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/flores-dev.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
30.1
30.7

2
evaluation/en-cs/flores-dev.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8495
0.8673

1978
evaluation/en-cs/flores-test.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/flores-test.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
30.2
31.8

2
evaluation/en-cs/flores-test.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8519
0.8665

3876
evaluation/en-cs/wmt08.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt08.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
19.0
19.7

2
evaluation/en-cs/wmt08.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8154
0.8167

5666
evaluation/en-cs/wmt09.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt09.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
20.8
21.9

2
evaluation/en-cs/wmt09.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8273
0.8408

4668
evaluation/en-cs/wmt10.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt10.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
20.9
22.1

2
evaluation/en-cs/wmt10.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8301
0.8485

5698
evaluation/en-cs/wmt11.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt11.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
20.7
23.3

2
evaluation/en-cs/wmt11.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8159
0.8359

5640
evaluation/en-cs/wmt12.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt12.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
18.8
20.4

2
evaluation/en-cs/wmt12.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8198
0.8372

5500
evaluation/en-cs/wmt13.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt13.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
23.4
25.4

2
evaluation/en-cs/wmt13.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8493
0.8624

5604
evaluation/en-cs/wmt14.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt14.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
28.7
28.4

2
evaluation/en-cs/wmt14.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8604
0.8743

4846
evaluation/en-cs/wmt15.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt15.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
25.4
24.3

2
evaluation/en-cs/wmt15.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8433
0.8566

5538
evaluation/en-cs/wmt16.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt16.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
25.8
27.8

2
evaluation/en-cs/wmt16.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8355
0.8475

5538
evaluation/en-cs/wmt17.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt17.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
23.4
23.1

2
evaluation/en-cs/wmt17.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8373
0.8451

5482
evaluation/en-cs/wmt18.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt18.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
22.7
22.3

2
evaluation/en-cs/wmt18.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8404
0.8522

3830
evaluation/en-cs/wmt19.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt19.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
27.1
24.8

2
evaluation/en-cs/wmt19.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8128
0.8158

2790
evaluation/en-cs/wmt20.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt20.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
32.7
32.8

2
evaluation/en-cs/wmt20.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8184
0.8377

1962
evaluation/en-cs/wmt21.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt21.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
19.5
18.6

2
evaluation/en-cs/wmt21.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8015
0.8098

3830
evaluation/en-cs/wmt22.bergamot.cs
Просмотреть файл

Разница между файлами не показана из-за своего большого размера Загрузить разницу

2
evaluation/en-cs/wmt22.bergamot.cs.bleu
Просмотреть файл

@ -1 +1 @@
31.7
33.6

2
evaluation/en-cs/wmt22.bergamot.cs.comet
Просмотреть файл

@ -1 +1 @@
0.8271
0.8472

Двоичные данные
evaluation/img/avg-bleu.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 44 KiB

После

Ширина:  |  Высота:  |  Размер: 44 KiB

Двоичные данные
evaluation/img/avg-comet.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 44 KiB

После

Ширина:  |  Высота:  |  Размер: 44 KiB

Двоичные данные
evaluation/img/en-cs-bleu.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 28 KiB

После

Ширина:  |  Высота:  |  Размер: 28 KiB

Двоичные данные
evaluation/img/en-cs-comet.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 29 KiB

После

Ширина:  |  Высота:  |  Размер: 29 KiB

4
models/dev/encs/lex.50.50.encs.s2t.bin.gz
Просмотреть файл

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39cb03d8d96b2b912abfc9582ff8b58dd9c59047a48e131e724a1b5a9b7fdacd
size 1913246
oid sha256:d383618429e64c704fcb7eef7e8a33debfd97a580b896ed967f4611ee9bfb192
size 1732597

4
models/dev/encs/model.encs.intgemm.alphas.bin.gz
Просмотреть файл

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:72385bed7b1de4906d4d9cbe71883267ada0edd832cc41217cf0374c058ef6be
size 12630325
oid sha256:6236ae9ba3d8f7e54c8827e992d4ada2ba0bcc98e9c2291829cab2b1d0689f7f
size 13054998

3
models/dev/encs/qualityModel.encs.bin.gz
Просмотреть файл

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d7f8da423effcf293c0acd49ce979ec61ceefc34fa6c462943c33e04da63f754
size 108

3
models/dev/encs/vocab.csen.spm.gz
Просмотреть файл

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:88fed2abdd3fc91b8715cf0bd5f661747234f4eb9ea09bf731a6c6c08e89b289
size 366392

3
models/dev/encs/vocab.encs.spm.gz Normal file
Просмотреть файл

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8d018568b00653c23bbd0ac1df86de03144ee7b0cbfbb4331537a90253f33e61
size 403234

69
registry.json
Просмотреть файл

@ -182,19 +182,19 @@
"expectedSha256Hash": "38fb44bad1fd5f1e6bfdcf15cc8baa09d61aad2a4f9c587914e24e7b5c25c32c",
"modelType": "prod"
},
"model": {
"name": "model.enfr.intgemm.alphas.bin",
"size": 17140961,
"estimatedCompressedSize": 12293754,
"expectedSha256Hash": "0678019c4d74c8c81d2de17e3e58d3aba5f5eb48f5595d9240c17f69d30461de",
"modelType": "prod"
},
"vocab": {
"name": "vocab.fren.spm",
"size": 831382,
"estimatedCompressedSize": 419721,
"expectedSha256Hash": "4c84b95b62c930f0791466d73eb996841eef474c96d0c2f0e6c6d80640f2005a",
"modelType": "prod"
},
"model": {
"name": "model.enfr.intgemm.alphas.bin",
"size": 17140961,
"estimatedCompressedSize": 12293754,
"expectedSha256Hash": "0678019c4d74c8c81d2de17e3e58d3aba5f5eb48f5595d9240c17f69d30461de",
"modelType": "prod"
}
},
"enit": {
@ -520,13 +520,6 @@
}
},
"slen": {
"lex": {
"name": "lex.50.50.slen.s2t.bin",
"size": 4052216,
"estimatedCompressedSize": 2074071,
"expectedSha256Hash": "1b7f14de28e3e840fe1e47092f688fb8d4396ea431f49492d78a25102cf0773a",
"modelType": "prod"
},
"model": {
"name": "model.slen.intgemm.alphas.bin",
"size": 17140836,
@ -534,6 +527,13 @@
"expectedSha256Hash": "48846d4029e51b449c642e1131f9224ede5ea7ffb08f415c848bbb58ddce7cd9",
"modelType": "prod"
},
"lex": {
"name": "lex.50.50.slen.s2t.bin",
"size": 4052216,
"estimatedCompressedSize": 2074071,
"expectedSha256Hash": "1b7f14de28e3e840fe1e47092f688fb8d4396ea431f49492d78a25102cf0773a",
"modelType": "prod"
},
"vocab": {
"name": "vocab.slen.spm",
"size": 817982,
@ -566,13 +566,6 @@
}
},
"uken": {
"model": {
"name": "model.uken.intgemm8.bin",
"size": 25315747,
"estimatedCompressedSize": 18520747,
"expectedSha256Hash": "90b6e21644af5bf5ce26442c724f55848a005d75e8bf688a51d2e64d6bc6b249",
"modelType": "prod"
},
"lex": {
"name": "lex.uken.s2t.bin",
"size": 9761460,
@ -580,6 +573,13 @@
"expectedSha256Hash": "763b9e0add9fd712305bc031ab86a58fb15f719dcad296046742176937b86841",
"modelType": "prod"
},
"model": {
"name": "model.uken.intgemm8.bin",
"size": 25315747,
"estimatedCompressedSize": 18520747,
"expectedSha256Hash": "90b6e21644af5bf5ce26442c724f55848a005d75e8bf688a51d2e64d6bc6b249",
"modelType": "prod"
},
"srcvocab": {
"name": "srcvocab.uken.spm",
"size": 984214,
@ -667,30 +667,23 @@
"encs": {
"model": {
"name": "model.encs.intgemm.alphas.bin",
"size": 17140756,
"estimatedCompressedSize": 12630325,
"expectedSha256Hash": "9a2fe0588bd972accfc801e2f31c945de0557804a91666ae5ab43b94fb74ac4b",
"size": 17140898,
"estimatedCompressedSize": 13054998,
"expectedSha256Hash": "7815671064234ff87bc20e9f30e736a23cdcb2286fd1aea44db16116a8ab8680",
"modelType": "dev"
},
"lex": {
"name": "lex.50.50.encs.s2t.bin",
"size": 3556124,
"estimatedCompressedSize": 1913246,
"expectedSha256Hash": "e19c77231bf977988e31ff8db15fe79966b5170564bd3e10613f239e7f461d97",
"size": 3365784,
"estimatedCompressedSize": 1732597,
"expectedSha256Hash": "ba42a89db077e6444e7409363a1c71165c6210c21579ad8a310d0d98b0ea9485",
"modelType": "dev"
},
"vocab": {
"name": "vocab.csen.spm",
"size": 769763,
"estimatedCompressedSize": 366392,
"expectedSha256Hash": "f71cc5d045e479607078e079884f44032f5a0b82547fb96eefa29cd1eb47c6f3",
"modelType": "dev"
},
"qualityModel": {
"name": "qualityModel.encs.bin",
"size": 68,
"estimatedCompressedSize": 108,
"expectedSha256Hash": "d7eba90036a065e4a1e93e889befe09f93a7d9a3417f3edffdb09a0db88fe83a",
"name": "vocab.encs.spm",
"size": 805211,
"estimatedCompressedSize": 403234,
"expectedSha256Hash": "8af48f972eeba9a9011c0525adf83e0e7abc54c328b2217351a3dad5a852040b",
"modelType": "dev"
}
},

5
scripts/pull_models.sh
Просмотреть файл

@ -13,7 +13,9 @@
LOCAL_DIR="../models/dev/${SRC}${TRG}"
TC_PREFIX="https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task"
rm -rf "${LOCAL_DIR}"
git rm -r "${LOCAL_DIR}"
git rm ../evaluation/${SRC}-${TRG}/*.bergamot*
mkdir "${LOCAL_DIR}"
wget -O "${LOCAL_DIR}"/lex.50.50.${SRC}${TRG}.s2t.bin.gz \
@ -24,4 +26,3 @@ wget -O "${LOCAL_DIR}"/vocab.${SRC}${TRG}.spm.gz \
${TC_PREFIX}/${TASK_ID}/runs/0/artifacts/public%2Fbuild%2Fvocab.${SRC}${TRG}.spm.gz
git add "${LOCAL_DIR}"/*