diff --git a/README.md b/README.md index 667d7d9..3958e57 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,7 @@ You can find that the original input sentence is restored from the vocabulary id * **MeCab**: [MeCab](http://taku910.github.io/mecab/) for Japanese. * **neologd**: [MeCab with neologd](https://github.com/neologd/mecab-ipadic-neologd) for Japanese. * **(Moses/KyTea)+SentencePiece**: Apply SentencePiece (Unigram) to pre-tokenized sentences. We have several variants with different tokenizers., e.g., **(Moses/MeCab)+SentencePiece**, **(MeCab/Moses)+SentencePiece**. + * *char**: Segments sentence by characters. * Data sets: * [KFTT](http://www.phontron.com/kftt/index.html) @@ -184,6 +185,7 @@ You can find that the original input sentence is restored from the vocabulary id #### English to Japanese |Setting|vocab size|BLEU(dev)|BLEU(test)|src #tokens/sent.|trg #tokens/sent.| |:---|---:|---:|---:|---:|---:| +|SentencePiece|4k (shared)|0.2857|0.2940|43.7478|29.6998| |SentencePiece|8k (shared)|0.2785|0.2955|30.9734|25.0540| |SentencePiece|16k (shared)|0.2664|0.2862|27.1827|21.5326| |SentencePiece|32k (shared)|0.2641|0.2849|25.0592|19.0840| @@ -198,14 +200,16 @@ You can find that the original input sentence is restored from the vocabulary id |SentencePiece/KyTea|8k/80k|0.2778|0.2918|27.0429|23.2161| |SentencePiece/MeCab|8k/80k|0.2673|0.2919|27.0429|21.2033| |SentencePiece/neolgod|8k80k|0.2280|0.2494|27.0429|18.4768| +|Char|3k (shared)|0.2509|0.2679|109.8662|33.6963| #### Japanese to English |Setting|vocab size|BLEU(dev)|BLEU(test)|src #tokens/sent.|trg #tokens/sent.| |:---|---:|---:|---:|---:|---:| +|SentencePiece|4k (shared)|0.1970|**0.2179**|29.6998|43.7478| |SentencePiece|8k (shared)|0.1966|0.2162|25.0540|30.9734| |SentencePiece|16k (shared)|0.1996|0.2160|21.5326|27.1827| |SentencePiece|32k (shared)|0.1949|0.2159|19.0840|25.0592| -|SentencePiece(BPE)|8k (shaerd)|0.1977|**0.2173**|25.4331|31.7693| +|SentencePiece(BPE)|8k (shaerd)|0.1977|0.2173|25.4331|31.7693| |(KyTea/Moses)+SentencePiece|8k (shared)|0.1921|0.2086|29.9854|31.2719| |(MeCab/Moses)+SentencePiece|8k (shared)|0.1909|0.2049|28.9537|31.4743| |(neologd/Moses)+SentencePiece|8k (shared)|0.1938|0.2137|28.8645|31.2985| @@ -216,6 +220,7 @@ You can find that the original input sentence is restored from the vocabulary id |KyTea/SentencePiece|80k/8k|0.1939|0.2141|23.2161|27.0429| |MeCab/SentencePiece|80k/8k|0.1892|0.2077|21.2033|27.0429| |neologd/SentencePiece|80k/8k|0.1641|0.1804|18.4768|27.0429| +|Char|3k (shared)|0.0824|0.0918|33.6963|109.8662| #### Discussion * **SentencePiece (Unigram/BPE)** outperforms word-based methods **(Moses/KyTea/MeCab/neologd)** even with a smaller vocabulary (10% of word-based methods).