sentencepiece/doc/normalization.md

Use custom normalization rule

By default, SentencePiece normalizes the input sentence with a variant of Unicode NFKC.

SentencePiece allows us to define custom normalization rule, which is stored in the model file.

Use pre-defined normalization rule

SentencePiece provides the following pre-defined normalization rule. It is recommended to use one of them unless you have any special reasons.

• nmt_nfkc: NFKC normalization with some additional normalization around spaces. (default)
• nfkc: original NFKC normalization.
• nmt_nfkc_cf: nmt_nfkc + Unicode case folding (mostly lower casing)
• nfkc_cf: nfkc + Unicode case folding.
• identity: no normalization

You can choose the normalization rule with --normalization_rule_name flag.

% spm_train --normalization_rule_name=identity --input=<input> --model_prefix=<model file> --vocab_size=8000                                                                                                                                                                                

NOTE: Due to the limitation of normalization algorithm, full NFKC normalization is not implemented. [builder.h] describes example character sequences not normalized by our NFKC implementation.

The difference between nmt_nfkc and nfkc can be found via diff -u data/nfkc.tsv data/nmt_nfkc.tsv command.

Use custom normalization rule

The normalization is performed with user-defined string-to-string mappings and leftmost longest matching.

You can use custom normalization rule by preparing a TSV file formatted as follows:

41 302 300      1EA6
41 302 301      1EA4
41 302 303      1EAA
...

In this sample, UCS4 sequence [41 302 300] (hex) is converted into [1EA6] (hex). When there are ambiguities in the conversions, the longest rule is used. Note that the tab is used as a delimiter for source and target sequence and space is used as a delimiter for UCS4 characters. We can make the target sequence empty to remove some specific characters from the text. See data/nfkc.tsv as an example. Once a TSV file is prepared, you can specify it with --normalization_rule_tsv flag.

% spm_train --normalization_rule_tsv=<rule tsv file> --input=<input> --model_prefix=<model file> --vocab_size=8000                                                                                                                                                                             

<model file> embeds the normalization rule so the same normalization rule is applied when <model file> is used.

Command line tool to perform normalization

% spm_normalize --model=<model_file> file1 file2.. 
% spm_normalize --normalizatoin_rule_tsv=custom.tsv file1 file2..

The first command line uses the normalization rule embedded in the model file. The second command line uses the normalization rule in TSV file and is useful to make normalization rule interactively.