Unicode Version 12.1.0 adds one single character, U+32FF SQUARE ERA NAME REIWA,
for the new Japanese era starting on May 1st. 12.1.0 will be finalized only on
May 7th, so we go with the beta version because further changes in the data we
need are highly unlikely, and we want to make sure Ruby is ready for the new era.
* common.mk: change UNICODE_VERSION to 12.1.0, UNICODE_BETA to YES
* enc/unicode/12.1.0, enc/unicode/12.1.0/casefold.h, enc/unicode/12.1.0/name2ctype.h:
add directory and generated data files for new version
* lib/unicode_normalize/tables.rb: update for new character
* test/ruby/test_regexp.rb: add test for character property age=12.1
* test/test_unicode_normalize.rb: add test for NFKC decomposition of new character
This (mostly) completes issue #15195.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/unicode_norm_gen.tmpl: Change formatting of output to produce only a
single item (or range) for each line to make future diffs shorter and easier
to understand and check.
* lib/unicode_normalize/tables.rb: output of the above
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67439 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
- common.mk: set UNICODE_VERSION and UNICODE_EMOJI_VERSION to 12.0.0
- lib/unicode_normalize/tables.rb: update table data to Unicode version 12.0.0
- enc/unicode/12.0.0/casefold.h, enc/unicode/12.0.0/name2ctype.h: add generated
files for Unicode version 12.0.0
This is the main commit for #15321.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
- common.mk: Change Unicode version to 11.0.0, and Emoji version to 11.0
- test/ruby/enc/test_emoji_breaks.rb: update hard-coded Emoji version
- enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h:
Add generated files. Files for Unicode 10.0.0 will be removed once we are
sure 11.0.0 works.
- lib/unicode_normalize/tables.rb: Updated table.
- regparse.c: Almost completely reimplement grapheme cluster detection in
function node_extended_grapheme_cluster().
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
- common.mk: Change Unicode version to 11.0.0
- enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with
Gregorian Mtavruli. This should bring us up to the same level as e.g.
Python 3.7, by following the Unicode tables exactly. But it will
produce undesirable (mixed-case) results for String#capitalize.
This will be addressed in a later commit.
- enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h:
Add generated files.
- lib/unicode_normalize/tables.rb: Updated table.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Fix the range check for trailing
Hangul jamo characters in Unicode normalization. Different from
leading or vowel jamos, where LBASE and VBASE are actual characters,
a value equal to TBASE expresses the absence of a trailing jamo.
This fix is technically correct, but there was no bug because
the regular expressions in lib/unicode_normalize/tables.rb
eliminate jamos equal to TBASE from normalization processing.
* test/test_unicode_normalize.rb: Add preventive test
test_no_trailing_jamo based on
d134809cd3
just for the case we ever get a regression.
This closes issue #14934, thanks to MaLin (Lin Ma) for reporting.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
- In common.mk, set UNICODE_VERSION to 10.0.0
- Generate and add enc/unicode/10.0.0/casefold.h and
enc/unicode/10.0.0/name2ctype.h
- Update lib/unicode_normalize/tables.rb
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
In lib/unicode_normalize/normalize.rb, add explanations and clarifications
about the status of the files and the module. [ci skip]
This is in response to discussions at https://github.com/ruby/spec/pull/433
and https://bugs.ruby-lang.org/issues/5481#note-58.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58617 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalized?
(including documentation). Leave a comment explaining that the file is now empty.
* string.c: Define String#unicode_normalized? in rb_str_unicode_normalized_p in C,
(including documentation)
* lib/unicode_normalize/normalize.rb: Remove (re)definition of
String#unicode_normalized? to avoid warnings (when $VERBOSE==true) and
problems when String is frozen
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58555 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalize!
(including documentation)
* string.c: Define String#unicode_normalize! in rb_str_unicode_normalize_bang in C,
(including documentation)
* lib/unicode_normalize/normalize.rb: Remove (re)definition of
String#unicode_normalize! to avoid warnings (when $VERBOSE==true) and
problems when String is frozen
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58553 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize.rb: Remove definition of String#unicode_normalize
(including documentation)
* string.c: Define String#unicode_normalize in rb_str_unicode_normalize in C,
(including documentation)
* lib/unicode_normalize/normalize.rb: Remove (re)definition of
String#unicode_normalize to avoid warnings (when $VERBOSE==true) and
problems when String is frozen
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58550 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
simplify String#unicode_normalize! and #unicode_normalized?
in lib/unicode_normalize.rb by redefining them
in lib/unicode_normalize/normalize.rb
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58538 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
simplify String#unicode_normalize in lib/unicode_normalize.rb
by redefining it in lib/unicode_normalize/normalize.rb
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58537 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* unicode/9.0.0/casefold.h, name2ctype.h, unicode/data/9.0.0:
new directories/files for Unicode version 9.0.0
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When you change this to true, you may need to add more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Version for character normalization data from 7.0.0 to 8.0.0.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52000 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/unicode_norm_gen.tmpl: pragma needs the end marker too,
not only the beginning marker.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51972 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
get rid of many .freeze commands by using frozen_string_literal
pragma.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb (UnicodeNormalize): REGEXP_K
matches only single chars which are keys of KOMPATIBLE_TABLE, so
string in nfkd_one is always single char and one of the key of
KOMPATIBLE_TABLE, that is that the default proc of NF_HASH_K only
copies a pair in KOMPATIBLE_TABLE. therefore NF_HASH_K is a
part of KOMPATIBLE_TABLE always, and just redundant.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49929 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/tables.rb: commit not to download and
convert Unicode data files every time.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48386 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/unicode_norm_gen.tmpl: expand kompatible_table so that
recursive expansion is not needed at runtime.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb (canonical_ordering_one):
use explicit separator, not to depend on $,.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48308 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
as trivially supported encoding (is always normalized,
and may appear mixed in with UTF-8 or other Unicode
encodings).
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48134 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
update easier and more predictable, and reducing compilation
time.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
relevant portion of Unicode standard for Hangul (de)composition
identifiers and algorithm.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb (hangul_decomp_one): use more
descriptive name. leave [SLVT]BASE and [LVTNS]COUNT as they are
vague names.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48055 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
is not hungarian notation. The variable name sIndex is
directly taken from the relevant part of the Unicode
Standard, where it is written SIndex and stands for
'syllable index'. See pp. 144/145 of
http://www.unicode.org/versions/Unicode7.0.0/ch03.pdf.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48052 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb (NF_HASH_{D,C,K}): remove
first element by Hash#shift.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb (UnicodeNormalize): use self
instead of module name and remove module name if unnecessary.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
File name change from lib/unicode_normalize/normalize_tables.rb
to lib/unicode_normalize/tables.rb.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48015 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Martin Dürst
duerst
marcandre
stomar
kazu
naruse
akr
nobu
svn