In the function onigenc_unicode_case_map() in enc/unicode.c, deal
with the case that the ONIGENC_CASE_IS_TITLECASE flag is set on
lowercase characters. This is in preparation for Georgian Mtavruli,
which are uppercase but not titlecase, in Unicode 11.0.0.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
In ASCII, 'a' is bigger than 'A'. Which means 'A' - 'a' is a negative
number (-32, to be precise). In C, the type of 'a' and 'A' are signed
int (cf: ISO/IEC 9899:1990 section 6.1.3.4). So 'A' - 'a' is also a
signed int. It is `(signed int)-32`.
The problem is, OnigCodePoint is unsigned int. Adding a negative
number to a variable of OnigCodepoint (`code` here) introduces an
unintentional cast of `(unsigned)(signed)-32`, which is
4,294,967,264. Adding this value to code then overflows, and the
result eventually becomes normal codepoint.
The series of operations are not a serious problem but because
`code >= 'a'` holds, we can `(code - 'a') + 'A'` to reroute this.
See also: https://github.com/k-takata/Onigmo/pull/107
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65752 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
- common.mk: Change Unicode version to 11.0.0
- enc/unicode/case-folding.rb, enc/unicode.c: Initial changes to deal with
Gregorian Mtavruli. This should bring us up to the same level as e.g.
Python 3.7, by following the Unicode tables exactly. But it will
produce undesirable (mixed-case) results for String#capitalize.
This will be addressed in a later commit.
- enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h:
Add generated files.
- lib/unicode_normalize/tables.rb: Updated table.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: moved additional Grapheme Cluster Break ranges
which depend on the Unicode version.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* regparse.c (node_extended_grapheme_cluster): as Unicode 10 has
added Grapheme_Cluster_Break properties to some characters,
remove duplicated ranges for Unicode 9.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: Remove special processing for U+03B9/U+03BC/U+A64B
(GREEK SMALL LETTERs IOTA/MU, CYRILLIC SMALL LETTER MONOGRAPH UK)
from onigenc_unicode_case_map and simplify code.
* enc/unicode/case-folding.rb: Remove check for U+03B9/U+03BC/U+A64B.
This and the previous few related commits make sure that we won't hit
the equivalent of bug #12990 anymore for future updates of Unicode versions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56976 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: Add U+A64B to the special cases 03B9 and 03BC
at the end of onigenc_unicode_case_map (Bug #12990).
* enc/unicode/case-folding.rb: Add U+A64B to the special cases
03B9 and 03BC. Add a comment pointing to enc/unicode.c.
Change warnings to exceptions for unpredicted cases,
because this would have been more easily noticed
(the warning was not noticed when upgrading to Unicode 9.0.0).
* test/ruby/enc/test_case_comprehensive.rb: Remove temporary
exclusion of U+A64B from testing.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* common.mk, enc/depend (casefold.h, name2ctype.h): move to
unicode data directory per version.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55701 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
case mapping methods.
* enc/unicode.c: Check for invalid string and signal with negative
length value.
* test/ruby/enc/test_case_mapping.rb: Add tests for above.
* test/ruby/test_m17n_comb.rb: Add a message to clarify test failure.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55253 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
swapcase functionality for titlecase characters. Swapcase isn't defined
by Unicode, because the purpose/usage of swapcase is unclear anyway.
The implementation follows a proposal from Nobu, swaping the case of
each component of a titlecase character individually.
This means that the titlecase characters have to be decomposed.
* enc/unicode.c: Code using the above data.
* test/ruby/enc/test_case_mapping.rb: Tests for the above.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54469 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
special cases in CaseUnfold_11_Table.
* enc/unicode.c: Adjustments for above.
* test/ruby/enc/test_case_mapping.rb: Tests for the above: Some tests in
test_titlecase activated; test_greek added. A test in test_cherokee fixed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54383 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode/case-folding.rb, casefold.h: Using above flag in data.
* enc/unicode.c: Marking capitalized character as unmodified if it is
already titlecase.
* test/ruby/enc/test_case_mapping.rb: Tests for above functionality.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54229 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/enc/test_case_mapping.rb: Test cases that detected
the above bugs.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54140 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
and macros to work with unified CaseMappingSpecials array.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54101 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
case mapping data not available from case folding by unifying all
three cases (special title, special upper, special lower).
* enc/unicode.c: Adjust macro names for above (macros are currently inactive).
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
space for titlecase indices; adding additional macros to add or
extract titlecase index; adding comments for better documentation.
* enc/unicode.c: Moving some macros to include/ruby/oniguruma.h;
activating use of titlecase indices.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
data (new table, with indices from other tables).
* enc/unicode.c: Ignoring titlecase data indices for the moment.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53906 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
single-letter; use flags in casefold.h for logic.
* enc/unicode/case-folding.rb: Added flag for case folding.
Changed parameter passing.
* enc/unicode/casefold.h: New flags added.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53775 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/unicode.c: Added shortening macros for enc/unicode/casefold.h
* enc/unicode/case-folding.rb: Fixed file encoding for CaseFolding.txt
to ASCII-8BIT (should fix some ci errors). Clarified usage. Created
class MapItem. Partially implemented class CaseMapping.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53767 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
String#downcase :fold.
* enc/unicode.c: Fixed a range error (lowest non-ASCII character affected
by case operations is U+00B5, MICRO SIGN)
* test/ruby/enc/test_case_mapping.rb: Explicit test for case folding of
MICRO SIGN to Greek mu.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53749 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
option for String#downcase by using case folding data from
regular expression engine, and added a few simple tests.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53747 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
be able to use the remaining bits for flags.
(with Kimihito Matsui)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
duerst
shyouhei
nobu
naruse
svn
kazu