2873edeafb
Merge Onigmo 6.0.0
...
* https://github.com/k-takata/Onigmo/blob/Onigmo-6.0.0/HISTORY
* fix for ruby 2.4: https://github.com/k-takata/Onigmo/pull/78
* suppress warning: https://github.com/k-takata/Onigmo/pull/79
* include/ruby/oniguruma.h: include onigmo.h.
* template/encdb.h.tmpl: ignore duplicated definition of EUC-CN in
enc/euc_kr.c. It is defined in enc/gb2313.c with CRuby macro.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@57045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-12-10 17:47:04 +00:00
6ed393ad89
* regenc.h/c, include/ruby/oniguruma.h, enc/ascii.c, big5.c, cp949.c,
...
emacs_mule.c, euc_jp.c, euc_kr.c, euc_tw.c, gb18030.c, gbk.c,
iso_8859_1|2|3|4|5|6|7|8|9|10|11|13|14|15|16.c, koi8_r.c, koi8_u.c,
shift_jis.c, unicode.c, us_ascii.c, utf_16|32be|le.c, utf_8.c,
windows_1250|51|52|53|54|57.c, windows_31j.c, unicode.c:
Remove conditional compilation macro ONIG_CASE_MAPPING. [Feature #12386 ].
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55740 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-24 07:33:15 +00:00
fd7925ffa5
* regenc.h/c: Rename onigenc_not_support_case_map to
...
onigenc_ascii_only_case_map.
* regenc.h: Add definition of onigenc_single_byte_ascii_only_case_map.
* enc/iso_8859_X.c, windows_125X.c, ascii.c, us-ascii.c, koi8_x.c:
Replace onigenc_not_support_case_map by
onigenc_single_byte_ascii_only_case_map.
* enc/big5.c, cp949.c, emacs_mule.c, euc_X.c, gbX.c, shift_jis.c,
windows_31j.c: Replace onigenc_not_support_case_map by
onigenc_ascii_only_case_map.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55305 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-06-07 06:05:18 +00:00
78c5ca7074
* include/ruby/oniguruma.h: Extend OnigEncodingTypeDefine to define a
...
new encoding primitive 'case_map' for case mapping
* enc/utf-8.c, utf_16be/le.c, utf_32be/le.c:
add onigenc_unicode_case_map as case_map primitive
* enc/ascii.c, big5.c, cp949.c, emacs_mule.c, euc_jp/kr/tw.c, gb18030.c,
gbk.c, iso_8859_1/2/3/4/5/6/7/8/9/10/11/13/14/15/16.c, koi8_r/u.c,
shift_jis.c, us_ascii.c, windows_1250/1251/1252.c:
add onigenc_not_support_case_map as case_map primitive
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55113 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-05-22 05:57:44 +00:00
d2a5354255
* reg*.c: Merge Onigmo 5.15.0 38a870960aa7370051a3544
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-15 16:18:41 +00:00
0424e152c6
* Merge Onigmo-5.13.1. [ruby-dev:45057] [Feature #5820 ]
...
https://github.com/k-takata/Onigmo
cp reg{comp,enc,error,exec,parse,syntax}.c reg{enc,int,parse}.h
cp oniguruma.h
cp tool/enc-unicode.rb
cp -r enc/
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34663 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-17 07:42:23 +00:00
3a47cf3395
* remove trailing spaces.
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-05-15 11:55:52 +00:00
b98ea1505c
* enc/trans/big5-hkscs-tbl.rb: Update table as HKSCS-2008.
...
patched by oCameLo oTnTh [ruby-core:33256]
* enc/big5.c: add alias Big5-HKSCS:2008 to Big5-HKSCS.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29922 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-24 16:40:38 +00:00
60dfa6b655
* enc/big5.c: split CP950 from Big5.
...
* enc/big5.c: split CP951 from Big5-HKSCS.
* enc/trans/big5.trans: import conversion table of Big5, Big5-HKSCS,
CP950, and CP951 from ICU. they need fallback conversions.
ref [ruby-core:33256]
http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/
* tool/transcode-tblgen.rb (import_ucm): add to import ucm files.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29869 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-22 09:35:08 +00:00
e0436c54c2
* enc/big5.c, enc/trans/big5.trans, enc/trans/big5-uao-tbl.rb,
...
test/ruby/test-transcode.rb: Added Encoding 'Big5-UAO' and transcoding
for it (from Tatsuya Mizuno) (see Bug #1784 )
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25822 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-11-17 08:56:11 +00:00
8c658137d9
More strict for Big5 series.
...
* enc/big5.c (EncLen_Big5): back to original Big5 table.
(EncLen_Big5_HKSCS): for Big5-HKSCS.
(trans): add the lead byte table for Big5-HKSCS.
(big5_mbc_enc_len): abstract function for Big5 series.
(big5_mbc_enc_len): for Big5.
(big5_hkscs_mbc_enc_len): for Big5-HKSCS.
(BIG5_HKSCS_P): added.
(BIG5_ISMB_FIRST): add routine for Big5-HKSCS.
(big5_hkscs): add for Big5-HKSCS.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24384 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-04 00:51:22 +00:00
4fd615943e
* enc/big5.c: not executable.
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-25 04:42:48 +00:00
a8951a5b3a
* enc/big5.c: Fix EncLen_BIG5 for Big5-HKSCS. see [ruby-core:24390]
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-24 16:15:53 +00:00
2886207584
* enc/trans/big5.trans, big5-hkscs-tbl.rb:
...
new Chinese BIG5-HKSCS transcoding (with Tatsuya Mizuno)
* test/ruby/test_transcode.rb: added tests for the above
(with Tatsuya Mizuno)
* enc/big5.c: Added BIG5-HKSCS as a replicate encoding of BIG5
(short term solution, needs more work; with Tatsuya Mizuno)
* tool/transcode-tblgen.rb: made 'pat' directly accessible in
class StrSet
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-24 10:26:18 +00:00
081c802cb9
* grapheme cluster implementation reverted. [ruby-dev:36375]
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19417 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-18 12:53:25 +00:00
a67d4fa01c
* include/ruby/oniguruma.h (OnigEncodingTypeST): add precise_ret
...
argument for mbc_to_code.
(ONIGENC_MBC_TO_CODE): provide NULL for precise_ret.
(ONIGENC_MBC_PRECISE_CODEPOINT): defined.
* include/ruby/encoding.h (rb_enc_mbc_precise_codepoint): defined.
* regenc.h (onigenc_single_byte_mbc_to_code): precise_ret argument
added.
(onigenc_mbn_mbc_to_code): ditto.
* regenc.c (onigenc_single_byte_mbc_to_code): precise_ret argument
added.
(onigenc_mbn_mbc_to_code): ditto.
* string.c (count_utf8_lead_bytes_with_word): removed.
(str_utf8_nth): removed.
(str_utf8_offset): removed.
(str_strlen): UTF-8 codepoint oriented optimization removed.
(rb_str_substr): ditto.
(enc_succ_char): use rb_enc_mbc_precise_codepoint.
(enc_pred_char): ditto.
(rb_str_succ): ditto.
* encoding.c (rb_enc_ascget): check length with
rb_enc_mbc_precise_codepoint.
(rb_enc_codepoint): use rb_enc_mbc_precise_codepoint.
* regexec.c (string_cmp_ic): add text_end argument.
(match_at): check end of character after exact string matches.
* enc/utf_8.c (graphme_table): defined for extended graphme cluster
boundary.
(grapheme_cmp): defined.
(get_grapheme_properties): defined.
(grapheme_boundary_p): defined.
(MAX_BYTES_LENGTH): defined.
(comb_char_enc_len): defined.
(mbc_to_code0): extracted from mbc_to_code.
(mbc_to_code): use mbc_to_code0.
(left_adjust_combchar_head): defined.
(utf_8): use a extended graphme cluster as a unit.
* enc/unicode.c (onigenc_unicode_mbc_case_fold): use
ONIGENC_MBC_PRECISE_CODEPOINT to extract codepoints.
(onigenc_unicode_get_case_fold_codes_by_str): ditto.
* enc/euc_jp.c (mbc_to_code): follow mbc_to_code field change.
use onigenc_mbn_mbc_to_code.
* enc/shift_jis.c (mbc_to_code): ditto.
* enc/emacs_mule.c (mbc_to_code): ditto.
* enc/gbk.c (gbk_mbc_to_code): follow mbc_to_code field and
onigenc_mbn_mbc_to_code change.
* enc/cp949.c (cp949_mbc_to_code): ditto.
* enc/big5.c (big5_mbc_to_code): ditto.
* enc/euc_tw.c (euctw_mbc_to_code): ditto.
* enc/euc_kr.c (euckr_mbc_to_code): ditto.
* enc/gb18030.c (gb18030_mbc_to_code): ditto.
* enc/utf_32be.c (utf32be_mbc_to_code): follow mbc_to_code field
change.
* enc/utf_16be.c (utf16be_mbc_to_code): ditto.
* enc/utf_32le.c (utf32le_mbc_to_code): ditto.
* enc/utf_16le.c (utf16le_mbc_to_code): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19389 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-16 16:48:05 +00:00
19416601a0
* include/ruby/oniguruma.h (OnigEncodingTypeST): add end argument for
...
left_adjust_char_head.
(ONIGENC_LEFT_ADJUST_CHAR_HEAD): add end argument.
(onigenc_get_left_adjust_char_head): ditto.
* include/ruby/encoding.h (rb_enc_left_char_head): add end argument.
* regenc.h (onigenc_single_byte_left_adjust_char_head): ditto.
* regenc.c (onigenc_get_right_adjust_char_head): follow the interface
change.
(onigenc_get_right_adjust_char_head_with_prev): ditto.
(onigenc_get_prev_char_head): ditto.
(onigenc_step_back): ditto.
(onigenc_get_left_adjust_char_head): ditto.
(onigenc_single_byte_code_to_mbc): ditto.
* re.c: ditto.
* string.c: ditto.
* io.c: ditto.
* regexec.c: ditto.
* enc/euc_jp.c: ditto.
* enc/cp949.c: ditto.
* enc/shift_jis.c: ditto.
* enc/gbk.c: ditto.
* enc/big5.c: ditto.
* enc/euc_tw.c: ditto.
* enc/euc_kr.c: ditto.
* enc/emacs_mule.c: ditto.
* enc/gb18030.c: ditto.
* enc/utf_8.c: ditto.
* enc/utf_16le.c: ditto.
* enc/utf_16be.c: ditto.
* enc/utf_32le.c: ditto.
* enc/utf_32be.c: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19334 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-13 19:23:52 +00:00
0ef21e44e7
forgot to commit
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@17773 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-01 09:50:30 +00:00
ffbf8ab367
* enc/big5.c (big5_mbc_enc_len): validation implemented.
...
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15289 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-28 06:33:57 +00:00
0a640a9386
* enc/*: add ARG_UNUSED.
...
* enc/koi8_u.c: added.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-15 14:54:40 +00:00
edc61cf4c1
* encoding.c (ENC_REGISTER): use &OnigEncoding*.
...
(ENCINDEX_UTF_8): renamed from ENCINDEX_UTF8.
(rb_enc_init): use ENC_REGISTER.
* include/ruby/oniguruma.h (OnigEncodingUTF8, ONIG_ENCODING_UTF8):
removed.
* enc/*.c: remove use of &encoding_*; use enc argument instead.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15067 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-15 13:36:18 +00:00
5b46f99ce1
* enc/*.c: add replicas and aliases.
...
* enc/make_encdb.h: add duplicate and undefined check.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15028 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-13 20:46:00 +00:00
8f15b8128c
* include/ruby/oniguruma.h: remove ONIG_ENCODING_* and OnigEncoding*
...
which are not builtin.
* regenc.{c,h} (onigenc_mb2_code_to_mbclen, onigenc_mb4_code_to_mbclen):
fix prototype.
* enc/big5.c, enc/euc_kr.c, enc/euc_tw.c, enc/gb18030.c,
enc/koi8_r.c, enc/windows_1251.c: imported from Oniguruma.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15026 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-13 17:16:09 +00:00
naruse
duerst
nobu
akr
shyouhei