* string.c (coderange_scan): extracted from rb_enc_str_coderange.
(rb_enc_str_coderange): use coderange_scan.
(rb_str_shared_replace): copy encoding and coderange.
(rb_enc_str_buf_cat): new function for linear complexity string
accumulation with encoding.
(rb_str_sub_bang): don't conflict substituted part and replacement.
(str_gsub): use rb_enc_str_buf_cat.
(rb_str_clear): clear coderange.
* re.c (rb_reg_regsub): use rb_enc_str_buf_cat.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14910 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* regexec.c: unset USE_MATCH_RANGE_MUST_BE_INSIDE_OF_SPECIFIED_RANGE
which is turned on since oniguruma 5.9.1.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14878 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ruby.c, transcode.c: rename rb_ascii_encoding. to
rb_ascii8bit_encoding. rb_ascii_encoding is ambiguous with
ASCII-8BIT and US-ASCII.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(REG_ENCODING_NONE): ditto.
(rb_char_to_option_kcode): return ARG_ENCODING_NONE for n.
(rb_reg_prepare_re): warn /ascii/n =~ "non-ascii".
(rb_reg_initialize): set REG_ENCODING_NONE from ARG_ENCODING_NONE.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14438 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ASCII-8BIT encoding explicitly.
* re.c (rb_reg_prepare_re): add encoding name in the message.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14402 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* re.c (reg_match_pos): pos adjustment should be based on
characters.
* test/ruby/test_m17n.rb (TestM17N::test_str_insert): test updated
to check negative offset behavior.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14340 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
literal is used.
(reg_named_capture_assign_gen): assign the result of named capture
into local variables.
[ruby-dev:32588]
* re.c: document the assignment by named captures.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14297 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
encoding option is specified for regexp literals with \u{}
escapes.
* string.c (rb_str_squeeze_bang): should squeeze multibyte
characters as well.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14275 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
of the regular expression.
* re.c (rb_reg_initialize): fix encoding of regular expression if
embedded string has its own encoding specified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14218 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* string.c (rb_str_sub_bang, str_gsub): should check and copy encoding
to be replaced.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14197 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(rb_reg_named_captures): new method Regexp#named_captures
(match_regexp): new method MatchData#regexp.
(match_names): new method MatchData#names.
* lib/pp.rb (MatchData#pretty_print): show names of named captures.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14163 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
name/number to an integer.
(match_offset): use match_backref_number.
(match_begin): ditto.
(match_end): ditto.
(name_to_backref_number): raise IndexError instead of RuntimeError.
(match_inspect): show capture index.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14158 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(rb_enc_nth): don't check the return value of rb_enc_mbclen.
(rb_enc_strlen): ditto.
(rb_enc_precise_mbclen): return needmore(1) if e <= p.
(rb_enc_get_ascii): new function for extracting ASCII character.
* include/ruby/encoding.h (rb_enc_get_ascii): declared.
* include/ruby/regex.h (ismbchar): removed.
* re.c (rb_reg_expr_str): use rb_enc_get_ascii.
(unescape_escaped_nonascii): use rb_enc_precise_mbclen to determine
the termination of escaped non-ASCII character.
(unescape_nonascii): use rb_enc_precise_mbclen.
(rb_reg_quote): use rb_enc_get_ascii.
(rb_reg_regsub): use rb_enc_get_ascii.
* string.c (rb_str_reverse) don't check the return value of
rb_enc_mbclen.
(rb_str_split_m): don't call rb_enc_mbclen with e <= p.
* parse.y (is_identchar): use ISASCII.
(parser_ismbchar): removed.
(parser_precise_mbclen): new macro.
(parser_isascii): new macro.
(parser_tokadd_mbchar): use parser_precise_mbclen to check invalid
character precisely.
(parser_tokadd_string): use parser_isascii.
(parser_yylex): ditto.
(is_special_global_name): don't call is_identchar with e <= p.
(rb_enc_symname_p): ditto.
[ruby-dev:32455]
* ext/tk/sample/tkextlib/vu/canvSticker2.rb: remove coding cookie
because the encoding is not UTF-8. [ruby-dev:32475]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14131 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* re.c (rb_reg_preprocess): new function for dynamic regexp with
\u{} such as Regexp.new("\\u{6666}").
(rb_reg_prepare_re): preprocess regexp for recompiling.
(read_escaped_byte): new function.
(unescape_escaped_nonascii): new function.
(append_utf8): new function.
(unescape_unicode_list): new function.
(unescape_unicode_bmp): new function.
(unescape_nonascii): new function.
(rb_reg_initialize): preprocess regexp.
* pack.c (rb_uv_to_utf8): renamed from uv_to_utf8.
* parse.y (STR_NEW3): take func instead of has8 and hasmb.
(parser_str_new): use default coderange mechanism except for regexp.
(parser_tokadd_utf8): copy regexp source as-is.
(parser_read_escape): UTF-8 stuff removed.
(parser_tokadd_escape): has8bit and hasmb removed.
(parser_tokadd_string): fix 8-bit single byte character with \u.
(parser_parse_string): has8bit and hasmb removed.
(parser_here_document): has8bit and hasmb removed.
(parser_yylex): call parser_tokadd_utf8 instead of read_escape for
UTF-8 character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
rename ENC_CODERANGE_SINGLE to ENC_CODERANGE_7BIT.
rename ENC_CODERANGE_MULTI to ENC_CODERANGE_8BIT.
Because single byte 8bit character, such as Shift_JIS 1byte katakana,
is represented by ENC_CODERANGE_MULTI even if it is not multi byte.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14027 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(rb_enc_str_asciicompat_p): defined.
* re.c (rb_reg_initialize_str): use rb_enc_str_asciionly_p.
(rb_reg_quote): return ascii-8bit string if the argument is
ascii-only to generate encoding generic regexp if possible.
(rb_reg_s_union): fix encoding handling. [ruby-dev:32094]
* string.c (rb_enc_str_asciionly_p): defined.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14013 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
matches to current encoding.
* re.c (char_to_option, rb_char_to_option_kcode): 'n' is not kcode
option now.
* re.c (rb_reg_to_s, rb_reg_error_desc): copy encoding rather than
append as an option.
* re.c (make_regexp, rb_reg_prepare_re): use encoding of Regexp and
String instead of kcode.
* re.c (rb_reg_initialize): set fixed option if none is set.
* re.c (rb_reg_regcomp): ditto.
* re.c (rb_reg_equal): check if encodings are equal.
* re.c (rb_reg_initialize_m): encoding option is obsolete.
* re.c (rb_kcode, rb_get_kcode, rb_set_kcode): removed.
* re.c (Init_Regexp): removed Regexp#kcode method.
* ruby.c (proc_options): allow long encoding name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* re.c (Init_Regexp): new method Regexp#encoding.
* string.c (str_encoding): moved to encoding.c
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13613 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
condition statement much shorter, if no else clause is needed.
* string.c (rb_str_match_m): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13475 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* string.c (rb_str_rindex_m): was confusing character offset and
byte offset.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
object or nil if it's not target-type. this mechanism is used
to convert types in the C implemented methods.
* hash.c (rb_hash_s_try_convert): ditto.
* io.c (rb_io_s_try_convert): ditto.
* re.c (rb_reg_s_try_convert): ditto.
* string.c (rb_str_s_try_convert): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* re.c (rb_reg_error_desc): make RegexpError for initialization error.
* re.c (rb_reg_compile): return nil and set errinfo if error.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13092 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
between int and string.
* re.c (rb_reg_compile): append regexp options to error message.
[ruby-dev:31334]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@12863 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
literal. [ruby-dev:31336]
* re.c (rb_reg_compile): should not use regexp which could not get
initialized. [ruby-dev:31333]
return error message to let the parser know it.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@12862 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* regsyntax.c, unicode.c: new files along with Oniguruma 5.x.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@12376 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
akr
nobu
matz