Граф коммитов

653 Коммитов

Автор SHA1 Сообщение Дата
svn 3e82668e62 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53768 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 13:10:21 +00:00
duerst 49ca434bcf * common.mk: Added two more precondition files for enc/unicode/casefold.h
* enc/unicode.c: Added shortening macros for enc/unicode/casefold.h
* enc/unicode/case-folding.rb: Fixed file encoding for CaseFolding.txt
  to ASCII-8BIT (should fix some ci errors). Clarified usage. Created
  class MapItem. Partially implemented class CaseMapping.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53767 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 13:10:20 +00:00
duerst e40d117339 * enc/unicode/case-folding.rb: Fixing parameter passing.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53765 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 05:12:44 +00:00
duerst 5cbe9fe5c4 * enc/unicode/case-folding.rb: New classes CaseMapping/CaseMappingDummy
to pass as parameters; not yet implemented or used.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53764 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 02:44:14 +00:00
duerst ee696372a8 * common.mk: using new option in recipe for enc/unicode/casefold.h
* enc/unicode/case-folding.rb: Correctly specify argument to new option.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 02:16:12 +00:00
svn b0b7cb5a46 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53760 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 01:39:27 +00:00
duerst 07ad033850 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e 2016-02-07 01:39:26 +00:00
duerst e8dde46b60 * test/ruby/enc/test_regex_casefold.rb: Added data-based testing for
String#downcase :fold.
* enc/unicode.c: Fixed a range error (lowest non-ASCII character affected
  by case operations is U+00B5, MICRO SIGN)
* test/ruby/enc/test_case_mapping.rb: Explicit test for case folding of
  MICRO SIGN to Greek mu.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53749 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-06 06:18:38 +00:00
duerst 81515b2381 * enc/unicode.c, test/ruby/enc/test_case_mapping.rb: Implemented :fold
option for String#downcase by using case folding data from
  regular expression engine, and added a few simple tests.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53747 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-06 05:37:29 +00:00
duerst b658249cef * enc/unicode.c: Activated :ascii flag for ASCII-only case conversion
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53740 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-04 12:05:23 +00:00
nobu 7373320615 enc: workdir timestamps
* enc/depend: make timestamps for each work directory, instead of
  making for each compilation and link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53714 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-02 04:39:38 +00:00
duerst a7c987968d * enc/unicode.c: Fixed bit mask in macro OnigCodePointCount
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53670 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-27 09:54:38 +00:00
duerst 415949faba * enc/unicode.c: Protect code point count by macro, in order to
be able to use the remaining bits for flags.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-27 08:55:40 +00:00
duerst f307d1fe21 * enc/unicode.c: Fixed a logical error and some comments.
* test/ruby/enc/test_case_mapping.rb: Made tests more general.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53564 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 11:10:45 +00:00
nobu 39f44f0113 get rid of non-ascii chars
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53563 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 09:03:11 +00:00
duerst 959bbb6f72 * enc/unicode.c: Removed artificial expansion for Turkic,
added hand-coded support for Turkic, fixed logic for swapcase.
* string.c: Made use of new case mapping code possible from upcase,
  capitalize, and swapcase (with :lithuanian as a guard).
* test/ruby/enc/test_case_mapping.rb: Adjusted for above.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 08:42:16 +00:00
duerst c12af76763 * enc/unicode.c: Artificial mapping to test buffer expansion code.
* string.c: Fixed buffer expansion logic.
* test/ruby/enc/test_case_mapping.rb: Tests for above.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53554 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 08:24:58 +00:00
hsbt 219467abde * enc/unicode.c: fix implicit conversion error with clang. fixup r53548.
* string.c: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53552 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 01:51:58 +00:00
duerst be897c2507 * string.c, enc/unicode.c: New code path as a preparation for Unicode-wide
case mapping. The code path is currently guarded by the :lithuanian
  option to avoid accidental problems in daily use.
* test/ruby/enc/test_case_mapping.rb: Test for above.
* string.c: function 'check_case_options': fixed logical errors

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 01:24:03 +00:00
nobu fabb8b45cd enc/Makefile.in: get rid of nmake bug
* enc/Makefile.in (ECHO1): expand NULLCMD by configured value to
  get rid of a bug of nmake, that it can expand bare single name
  variable but cannot in substition.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53437 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-05 00:48:04 +00:00
hsbt 701bfa0fed * enc/x_emoji.h: fix dead-link.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53345 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 11:00:36 +00:00
nobu bff89a030a depend: version dependency
* enc/depend (enc, trans): fix version dependency, let encoding
  and transcoding shared object files depend on config.status,
  instead of enc.mk which is regenerated at each build, for the
  RUBY_SO_NAME value used at runtime link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53325 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 01:51:50 +00:00
nobu c7b2450e89 depend: version dependency
* enc/depend (enc, trans): fix version dependency, shared object
  files depend on the RUBY_SO_NAME value for runtime link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53324 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 01:03:16 +00:00
ngoto a2bf609304 * enc/windows_1250.c: Should not use C++ style comments (C99 feature).
[Bug #11843]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53202 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-19 11:09:28 +00:00
duerst 6973c8d51e enc/iso_8859_2.c, enc/windows_1250.c: separate Windows-1250
from ISO-8859-2 to fix 0x80..0x9e range (from Kimihito Matsui)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53198 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-19 01:33:38 +00:00
naruse abfc03c6cf follow the change of the name
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53128 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 13:11:33 +00:00
duerst 1db69bf3ef reverting accidental commit at r53124 by re-committing version from r52612
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53127 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 10:25:44 +00:00
duerst 81e7be08ed enc/ebcdic.h, enc/trans/ebcdic.trans,
test/ruby/test_transcode.rb: Fixed encoding name
to the correct one in the IANA registry (IBM037)
and added an alias (ebcdic-cp-us)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53124 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 08:57:58 +00:00
svn 3473c9e7d4 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53113 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 13:11:59 +00:00
duerst 83304b75c1 * enc/ebcdic.h: new dummy encoding EBCDIC-US
* enc/trans/ebcdic.trans: transcodings between EBCDIC-US
  and iso-8859-1 [with code from Andrea Ribuoli]
* test/ruby/test_transcode.rb: tests for above
* tool/transcode_tablegen.rb: additional argument for
  method transcode_tblgen

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53112 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 13:11:31 +00:00
hsbt 43ba3280c3 * enc/iso_2022_jp.h: fix typos.
[ci skip][fix GH-1140] Patch by @jutaz
* enc/utf_16_32.h: ditto.
* enc/utf_7.h: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53102 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 02:50:21 +00:00
nobu 3ee57230b3 enc/windows_1252.c: new
* enc/windows_1252.c: separate from ISO-8859-1 to fix 0x80..0x9e
  range.  [ruby-core:64049] [Bug #10097]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-11 22:05:31 +00:00
svn c949208d32 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52885 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-04 10:53:16 +00:00
duerst e1f69b39ae enc/iso_8859_13.c: Added three missing lower/upper-case
character pairs (from Kimihito Matsui)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52884 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-04 10:52:54 +00:00
duerst dddbd864d1 enc/iso_8859_4.c: Added missing lower/upper-case character
pair (U+014A and U+014B, LATIN CAPITAL/SMALL LETTER ENG)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52883 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-04 09:57:59 +00:00
naruse 9ed1d63f41 * regcomp.c, regenc.c, regexec.c, regint.h, enc/unicode.c:
Merge Onigmo 58fa099ed1a34367de67fb3d06dd48d076839692
  + https://github.com/k-takata/Onigmo/pull/52

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52756 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-26 08:31:27 +00:00
duerst 5d3cdd6556 enc/unicode/casefold.h, name2ctype.h: Change Unicode Version for
regular expressions from 7.0.0 to 8.0.0
(with help from Kimihito Matsui) [Feature #11563]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52612 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-17 06:30:39 +00:00
naruse 95d8076081 refix through 52016,52017,52019,52020 [Bug #11486]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52021 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-10-04 07:25:56 +00:00
nobu ec0793e437 euc_jp.c: fix compile error
* enc/euc_jp.c (mbc_case_fold): no longer use get_lower_case
  result.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52020 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-10-04 06:56:58 +00:00
naruse db152cd2dd * enc/euc_jp.c (mbc_case_fold): check given string is valid or not,
and if invalid, return 1. [Bug #11486]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52016 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-10-04 06:13:48 +00:00
nobu 75a1d33bcf enc: fundamental encindex
* enc/{ascii,us_ascii,utf_8}.c: set encoding indexes of
  fundamental built-in encodings so that usable as well as
  allocated rb_encoding before rb_enc_init().

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51862 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-09-15 05:49:52 +00:00
nobu 542e1d8bc4 make_encmake.rb: expand srcdir
* enc/make_encmake.rb: @srcdir@ in enc/Makefile.in needs to be
  expanded.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51770 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-09-05 15:32:20 +00:00
nobu c56dd7fd05 unicode.c: no st.h
* enc/unicode.c: no longer use st.h.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51711 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-08-27 19:50:23 +00:00
nobu 8b9ad9d6f0 oniguruma.h: constify
* include/ruby/oniguruma.h (OnigEncodingTypeST): constify
  property_name_to_ctype arguments.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@51710 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-08-27 19:49:45 +00:00
mame e4fa17a9b9 * enc/make_encmake.rb: the list of encoding extension libraries must
not include encinit.c itself.  It caused "undefined reference to
  Init_encinit".

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-06-19 19:02:45 +00:00
mame e8ce2a92d3 * enc/make_encmake.rb: added --transes and --no-transes options.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50957 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-06-18 14:03:51 +00:00
nobu cf44e74092 make_encmake.rb: mandatory dbs
* enc/make_encmake.rb (target_transcoders, target_encodings):
  encdb and transdb are always mandatory.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-06-18 13:30:44 +00:00
nobu 859f88f330 enc/utf_8.c: limit UTF-8
* enc/utf_8.c (code_to_mbclen, code_to_mbc): reject values larger
  than UTF-8 max codepoints.  [Feature #11094]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50392 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-04-25 22:36:52 +00:00
nobu 4c3d694fb4 common.mk: make enc/trans
* common.mk: buildtin encoding and transcoder objects need output
  directories when out-place build.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49326 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-18 13:12:15 +00:00
nobu fbbf3afc11 builtin encodings and transcoder locations
* Makefile.in (VPATH, NEWLINE_C), common.mk (common-srcs): make
  and use newline.c under enc/trans directory, not toplevel.  no
  longer search enc directory implicitly.
* configure.in, enc/Makefile.in (BUILTIN_ENCS, BUILTIN_TRANSES):
  prefix respective directory names to builtin encodings and
  transcoder source names.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-18 03:56:52 +00:00
nobu 378480e172 enc/unicode/name2ctype.h.blt: update for r46831
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-17 02:45:20 +00:00
akr ecedd3b224 Update dependencies.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48898 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-19 13:34:54 +00:00
nobu 638ba47303 load.c: defer static linked init
* enc/encinit.c.erb (Init_enc): initialize encdb and transdb
  directly.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48699 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-03 07:47:37 +00:00
nobu 673f1b3064 enc/prelude.rb: no encdb and transdb
* enc/prelude.rb: no longer need to load encdb and transdb here.
  Init_enc should load them if possible.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48698 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-03 07:47:11 +00:00
nobu 8299027f2e make_encmake.rb: fix typo
* enc/make_encmake.rb: fix typo, and use real read filename.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48621 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-27 16:21:51 +00:00
akr 6ca202fcee Update dependency.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-21 03:04:44 +00:00
akr 20fe728a24 Dependencies updated.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48483 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-18 15:28:08 +00:00
akr 4ac7afa500 * enc/encdb.c: Include internal.h.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48481 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-18 15:24:41 +00:00
akr 2b9191e557 * internal.h: Gather declarations in non-header files.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48480 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-18 15:13:05 +00:00
akr dd547e1b22 Mark auogenerated part.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-15 11:14:08 +00:00
akr 57682b05e1 * common.mk: Remove comments in Dependency lines.
Notified by usa.

* enc/depend: Ditto.

* ext/**/depend: Ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48434 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-15 10:28:25 +00:00
akr 8727eb77be * tool/update-deps: Extend to fix dependencies.
* common.mk: Dependencies updated by tool/update-deps.

* enc/depend: Ditto.

* ext/**/depend: Ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48431 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-15 05:15:49 +00:00
usa ee4782f774 * enc/prelude.rb: we sometimes run ruby without library path (especially
for test), so should permit to run ruby if unicode_normalize.rb is
  missing.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48060 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-20 15:01:13 +00:00
duerst 824670aa9e enc/prelude.rb: Added automatic loading of lib/unicode_normalize.rb. This makes sure that all
the methods that are available on String are available without explicit require.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48023 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-10-19 09:39:41 +00:00
naruse 106075eac8 Re-apply r43023 which is accidentally reverted by r47598
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47615 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-17 14:53:08 +00:00
naruse d2a5354255 * reg*.c: Merge Onigmo 5.15.0 38a870960aa7370051a3544
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-09-15 16:18:41 +00:00
nobu f587347c28 euckr-tbl.rb: euro and registered signs
* enc/trans/euckr-tbl.rb (EUCKR_TO_UCS_TBL): add missing euro and
  registered signs.  [ruby-core:64452] [Bug #10149]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47221 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-08-19 13:22:46 +00:00
naruse 64c81e40d4 * regcomp.c: Merge Onigmo 5.14.1 25a8a69fc05ae3b56a09.
this includes Support for Unicode 7.0 [Bug #9092].

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-07-16 03:27:25 +00:00
kazu 72912ec3f3 fix usage
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-06-02 13:46:33 +00:00
nobu 046831094b constify rb_encoding and OnigEncoding
* include/ruby/encoding.h: constify `rb_encoding` arguments.
* include/ruby/oniguruma.h: constify `OnigEncoding` arguments.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-06-01 22:06:11 +00:00
nobu dd20f90408 unicode.c: no initialization
* enc/unicode.c (init_case_fold_table): no longer need to
  initialize tables at runtime.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46273 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:58:34 +00:00
nobu 7e67b39679 case-folding.rb: perfect hash for case unfolding3
* enc/unicode/case-folding.rb (lookup_hash): make perfect hash to
  lookup case unfolding table 3.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46272 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:58:24 +00:00
nobu 8f59867651 case-folding.rb: perfect hash for case unfolding2
* enc/unicode/case-folding.rb (lookup_hash): make perfect hash to
  lookup case unfolding table 2.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46271 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:58:14 +00:00
nobu 35348a0806 case-folding.rb: perfect hash for case unfolding1
* enc/unicode/case-folding.rb (lookup_hash): make perfect hash to
  lookup case unfolding table 1.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46270 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:58:01 +00:00
nobu c39e659263 case-folding.rb: perfect hash for case folding
* enc/unicode/case-folding.rb (lookup_hash): make perfect hash to
  lookup case folding table.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:57:45 +00:00
nobu 88eae35862 case-folding.rb: merge tables
* enc/unicode/case-folding.rb (print_table): merge non-locale and
  locale tables, and reduce initializing loops.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46268 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:56:00 +00:00
nobu 1d588b4fde case-folding.rb: modularize
* enc/unicode/case-folding.rb (CaseFolding): modularize, and add
  --output-file option.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:55:00 +00:00
nobu 0148bd15e4 case-folding.rb: conversion script
* enc/unicode/case-folding.rb: script to convert CaseFolding.txt,
  tranlated from CaseFolding.py.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46266 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-30 23:49:54 +00:00
nobu 90fb7538f8 enc/unicode.c: lookup functions
* enc/unicode.c (onigenc_unicode_{fold,unfold{1,2,3}}_lookup):
  abstract lookup functions.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46057 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-23 04:44:15 +00:00
nobu dcaf699ee9 enc/unicode.c: constify
* enc/unicode.c (code{2,3}_{cmp,hash}): constify and adjust
  argument types.

* enc/unicode.c (onigenc_unicode_fold_lookup): constify.

* enc/unicode.c (onigenc_unicode_apply_all_case_fold): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46056 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-23 04:37:02 +00:00
nobu 7f2c371fba Makefile.in: suppress warnings
* Makefile.in (enc/jis/props.h): suppress `pointer-to-int-cast`
  warnings by using `offsetof`.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46052 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-22 15:09:11 +00:00
nobu 095e1f8eaf jis/props.kwd: constify
* enc/jis/props.kwd (PropertyList): constify table itself.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46051 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-22 14:34:58 +00:00
nobu bd403755d0 remove trailing spaces
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-22 10:58:08 +00:00
nobu f1b49b7bb9 constify character property tables
* enc/jis/props.kwd: constify character property tables of JIS
  based encodings by perfect hash.
* enc/euc_jp.c, enc/shift_jis.c: use character property functions.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@46039 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-05-22 05:02:15 +00:00
nobu 6224216294 unused functions and variables
* complex.c, rational.c: remove unused functions, which are warned
  by clang 5.1, and also variables only used by removed functions.

* ext/date/date_core.c: ditto.

* enc/utf_16be.c, enc/utf_16le.c: comment out constants only used
  by commented out functions.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@45354 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-03-17 04:57:14 +00:00
nobu d8441fcc6e encdb.c, encoding.c: make BOM-encodings dummy
* enc/encdb.c (ENC_DUMMY_UNICODE): make BOM-encodings dummy.
* encoding.c (enc_autoload): keep dummy encodings dummy.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@43034 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-09-24 08:15:18 +00:00
nobu 3a2aca5075 encdb.c, utf_16_32.h: Unicode with BOM
* enc/encdb.c, enc/utf_16_32.h (ENC_DUMMY_UNICODE): Unicode with BOM
  must be based on big endian variants, so that actual encodings would
  work.  [ruby-core:57318] [Bug #8940]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@43023 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-09-23 13:16:11 +00:00
naruse 0e92ae9636 * enc/trans/utf8_mac-tbl.rb: fix r42789.
Fix conversion table and logic. [ruby-dev:47680]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42823 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-09-04 06:40:39 +00:00
naruse 853c346dde * enc/trans/utf8_mac-tbl.rb: update conversion table to recent OS X.
Previous table is used on Mac OS X 10.1 or prior.
  This table is used on 10.2 or later. [ruby-dev:47680]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42789 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-09-02 23:36:01 +00:00
naruse 9ae509504b Revert r41786 "broken utf-8"
revert unexpected commit

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41855 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-07-09 07:38:15 +00:00
naruse ed37bfcc9d broken utf-8
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41786 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-07-05 00:54:11 +00:00
ktsj 166d8dc2d6 * enc/trans/japanese_euc.trans, test/ruby/test_transcode.rb,
tool/transcode-tblgen.rb: change EUC-JP-2004 to EUC-JIS-2004.
  This is follow up to changes in r41024.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41035 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-06-02 14:36:41 +00:00
naruse b285af6468 * enc/euc_jp.c: fix typo: the name of EUC-JIS-2004.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@41024 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-06-01 17:11:48 +00:00
nobu a436540271 ext/depend: fix dependency for nmake
* ext/depend (ENCOBJS, TRANSOBJS): use explicit path to ruby.h for
  nmake.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40187 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-08 14:56:01 +00:00
nobu 1d42889c4e ext/depend: fix dependency
* ext/depend (ENCOBJS, TRANSOBJS): fix header dependency, VPATH has
  $(srcdir)/include/ruby but not $(srcdir)/include, so cannot find out
  ruby/ruby.h.  use ruby.h instead and ../ruby for include/ruby.h.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40186 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-08 14:09:42 +00:00
yugui 4fe59003f0 * ext/depend (ENCOBJS, TRANSOBJS): Add missing dependencies.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@40185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-04-08 11:31:38 +00:00
nobu 773248551b mswin: static-linked-ext
* enc/depend (CPPFLAGS), lib/mkmf.rb (MakeMakefile#create_makefile):
  define RUBY_EXPORT for static-linked-ext mswin.  [Bug #7960]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@39556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-03-02 15:04:23 +00:00
naruse 78dbaa1648 * Merge Onigmo 0fe387da2fee089254f6b04990541c731a26757f
v5.13.3 [Bug#7972] [Bug#7974]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@39547 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-03-01 16:36:37 +00:00
nobu be26224be5 depend: fix for static library on mswin
* enc/depend (ARFLAGS, RANLIB): these values can be nil.  [Bug #7950]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@39490 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-02-25 06:59:10 +00:00
nobu 72aee03262 depend: fix for static library on mswin
* enc/depend (ARFLAGS): VisualC++ linker does not allow spaces between
  output option and the output file name.  [Bug #7950]
* enc/depend (RANLIB): set default command to do nothing, or make the
  entire line a label on Windows.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@39489 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2013-02-25 06:47:23 +00:00
nobu fa8c97438f * enc/depend (link_so): prefix $(Q) for each commands.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@38290 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-12-10 04:32:11 +00:00
nobu 928d1a5261 encoding.c: unicode_p
* enc/encdb.c, enc/utf_16_32.h (ENC_DUMMY_UNICODE): endian-less wide
  UTF encodings are dummy but Unicode.
* encoding.c (rb_encdb_set_unicode): set Unicode flag.
* template/encdb.h.tmpl: allow ENC_DUMMY variants.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@37101 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-10-06 00:42:14 +00:00
usa 7d59a96314 * enc/depend: fixed wrong change in a part of r34802.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36945 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-09-10 01:20:16 +00:00
nobu e6de05d7c4 enc/depend: fix inplace-build condition
* enc/depend: fix inplace-build condition.  enc.mk is generated with
  setting $srcdir to enc, but pwd is still top build direcory.
  [ruby-core:47236] [Bug #6888]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36725 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-08-18 00:57:48 +00:00
nobu fbee46fe66 enc/Makefile.in: TARGET_NAME TARGET_ENTRY
* enc/Makefile.in (TARGET_NAME, TARGET_ENTRY): needed for EXTDLDFLAGS
  on some platforms.  [ruby-core:46600] [Bug #6768]
* enc/depend: no longer needs tweaking DLDFLAGS for TARGET names.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-07-23 14:58:42 +00:00
naruse 06d483006c * Makefile.in: don't remove macros. now name2ctype uses macros.
* tool/enc-unicode.rb: add comment why it uses Hash#index.

* enc/unicode/{name2ctype.kwd,name2ctype.src,name2ctype.h.blt}:
  update to follow the current name2ctype.h.
  FYI current Unicode version is 6.1.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@36070 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-06-13 17:54:14 +00:00
nobu fb2e6803b0 enc: fix dependencies
* enc/depend (ENCOBJS): add dependencies.
* enc/make_encmake.rb (target_encodings): extract dependencies.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35734 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-05-21 03:11:28 +00:00
naruse 2053b5f453 * enc/shift_jis.c (code_to_mbclen): return
ONIGERR_INVALID_CODE_POINT_VALUE if the code is invalid.

* enc/shift_jis.c (tr_next): increment character until the code
  is a valid character. [ruby-dev:45652] [Bug #6450]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35724 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-05-20 13:32:16 +00:00
nobu 08b1523553 no K&R style
* enc/encinit.c.erb (Init_enc): no K&R style anymore.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35711 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-05-19 03:44:04 +00:00
nobu 7333fbf0f7 enc/encinit.c.erb: use %-lines
* enc/encinit.c.erb: use %-lines to adjust indent in the generated file.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35670 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-05-17 01:39:39 +00:00
yugui 3fa3f9abb9 Supports static linking of extensions and encodings again.
Fixes --with-static-linked-ext.

Patch by Google Inc. [ruby-core:45073].

* Makefile.in (ENCOBJS, EXTOBJS): New variables to specify static
  linked libraries. Also reintroduces extinit.o, introduces encinit.o
  introduces encinit.o

* common.mk: Builds static libraries rather than shared objects if
  specified.

* configure.in (LD): new substitution. 
  Avoids PIE if s

* enc/depend: Supports static linked libraries
  (libencs, libenc, libtrans): New target.

* enc/encinit.c.erb: new template to generate the initialization of
  statically linked encodings.

* enc/make_encmake.rb (--module): new flag to specify whether static
  or dynamic.

* transcode_data.h (TRANS_INIT): New macro to get rid of the name
  collision of encoding initializers and transcoder initializers.

* ext/extmk.rb: Fixes the behavior on $extstatic is true.

* lib/mkmf.rb (clean-static): new target to clean up static linked
  libraries.

* ruby.c (process_options): New initializes statically linked
  encodings here.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35662 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-05-16 05:39:06 +00:00
usa 756ffef448 * enc/euc_jp.c: added EUC-JP-2004 and its alias EUC-JISX0213.
[ruby-dev:45571] [Feature #6349]
  Requested by Kyouhei Yanagita <yanagi@shakenbu.org>.

* enc/trans/japanese_euc.trans: ditto.

* enc/trans/JIS/JISX0213-[12]%UCS@{BMP,SIP}.src: JIS X 0213:2004 ->
  Unicode mapping table from NetBSD.

* enc/trans/JIS/UCS@{BMP,SIP}%JISX0213-[12].src: Unicode -> JIX X
  0213:2004 mapping table from NetBSD.

* tool/transcode-tblgen.rb: added SIP support.

* test/ruby/test_transcode.rb: tests of above changes.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-24 11:14:18 +00:00
nobu 8354d2f700 * enc/unicode/name2ctype.h, tool/transcode-tblgen.rb: revert
unlogged changes which committed by accident probably.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35247 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-06 18:17:01 +00:00
duerst 6111599230 struct.c (documentation for rb_struct_members_m):
fix 'array of strings' to 'array of symbols'
  [ruby-core:44152][Bug #6264]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@35244 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-04-06 07:23:40 +00:00
naruse 291fa223cf * regparse.c (is_onechar_cclass): optimize character class
Merge Onigmo 27278c12e6674043cc8affca6507e20e119a86ee.

* regparse.c (is_onechar_cclass): [bug] unexpected match occurs when a
  char class contains no char

* enc/unicode.c (init_case_fold_table): define the sizes of case
  folding tables in casefold.h

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34860 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-29 16:29:06 +00:00
naruse 57657a0b9b suppress warnings.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34802 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-25 07:08:47 +00:00
nari c98fd7da8a revert a file which is committed wrong.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34720 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-21 09:41:09 +00:00
nari 0d44bf83da * gc.c : remove gc_clear_mark_on_sweep_slots() and use
rest_sweep() instead of it, because some dead objects might be
  marked in next the mark phase by false pointers.
  [ruby-core:42672]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34719 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-21 09:35:07 +00:00
usa 07d8dffe82 * enc/depend: ignore mktable.c because it's not encoding library.
[ruby-core:42760] [Bug #6049]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34708 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-20 14:40:01 +00:00
nobu 310c2adf1b * enc/mktable.c (ENC_INFO): constify to suppress warnings.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-19 06:36:45 +00:00
naruse 0424e152c6 * Merge Onigmo-5.13.1. [ruby-dev:45057] [Feature #5820]
https://github.com/k-takata/Onigmo
  cp reg{comp,enc,error,exec,parse,syntax}.c reg{enc,int,parse}.h
  cp oniguruma.h
  cp tool/enc-unicode.rb
  cp -r enc/

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34663 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-17 07:42:23 +00:00
naruse ff4c13688e Add more description about Shift_JIS and CP50220.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34623 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-02-15 06:52:10 +00:00
naruse 800f04c6a5 * numeric.c (rb_enc_uint_char): raise RangeError when added codepoint
is invalid. [Feature #5855] [Bug #5863] [Bug #5864]

* string.c (rb_str_concat): ditto.

* string.c (rb_str_concat): set encoding as ASCII-8BIT when the string
  is US-ASCII and the argument is an integer greater than 127.

* regenc.c (onigenc_mb2_code_to_mbclen): rearrange error code.

* enc/euc_jp.c (code_to_mbclen): ditto.

* enc/shift_jis.c (code_to_mbclen): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34236 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2012-01-08 20:42:45 +00:00
naruse c1d369b0ab * enc/trans/iso-8859-16-tbl.rb: add ISO-8859-16 converter.
* enc/trans/single_byte.trans: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@33993 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-12-09 10:27:37 +00:00
naruse be276c140d * enc/unicode.c (PROPERTY_NAME_MAX_SIZE): +1.
reported by Ken Takata. [ruby-dev:44894][Bug #5652]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@33797 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-11-20 13:44:11 +00:00
nobu f6a635a919 * Makefile.in (enc/unicode/name2ctype.h): remove duplicated
ifdefs.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@33793 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-11-19 19:01:49 +00:00
yugui ddeccd20f1 Forward-ports r32780 from branches/ruby_1_9_3 to trunk.
--
* enc/Makefile.in (ECHO1): Same as the recent fix in common.mk.
  ":" in a make variable replacement cause a syntax error with
  /usr/ccs/bin/make on Solaris.  Uses $(NULLCMD) instead.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@32787 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-07-31 14:17:42 +00:00
nobu ef38cb7a05 * enc/depend: show srcdir.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@32112 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-06-16 00:10:10 +00:00
nobu 2acc71b2d5 * enc/trans/ibm737-tbl.rb: greek code page. fixes #4738
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31644 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-05-19 15:58:09 +00:00
nobu 3a47cf3395 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-05-15 11:55:52 +00:00
naruse 06911f90ce * enc/trans/emoji_iso2022_kddi.trans: ISO-2022-JP-KDDI doesn't have
CP932 UDA. Another reason is emacs-mule: the implementation of
  stateless-iso-2022-jp doesn't support beyond 94x94 (0x7fxx);
  but CP932 UDA is in 7Fxx-92xx.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31366 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-04-27 02:45:36 +00:00
naruse 78df33a0c1 * enc/utf_16le.c: surpress warning: shorten-64-to-32.
* ext/dbm/dbm.c: ditto.

* ext/gdbm/gdbm.c: ditto.

* parse.y (Init_ripper): surpress warning: unused value.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-03-21 21:23:42 +00:00
naruse 89c0d70e42 * enc/shift_jis.c: Change SJIS as an alias of Windows-31J.
* enc/shift_jis.c: Add PCK as an alias of Windows-31J.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@31073 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-03-09 02:54:40 +00:00
akr 113de0083e * enc/trans/utf8_mac.trans: parenthesize macro arguments.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-02-04 10:14:53 +00:00
naruse 10354c7b5c * enc/shift_jis.c (code_to_mbc): cast as int from the subtraction of pointers.
* enc/utf_16le.c (utf16le_mbc_enc_len): use ptrdiff_t.

* enc/utf_32be.c (utf32be_left_adjust_char_head): ditto.

* enc/utf_32le.c (utf32le_left_adjust_char_head): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30771 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-02-03 07:37:25 +00:00
akr 0f7fc54678 * enc/encdb.c: parenthesize macro arguments.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30755 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-02-01 11:47:01 +00:00
nobu b4be48e88d * enc/emacs_mule.c (emacsmule_islead): 7bit range is also leading
byte.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30731 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-01-30 06:13:25 +00:00
nobu 286615d430 * enc/emacs_mule.c (emacsmule_islead): fix inverse condition.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30727 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-01-30 03:53:44 +00:00
nobu 5528903d01 * enc/{emacs_mule,euc_jp}.c (code_to_mbc): suppress warnings.
* enc/iso_8859_{1,2}.c (apply_all_case_fold): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30468 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2011-01-05 20:16:32 +00:00
nobu 4f15664f6e * enc/depend (clean): remove name2ctype.h when out-of-place build.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30421 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-12-29 11:31:23 +00:00
nobu 0dfd81cb23 * win32/Makefile.sub (clean-enc): pass V to inferior make.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30420 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-12-29 11:27:36 +00:00
nobu 12329a58e1 * configure.in (target_archs): remove temporary objects.
* enc/Makefile.in, enc/depend (clean): remove work directories.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30286 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-12-21 03:45:21 +00:00
naruse b98ea1505c * enc/trans/big5-hkscs-tbl.rb: Update table as HKSCS-2008.
patched by oCameLo oTnTh [ruby-core:33256]

* enc/big5.c: add alias Big5-HKSCS:2008 to Big5-HKSCS.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29922 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-24 16:40:38 +00:00
naruse 38b482be8c * enc/trans/utf_16_32.trans: add the UTF-32 converter.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29895 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-24 00:08:04 +00:00
naruse 7f38397b6c * enc/trans/utf_16_32.trans: add a convert from UTF-8 to UTF-16.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29892 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-23 20:49:56 +00:00
naruse 3ab82a65d7 * enc/trans/utf_16_32.trans: raise error on unpaired upper
surrogates.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29891 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-23 18:23:03 +00:00
naruse 78bee9c26a * enc/utf_16_32.h: add UTF-16 and UTF-32 as a dummy encoding.
* enc/trans/utf_16_32.trans: add a converter from UTF-16 to UTF-8.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29889 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-23 16:42:47 +00:00
naruse 5d8a64b1af Add missing tables.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29871 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-22 11:35:55 +00:00
naruse 60dfa6b655 * enc/big5.c: split CP950 from Big5.
* enc/big5.c: split CP951 from Big5-HKSCS.

* enc/trans/big5.trans: import conversion table of Big5, Big5-HKSCS,
  CP950, and CP951 from ICU. they need fallback conversions.
  ref [ruby-core:33256]
  http://source.icu-project.org/repos/icu/data/trunk/charset/data/ucm/

* tool/transcode-tblgen.rb (import_ucm): add to import ucm files.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29869 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-22 09:35:08 +00:00
nobu 023eab5201 * win32/Makefile.sub (MAKEDIRS): should not include silent flag.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29818 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-17 09:38:38 +00:00
naruse 07b1f01b33 * enc/shift_jis.c (property_name_to_ctype): fix memory leak.
* enc/euc_jp.c (property_name_to_ctype): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29810 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-17 00:49:49 +00:00
nobu 54e1affd16 * enc/Makefile.in (distclean): should not remove sources which are
distributed in tarball.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29783 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-14 08:57:53 +00:00
nobu fd7c0279f8 * Makefile.in, common.mk, cygwin/GNUmakefile.in, enc/depend,
ext/ripper/depend, lib/mkmf.rb, win32/Makefile.sub: caddle up.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29769 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-12 11:51:42 +00:00
nobu f2f7a7740a * common.mk: hide long command lines by default. verbose-mode is
turned on by V=1 as before.
  http://jarp.does.notwork.org/diary/200605b.html#200605121

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29748 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-11 12:46:23 +00:00
naruse dbf7e6f9f2 * regenc.c (onigenc_minimum_property_name_to_ctype):
\p{...} should be case insensitive. [ruby-core:33000]

* regenc.c (onigenc_property_list_add_property):
  ditto.

* enc/euc_jp.c (init_property_list, property_name_to_ctype):
  to lowercase property names.

* enc/shift_jis.c (init_property_list, property_name_to_ctype):
  ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29732 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-09 07:36:33 +00:00
naruse a0265b0662 * tool/enc-unicode.rb,
enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  Add Age property to regexp. [ruby-core:33019]
  patched by Ammar Ali, tested by Run Paint Run Run

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-08 05:32:45 +00:00
naruse 7b5e9245ac * enc/trans/gbk-tbl.rb: Add euro sign. [ruby-core:33094]
CP936, which is de facto definition of GBK, has it.
  http://msdn.microsoft.com/en-us/goglobal/cc305153.aspx

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29713 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-11-08 00:50:13 +00:00
naruse f85b841a01 * tool/enc-unicode.rb,
enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  Add 'Unknown' Script.
  patched by Run Paint Run Run. [ruby-core:32937] #3998

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29626 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-10-29 01:03:21 +00:00
naruse fc9176ac0e * tool/enc-unicode.rb,
enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  Update Oniguruma for Unicode 6.
  patched by Run Paint Run Run. [ruby-core:32923] #3989

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29620 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-10-28 11:14:05 +00:00
naruse 81e40399fb add references.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29530 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-10-18 06:42:01 +00:00
nobu b238a3f3fd * tool/enc-unicode.rb: get rid of lots of warnings.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29489 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-10-13 14:16:49 +00:00
naruse e1d5d4e7f2 * enc/unicode.c (onigenc_unicode_property_name_to_ctype):
remove useless assignment.

* vm.c (vm_make_proc_from_block): ditto.

* variable.c (rb_ivar_count): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29405 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-10-03 22:57:23 +00:00
yugui 60b41810f8 Forward-ports r28360 from ruby_1_9_2 into trunk.
--
* enc/Makefile.in (realclean): has been missing. necessary
  for make realclean-enc.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@29177 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-09-03 15:57:33 +00:00
naruse 775188a0a2 * enc/depend: add space at the begginig of @ignore_error.
* lib/mkmf.rb: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28883 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-08-06 08:44:33 +00:00
naruse 2b071787d2 * Makefile.in: ignore error from RMDIRS.
* enc/depend: ditto.

* lib/mkmf.rb: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28881 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-08-06 08:40:27 +00:00
nobu 6abe539cfc * configure.in (RMDIR): use --ignore-fail-on-non-empty if possible.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28708 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-07-21 21:26:56 +00:00
naruse 78f5b54f1b * enc/trans/utf8_mac.trans (buf_apply): fix for patterns
whose result is 2 bytes. [ruby-core:30751]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@28307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-06-12 17:13:54 +00:00
naruse f8d97b0026 * enc/iso_2022_jp.h: add CP50220.
* enc/trans/iso2022.trans: add converter for CP50220.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27860 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-05-17 06:28:16 +00:00
naruse 90be970018 * enc/utf_8.c: Add new alias UTF-8-HFS for UTF8-MAC.
http://www.gnu.org/software/emacs/NEWS.23.2

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27736 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-05-11 06:15:53 +00:00
nobu a0136f4f27 * properties.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27658 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-05-07 13:29:44 +00:00
naruse afd64aafd1 * enc/trans/iso2022.trans: CP50221 supports 8bit JIS.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27149 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-04-01 08:18:38 +00:00
nobu 7e3e79d083 * enc/utf_16{be,le}.c (utf16{be,le}_mbc_to_code): simplified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27143 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-04-01 05:30:25 +00:00
muraken e4d8dc5c46 * bignum.c, node.h, strftime.c, enc/trans/utf8_mac.trans: added explicit casts for supplessing warnings.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27040 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-25 03:08:28 +00:00
akr 49d993729f * tool/transcode-tblgen.rb (transcode_compile_tree): make
valid_encoding mandatory unless from_encoding is registered in
  ValidEncoding.
  (transcode_tbl_only): ditto.
  (transcode_tblgen): ditto.
  (ValidEncoding): new function.

* enc/trans/escape.trans: specify valid_encoding.

* enc/trans/emoji_sjis_docomo.trans: ditto.

* enc/trans/emoji.trans: ditto.

* enc/trans/emoji_iso2022_kddi.trans: ditto.

* enc/trans/big5.trans: ditto.

* enc/trans/emoji_sjis_softbank.trans: ditto.

* enc/trans/emoji_sjis_kddi.trans: ditto.

* enc/trans/chinese.trans: use ValidEncoding() instead of
  ValidEncoding[].


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26995 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-21 03:38:58 +00:00
muraken 04d90693dc * enc/trans/emoji.trans: added codepoints leading 0xf4 into nomap_table.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-16 11:18:03 +00:00
akr a73374bb57 * tool/transcode-tblgen.rb (transcode_tblgen): add valid_encoding
optional argument.

* enc/trans/single_byte.trans use valid_encoding argument for
  transcode_tblgen.

* enc/trans/chinese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-15 12:25:20 +00:00
akr ff39d22c33 * enc/trans/emoji.trans: fix nomap_table.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-14 06:49:22 +00:00
akr fa37ab769f * tool/transcode-tblgen.rb: reject ambiguous mapping.
* enc/trans/single_byte.trans: remove ambiguous maping such as
  \xD6 -> U+05F2 and \xD6\xC7 -> U+FB1F in Windows-1255


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-13 17:54:43 +00:00
muraken 9eb49ff8d7 * enc/x_emoji.h: renamed from enc/x-emoji.c.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26863 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-10 03:12:17 +00:00
muraken 62f8df2d3c * enc/trans/EMOJI/*.src, enc/trans/emoji*, enc/x-emoji.c, test/ruby/enc/test_emoji.rb, tool/enc-emoji-citrus-gen.rb, tool/enc-emoji4unicode.rb, tool/jisx0208.rb, tool/test/test_jisx0208.rb: new encodings to support emoji charsets, which are used by Japanese mobile phones [ruby-dev:40528]. Thanks Yoji Shidara for a lot of contribution.
* tool/transcode-tblgen.rb: modified for enc-emoji4unicode.rb.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26856 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-09 09:15:42 +00:00
matz db37773e13 * include/ruby/oniguruma.h: updated to follow Oniguruma 5.9.2.
* re.c (make_regexp): use onig_new() instead of onig_alloc_init().

* re.c (rb_reg_to_s): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26791 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-03-01 21:54:59 +00:00
naruse 6899b6ff80 * enc/trans/utf8_mac.trans (buf_shift_char): don't see uninitialised
value. [ruby-dev:40233]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26464 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2010-01-29 00:56:10 +00:00
duerst b32ee85f97 * transcode_data.h, transcode.c, tool/transcode-tblgen.rb: Added
support for new transcoding instruction FUNsio (with Tatsuya Mizuno)

* enc/trans/gb18030.trans: Significantly reduced GB18030 conversion
  table footprint using FUNsio and differences (with Tatsuya Mizuno)

* test/ruby/test_transcode.rb: Minor name fix (from Tatsuya Mizuno)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@26065 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-12-10 11:59:12 +00:00
duerst 9998481d4e * enc/trans/gb18030-tbl.rb: Fix omission of C1 region in code table
(from Tatsuya Mizuno)

* test/ruby/test_transcode.rb: Added test for converting full range of
  Unicode codepoints from/to GB18030 (from Tatsuya Mizuno)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25980 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-12-03 11:29:33 +00:00
akr cc128e3ecf * enc/trans/newline.trans (fun_so_universal_newline): generate \n
after \r\n detection instead of just after \r.
  [ruby-list:45988] [ruby-core:25881] [ruby-core:26788]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25883 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-11-22 19:15:55 +00:00
duerst e0436c54c2 * enc/big5.c, enc/trans/big5.trans, enc/trans/big5-uao-tbl.rb,
test/ruby/test-transcode.rb: Added Encoding 'Big5-UAO' and transcoding
  for it (from Tatsuya Mizuno) (see Bug #1784)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25822 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-11-17 08:56:11 +00:00
naruse d5537936ab * tool/enc-unicode.rb,
enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  use UTS#18 for POSIX character class.
  http://rubyspec.org/issues/show/161

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25338 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-14 16:51:52 +00:00
naruse 181eb7d5c1 Add derived core and binary property and aliases.
* tool/enc-unicode.rb,
  enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  Add DerivedCoreProperties, PropList (Binary Property),
  PropertyAlias and PropertyValueAlias.
  Now users of tool/enc-unicode.rb should specify
  the directory of UCD files.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25324 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-13 12:27:00 +00:00
nobu 7081875aa8 * enc/unicode/name2ctype.h: update.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25275 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-10 03:20:49 +00:00
naruse 5a4ce608e2 * tool/enc-unicode.rb: optimized.
* enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  U+100000-U+10FFFD is assigned, not Cn.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25271 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-08 18:07:08 +00:00
naruse 866c79e2de * tool/enc-unicode.rb: parse range notation of UnicodeData.txt.
* enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  follow above change. [ruby-dev:39444]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25260 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-08 02:49:11 +00:00
naruse 8d4ebdc8fe * enc/unicode/name2ctype.h: Updated to Unicode 5.2.0.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25195 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-02 16:03:20 +00:00
naruse 48eafcbc49 Updated to Unicode 5.2.0.
* enc/unicode/name2ctype.h.blt, enc/unicode/name2ctype.kwd,
  enc/unicode/name2ctype.src: Updated to Unicode 5.2.0.
  NOTE: when you update these data, download UnicodeData.txt
  and Scripts.txt from http://www.unicode.org/Public/UNIDATA/
  and run
  ruby1.9 tool/enc-unicode.rb UnicodeData.txt Scripts.txt \
  > enc/unicode/name2ctype.kwd

* enc/unicode/Scripts.txt: removed.

* enc/unicode/UnicodeData.txt: removed.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@25190 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-10-02 13:37:41 +00:00
naruse ee4b59a419 * unicode.c (onigenc_unicode_property_name_to_ctype):
ignore case of properties.

* tool/enc-unicode.rb: downcase properties list.

* enc/unicode/name2ctype.h, enc/unicode/name2ctype.h.blt,
  enc/unicode/name2ctype.kwd, enc/unicode/name2ctype.src:
  follow above.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24836 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-09-10 22:54:01 +00:00
nobu 31b7ae00c0 * include/ruby/st.h (st_hash_func): use st_index_t.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24792 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-09-08 13:10:04 +00:00
naruse 6e6a28183a * unicode.c (PROPERTY_NAME_MAX_SIZE): use MAX_WORD_LENGTH.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24677 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-26 17:01:10 +00:00
nobu 1af43ae867 * enc/unicode.c (onigenc_unicode_mbc_case_fold): balanced braces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24658 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-26 00:48:49 +00:00
nobu 1fd7f2e57d * enc/unicode/name2ctype.h: updated.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24657 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-25 21:54:03 +00:00
naruse f1eff95745 Update Oniguruma's UnicodeData to 5.1.
* tool/enc-unicode.rb: added for generate name2ctype.kwd.
  contributed by Run Paint Run Run [ruby-core:24775]
  use like following:
    ruby19 tool/enc-unicode.rb enc/unicode/UnicodeData.txt \
      enc/unicode/Scripts.txt > enc/unicode/name2ctype.kwd

* enc/unicode.c (CodeRanges): move definitions to name2ctype.h.

* enc/unicode/name2ctype.h.blt, enc/unicode/name2ctype.kwd,
  enc/unicode/name2ctype.src: updated to v5.1.

* enc/unicode/UnicodeData.txt, enc/unicode/Scripts.txt: added v5.1.

* Makefile.in: add rule to generate name2ctype.kwd from
  UnicodeData.txt and Scripts.txt.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24651 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-25 16:15:38 +00:00
nobu a7b920686a * enc/unicode/name2ctype.h: split from enc/unicode.c and made a
perfect hash.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24613 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-21 08:01:09 +00:00
nobu a606038c6a * enc/utf_8.c (code_to_mbc): suppressed a warning.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24607 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-21 06:37:36 +00:00
nobu e1c9ac6bd9 * enc/unicode.c (CodeRanges): initialized statically.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24582 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-19 02:32:49 +00:00
naruse 2b91cbbf11 * enc/Makefile.in (MKDIRS): revert r24525.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24538 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-14 08:20:13 +00:00
nobu 24c783e95e * configure.in, Makefile.in (MAKEDIRS): used MKDIR_P instead of
as_mkdir_p.  [ruby-dev:39063]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24525 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-13 07:20:16 +00:00
naruse 38107457a3 * enc/encdb.c (ENC_SET_BASE): fix typo. patch by ujihisa [ruby-dev:39004]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24386 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-04 03:42:11 +00:00
naruse 8c658137d9 More strict for Big5 series.
* enc/big5.c (EncLen_Big5): back to original Big5 table.
  (EncLen_Big5_HKSCS): for Big5-HKSCS.
  (trans): add the lead byte table for Big5-HKSCS.
  (big5_mbc_enc_len): abstract function for Big5 series.
  (big5_mbc_enc_len): for Big5.
  (big5_hkscs_mbc_enc_len): for Big5-HKSCS.
  (BIG5_HKSCS_P): added.
  (BIG5_ISMB_FIRST): add routine for Big5-HKSCS.
  (big5_hkscs): add for Big5-HKSCS.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24384 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-04 00:51:22 +00:00
naruse b3d7273dc1 Add functions and macros for second encoding definitions.
* encoding.c (rb_enc_set_base): Add for setting base encoding
  with their names. this is internal function.

* template/encdb.h.tmpl: specify ENC_SET_BASE for second
  encodings in each encoding files.

* enc/encdb.c (rb_enc_set_base): add a declaration.
  (ENC_SET_BASE): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24383 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-08-04 00:50:59 +00:00
nobu 4fd615943e * enc/big5.c: not executable.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-25 04:42:48 +00:00
naruse a8951a5b3a * enc/big5.c: Fix EncLen_BIG5 for Big5-HKSCS. see [ruby-core:24390]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-24 16:15:53 +00:00
duerst 2886207584 * enc/trans/big5.trans, big5-hkscs-tbl.rb:
new Chinese BIG5-HKSCS transcoding (with Tatsuya Mizuno)

* test/ruby/test_transcode.rb: added tests for the above
  (with Tatsuya Mizuno)

* enc/big5.c: Added BIG5-HKSCS as a replicate encoding of BIG5
  (short term solution, needs more work; with Tatsuya Mizuno)

* tool/transcode-tblgen.rb: made 'pat' directly accessible in
  class StrSet


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@24264 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-07-24 10:26:18 +00:00
nobu c030cf1975 * ruby.c (process_options), enc/prelude.rb: encdb and transdb are
extension libraries.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23813 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-06-22 05:41:51 +00:00
naruse d9cf0f822f * enc/trans/utf8_mac.trans: remove wrong optimization.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23686 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-06-13 18:55:55 +00:00
naruse 3abca796f4 Fix: DON'T move in_p because before in_p is replaced by buffered data.
* transcode.c: NOMAP is now multibyte direct map.

* transcode.c: remove ASIS.

* transcode_data.h: ditto.

* tool/transcode-tb (ActionMap#generate_info): remove :asis.

* tool/transcode-tb (ActionMap#generate_info): add :nomap0.

* enc/trans/utf8_mac.trans: replace :asis by :nomap0.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23344 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-05-05 00:05:11 +00:00
naruse f207f9fd51 * enc/trans/utf8_mac-tbl.rb: don't use Unicode escape.
* enc/trans/utf8_mac.trans: follow above.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23325 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-05-02 01:38:27 +00:00
nobu 8543ecee53 * enc/trans/utf8_mac.trans: get rid of a 1.9 feature for cross
compile.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-04-30 06:27:51 +00:00
naruse 80705b9fbf Add new transcoder: CP51932 <-> CP50221.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-04-29 12:50:57 +00:00
naruse d0a4f8ada9 * enc/trans/utf8_mac.trans: Add converter for UTF8-MAC.
* enc/trans/utf8_mac-tbl.rb: ditto.

* test/ruby/test_econv.rb: tests for above.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23296 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-04-26 14:21:43 +00:00
nobu 15265f8be6 * enc/depend (link_so): replaces $(TARGET) with basename of the
target.  [ruby-talk:330286]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@23035 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-03-22 21:51:18 +00:00
usa 39bc33d9a7 * enc/depend: extract comile rules to each target for VC++.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21892 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-30 05:13:22 +00:00
nobu c938de20cd * common.mk (distclean-enc, realclean-enc): do not call clean of
enc.mk twice or more.

* enc/depend (cleanobjs): added deffile.

* lib/mkmf.rb (create_makefile): removes deffile at clean instead
  of distclean.

* win32/Makefile.sub (miniruby, LIBRUBY_SO): removes lib and exp
  files.

* win32/Makefile.sub (clean, distclean): have moved to common.mk.

* win32/rmdirs.bat: omits `not empty' message.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21790 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-27 02:03:54 +00:00
nobu e24346d6c6 * enc/trans/gb18030.trans: get rid of a 1.9 feature for cross
compile.  [ruby-core:21345]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21512 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-14 15:06:19 +00:00
duerst 82c673d3a1 * enc/trans/gb18030.trans, gb18030-tbl.rb:
new Chinese GB18030 transcoding (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi)

* transcode_data.h, transcode.c, tool/transcode_tblgen.rb:
  added support for GB18030-specific 4-byte sequences
  (with Yoshihiro Kambayashi)



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-14 11:12:30 +00:00
nobu e668e36b49 * template/{encdb,transdb}.h.tmpl: moved enc/make_encdb.rb and
enc/trans/make_transdb.rb using tool/generic_erb.rb.

* common.mk (encdb.h, transdb.h): generates from avobe template.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21490 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-13 09:05:29 +00:00
nobu 4cb8d3316a * enc/trans/make_transdb.rb (converters): should not depend on the
hash order for cross compile.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21489 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-13 08:28:14 +00:00
duerst deeade6f3e * enc/trans/gbk.trans, gbk-tbl.rb:
new Chinese GBK transcoding (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21315 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04 09:12:14 +00:00
duerst fecce9e5e5 * test/ruby/test_transcode.rb: added tests for GB2312
(from Yoshihiro Kambayashi)

* enc/trans/chinese.trans: set valid byte patterns for
  GB2312 and GB12345


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21314 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04 08:55:04 +00:00
duerst 3bc30f0b73 * enc/trans/big5.trans, big5-tbl.rb:
new Chinese Big5 transcoding (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21313 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-04 08:40:26 +00:00
naruse 1240916075 change encoding name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21285 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-03 14:12:39 +00:00
naruse 2920aaa2d1 * enc/trans/chinese.trans: added for transcoding EUC-CN and GB12345.
* enc/trans/GB/: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@21283 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2009-01-03 06:31:05 +00:00
duerst a28fdecda7 * enc/trans/single_byte.trans, cp850-tbl.rb, cp852-tbl.rb,
cp855-tbl.rb, koi8-r-tbl.rb, koi8-u-tbl.rb, tis-620-tbl.rb:
  new single-byte transcodings (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi), small cosmetic fixes


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20599 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-12-09 09:39:25 +00:00
nobu a6d8d84a9e * enc/depend (clean-srcs): split out from clean.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20582 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-12-08 15:17:52 +00:00
nobu 8e6ad88737 * enc/depend (LIBS): fixed for disable-shared. [ruby-dev:37103]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20241 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-11-17 09:05:19 +00:00
duerst 831e804388 * enc/trans/single_byte.trans, macgreek-tbl.rb, macroman-tbl.rb,
macromania-tbl.rb, macturkish-tbl.rb, macukraine-tbl.rb,
  ibm437-tbl.rb, ibm852-tbl.rb, ibm855-tbl.rb, ibm857-tbl.rb,
  ibm860-tbl.rb, ibm861-tbl.rb, ibm862-tbl.rb, ibm863-tbl.rb,
  ibm865-tbl.rb, ibm866-tbl.rb, ibm869-tbl.rb, ibm775-tbl.rb:
  new single-byte transcodings (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20178 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-11-11 05:26:20 +00:00
duerst d37df9fb13 * enc/trans/single_byte.trans, maccroatioan-tbl.rb,
maccyrillic-tbl.rb, maciceland-tbl.rb: new single-byte
  transcodings (from Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added tests for the above
  (from Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20075 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-31 09:07:21 +00:00
duerst 6fd14ccae5 * enc/trans/single_byte.trans: refactoring to make it easier
to add more transcodings (with Yoshihiro Kambayashi)

* enc/trans/iso-8859-1-tbl.rb: new file to avoid having to
  treat ISO-8859-1 as special


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@20054 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-30 05:47:01 +00:00
nobu da6300e8f8 * enc/us_ascii.c (us_ascii_mbc_enc_len): made static. a patch by
Tadashi Saito <shiba AT mail2.accsnet.ne.jp> at [ruby-dev:36916]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19929 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-24 19:00:35 +00:00
duerst b014f1bc02 * enc/trans/single_byte.trans: adding WINDOWS-wwww encodings
(wwww = 874/1250/1251/1253/1254/1255/1256/1257)
  (contributed by Yoshihiro Kambayashi)

* enc/trans/windows-wwww-tbl.rb: 8 new files
  (contributed by Yoshihiro Kambayashi)

* test/ruby/test_transcode.rb: added test_windows_wwww
  (contributed by Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19846 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-19 09:15:37 +00:00
duerst 7adbfbb793 * tool/transcode-tblgen.rb: added set_valid_byte_pattern
to reduce coupling between table generation script and
  specific encodings.

* enc/trans/single_byte.trans: using set_valid_byte_pattern


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-18 08:10:57 +00:00
nobu 7485e91f76 * common.mk, enc/depend (enc, trans): targets for sources.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-16 05:34:25 +00:00
akr b968fa97f6 * enc/trans/single_byte.trans (transcode_tblgen_singlebyte): renamed
from transcode_tblgen_windows.
  (transcode_tblgen_iso8859): use transcode_tblgen_singlebyte.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-14 11:33:17 +00:00
duerst 48a303c027 * enc/trans/single_byte.trans: added windows-1252
* enc/trans/windows-1252-tbl.rb: new file
  (contributed by Yoshihiro Kambayashi)

* tool/transcode-tblgen.rb: listed windows-1252 as '1byte'

* test/ruby/test_transcode.rb: added test_windows_1252
  (contributed by Yoshihiro Kambayashi)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19778 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-10-14 04:37:10 +00:00
akr 081c802cb9 * grapheme cluster implementation reverted. [ruby-dev:36375]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19417 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-18 12:53:25 +00:00
akr b3d772643e fix typos.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-16 16:59:08 +00:00
akr a67d4fa01c * include/ruby/oniguruma.h (OnigEncodingTypeST): add precise_ret
argument for mbc_to_code.
  (ONIGENC_MBC_TO_CODE): provide NULL for precise_ret.
  (ONIGENC_MBC_PRECISE_CODEPOINT): defined.

* include/ruby/encoding.h (rb_enc_mbc_precise_codepoint): defined.

* regenc.h (onigenc_single_byte_mbc_to_code): precise_ret argument
  added.
  (onigenc_mbn_mbc_to_code): ditto.

* regenc.c (onigenc_single_byte_mbc_to_code): precise_ret argument
  added.
  (onigenc_mbn_mbc_to_code): ditto.

* string.c (count_utf8_lead_bytes_with_word): removed.
  (str_utf8_nth): removed.
  (str_utf8_offset): removed.
  (str_strlen): UTF-8 codepoint oriented optimization removed.
  (rb_str_substr): ditto.
  (enc_succ_char): use rb_enc_mbc_precise_codepoint.
  (enc_pred_char): ditto.
  (rb_str_succ): ditto.

* encoding.c (rb_enc_ascget): check length with
  rb_enc_mbc_precise_codepoint.
  (rb_enc_codepoint): use rb_enc_mbc_precise_codepoint.

* regexec.c (string_cmp_ic): add text_end argument.
  (match_at): check end of character after exact string matches.

* enc/utf_8.c (graphme_table): defined for extended graphme cluster
  boundary.
  (grapheme_cmp): defined.
  (get_grapheme_properties): defined.
  (grapheme_boundary_p): defined.
  (MAX_BYTES_LENGTH): defined.
  (comb_char_enc_len): defined.
  (mbc_to_code0): extracted from mbc_to_code.
  (mbc_to_code): use mbc_to_code0.
  (left_adjust_combchar_head): defined.
  (utf_8): use a extended graphme cluster as a unit.

* enc/unicode.c (onigenc_unicode_mbc_case_fold): use
  ONIGENC_MBC_PRECISE_CODEPOINT to extract codepoints.
  (onigenc_unicode_get_case_fold_codes_by_str): ditto.

* enc/euc_jp.c (mbc_to_code): follow mbc_to_code field change.
  use onigenc_mbn_mbc_to_code.

* enc/shift_jis.c (mbc_to_code): ditto.

* enc/emacs_mule.c (mbc_to_code): ditto.

* enc/gbk.c (gbk_mbc_to_code): follow mbc_to_code field and
  onigenc_mbn_mbc_to_code change.

* enc/cp949.c (cp949_mbc_to_code): ditto.

* enc/big5.c (big5_mbc_to_code): ditto.

* enc/euc_tw.c (euctw_mbc_to_code): ditto.

* enc/euc_kr.c (euckr_mbc_to_code): ditto.

* enc/gb18030.c (gb18030_mbc_to_code): ditto.

* enc/utf_32be.c (utf32be_mbc_to_code): follow mbc_to_code field
  change.

* enc/utf_16be.c (utf16be_mbc_to_code): ditto.

* enc/utf_32le.c (utf32le_mbc_to_code): ditto.

* enc/utf_16le.c (utf16le_mbc_to_code): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19389 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-16 16:48:05 +00:00
akr 0675246ba6 * transcode_data.h (rb_transcoder): resetsize_func and resetstate_func
also returns ssize_t.

* enc/trans/iso2022.trans: follow the type change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19354 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-15 02:11:50 +00:00
akr c767be3039 * transcode_data.h: return output functions ssize_t.
* transcode.c (transcode_restartable0): don't need to cast the result
  of output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19351 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:35:17 +00:00
akr a3c8c0adec * transcode_data.h: output function takes output buffer size.
* transcode.c: give output buffer size for output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:06:20 +00:00
akr 19416601a0 * include/ruby/oniguruma.h (OnigEncodingTypeST): add end argument for
left_adjust_char_head.
  (ONIGENC_LEFT_ADJUST_CHAR_HEAD): add end argument.
  (onigenc_get_left_adjust_char_head): ditto.

* include/ruby/encoding.h (rb_enc_left_char_head): add end argument.

* regenc.h (onigenc_single_byte_left_adjust_char_head): ditto.

* regenc.c (onigenc_get_right_adjust_char_head): follow the interface
  change.
  (onigenc_get_right_adjust_char_head_with_prev): ditto.
  (onigenc_get_prev_char_head): ditto.
  (onigenc_step_back): ditto.
  (onigenc_get_left_adjust_char_head): ditto.
  (onigenc_single_byte_code_to_mbc): ditto.

* re.c: ditto.

* string.c: ditto.

* io.c: ditto.

* regexec.c: ditto.

* enc/euc_jp.c: ditto.

* enc/cp949.c: ditto.

* enc/shift_jis.c: ditto.

* enc/gbk.c: ditto.

* enc/big5.c: ditto.

* enc/euc_tw.c: ditto.

* enc/euc_kr.c: ditto.

* enc/emacs_mule.c: ditto.

* enc/gb18030.c: ditto.

* enc/utf_8.c: ditto.

* enc/utf_16le.c: ditto.

* enc/utf_16be.c: ditto.

* enc/utf_32le.c: ditto.

* enc/utf_32be.c: ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19334 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-13 19:23:52 +00:00
akr 41d3a01486 * enc/trans/escape.trans: transcoder name renamed to use underscore.
* transcode.c: follow the renaming.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 17:30:07 +00:00
naruse d51b061565 * include/ruby/oniguruma.h (OnigCodePoint): unsigned long to unsigned int.
* include/ruby/encoding.h (rb_enc_codepoint): ditto.

* encoding.c (rb_enc_codepoint): signed int to unsigned int.

* encoding.c (rb_enc_ascget): ditto.

* string.c (rb_str_casecmp): ditto.

* string.c (enc_succ_alnum_char): ditto.

* string.c (rb_str_inspect): ditto.

* string.c (rb_str_upcase_bang): ditto.

* string.c (rb_str_downcase_bang): ditto.

* string.c (rb_str_capitalize_bang): ditto.

* string.c (rb_str_swapcase_bang): ditto.

* string.c (struct tr): ditto.

* string.c (trnext): ditto.

* string.c (tr_trans): ditto.

* string.c (tr_setup_table): ditto.

* string.c (tr_find): ditto.

* string.c (rb_str_delete_bang): ditto.

* string.c (rb_str_squeeze_bang): ditto.

* string.c (rb_str_count): ditto.

* string.c (rb_str_split_m): ditto.

* string.c (rb_str_each_line): ditto.

* string.c (rb_str_lstrip_bang): ditto.

* string.c (rb_str_rstrip_bang): ditto.

* string.c (rb_str_intern): ditto.

* dir.c (char_casecmp): ditto.

* sprintf.c (rb_str_format): ditto.

* enc/emacs_mule.c (mbc_to_code): to be 32bit clean.

* enc/emacs_mule.c (code_to_mbc): ditto.

* enc/gb18030.c (mbc_to_code): ditto.

* enc/gb18030.c (code_to_mbc): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-11 10:34:59 +00:00