Граф коммитов

360 Коммитов

Автор SHA1 Сообщение Дата
akr c3f55e6d48 * transcode.c (econv_last_error): new method.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18980 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31 05:27:52 +00:00
akr 9d2accff2b * transcode.c (econv_primitive_convert): make two arguments,
destination_byteoffset and destination_bytesize, optional.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18979 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31 05:18:29 +00:00
akr f3c43ae0c2 * transcode.c (make_econv_exception): error message simplified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31 05:12:32 +00:00
akr d9c5f88abc * transcode.c (econv_putback): make max argument optional.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18967 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30 19:39:16 +00:00
akr 97a5808a14 * transcode.c (rb_econv_putback): putback from last byte.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30 19:36:38 +00:00
akr 998a4ec5af * transcode.c (econv_putback): renamed from econv_primitive_putback.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18965 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30 19:27:35 +00:00
akr 5bbc51ed43 * transcode.c (econv_insert_output): renamed from
econv_primitive_insert_output.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18964 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30 19:23:23 +00:00
akr 5e222d6945 * transcode.c (econv_primitive_errinfo): 5th element (partial_input)
removed from result.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18962 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-30 18:45:48 +00:00
akr c2ba5cf9c5 rdoc update.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18929 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-29 16:25:27 +00:00
akr d4afbe49f3 rdoc update.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18914 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 18:28:52 +00:00
akr 2c80188627 * transcode.c (econv_finish): new method.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18913 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 18:00:02 +00:00
akr 23d10cbb00 * transcode.c (econv_convert): new method.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 17:46:18 +00:00
akr ad8cfb1b94 * transcode.c (econv_primitive_convert): accept nil as
destination_bytesize for unlimited destination size.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18911 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 17:39:02 +00:00
akr 3811bb53bd * transcode.c (econv_primitive_convert): accept nil as input for empty
input.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18910 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 17:13:49 +00:00
akr c23ec90e81 * transcode.c (sym_invalid_byte_sequence): new variable.
(sym_undefined_conversion): ditto.
  (sym_destination_buffer_full): ditto.
  (sym_source_buffer_empty): ditto.
  (sym_finished): ditto.
  (sym_output_followed_by_input): ditto.
  (sym_incomplete_input): ditto.
  (econv_result_to_symbol): use above variables.
  (Init_transcode): initialize above variables.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18908 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-28 16:59:17 +00:00
akr 1366802a1f * transcode.c (transcode_loop): simplified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18876 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26 16:14:49 +00:00
akr db6ec3105e * include/ruby/encoding.h (rb_econv_result_t): enumeration constant:
econv_incomplete_input. 

* io.c (finish_writeconv): check econv_incomplete_input.

* transcode.c (transcode_restartable0): return econv_incomplete_input
  for unexpected end of source buffer.
  (trans_sweep): check econv_incomplete_input.
  (rb_trans_conv): ditto.
  (rb_econv_convert0): ditto.
  (rb_econv_convert): ditto.
  (transcode_loop): ditto.
  (make_econv_exception): change message for econv_incomplete_input.
  (econv_result_to_symbol): return :incomplete_input for
  econv_incomplete_input.
  (ecerr_incomplete_input): new method.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18875 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26 16:09:29 +00:00
akr a68e409eaa * include/ruby/encoding.h (rb_econv_elem_t): move to transcode.c
(rb_econv_t): defined as an incomplete type.

* transcode.c (rb_econv_elem_t): moved from encoding.h.
  (rb_econv_t): complete type defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18872 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26 15:01:11 +00:00
akr 19ab08653e * transcode.c (rb_econv_open): disable newline conversion for ASCII
incompatible encodings.
  (str_transcode0): don't need disable newline conversion here.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18870 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26 12:55:14 +00:00
akr 18b097b73b * transcode.c (rb_econv_binmode): binmode is effective only once.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18869 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-26 12:45:20 +00:00
akr 873987f2e0 * transcode.c (str_transcode0): disable newline conversion for ASCII
incompatible encoding.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18853 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 16:27:23 +00:00
akr 1a805010ed * transcode.c (str_transcode0): don't short cut for newline conversion.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18852 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 15:58:35 +00:00
akr 86154f63b3 * transcode.c (make_econv_exception): show readagain part for invalid
byte sequence exception.  store the part as an instance variable.
  (ecerr_readagain_bytes): new method to access the readagain part.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18850 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 15:26:54 +00:00
akr 0486026389 * transcode.c (rb_econv_open): make last_tc NULL if there are only
additional transcoders.
  (econv_description): extracted from rb_econv_open_exc.
  (rb_econv_open_exc): use econv_description.
  (econv_inspect): use econv_description.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 15:01:36 +00:00
akr aeffc67dbb * transcode.c (rb_econv_open_by_transcoder_entries): initialize
last_error.  num_trans may be zero.
  (rb_econv_convert0): num_trans may be zero.
  (rb_econv_putbackable): ditto.
  (rb_econv_putback): ditto.
  (rb_econv_convert): input_ptr and output_ptr may be NULL.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18835 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 13:04:16 +00:00
nobu 10906176b4 * transcode.c (rb_econv_substr_append): fix for buffer overrun.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 08:35:42 +00:00
nobu 8c6040edca * transcode.c (rb_econv_open_exc): constified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18828 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-25 01:42:36 +00:00
akr 2c78899641 * configure.in (BUILTIN_TRANSSRCS): defined.
(BUILTIN_TRANSOBJS): defined.

* enc/Makefile.in (BUILTIN_TRANSES): defined.

* enc/make_encmake.rb (BUILTIN_TRANSES): defined.

* enc/depend: don't generate rules for builtin transcoders.

* common.mk (COMMONOBJS): add BUILTIN_TRANSOBJS.
  (enc.mk): pass BUILTIN_TRANSOBJS.
  (newline.c): new rule.
  (newline.$(OBJEXT)): new ru.e
  (srcs): newline.c added.

* Makefile.in (BUILTIN_TRANSSRCS): defined.
  (BUILTIN_TRANSOBJS): defined.

* transcode.c (Init_transcode): call Init_newline.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18826 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 15:58:43 +00:00
akr bf5e559ce0 * transcode.c (declare_transcoder): arguments order changed.
(rb_declare_transcoder): call declare_transcoder just once.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18824 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 15:13:37 +00:00
akr ae1b02a8da * include/ruby/encoding.h (rb_econv_t): use rb_econv_option_t.
* transcode.c: follow the rb_econv_t change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18823 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 10:49:36 +00:00
akr 071319fde1 * io.c (rb_io_s_pipe): accept optional hash.
(rb_io_set_encoding): ditto.
  (rb_io_extract_modeenc): use rb_econv_opts to initialize
  ecopts.
  (rb_file_open_generic): ditto.
  (rb_file_open_internal): ditto.
  (io_encoding_set): new argument: opt.
  (argf_set_encoding): copy fptr->encs.opts to argf_ecopts.

* transcode.c (rb_econv_opts): accept Qnil for initialization.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18817 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 09:40:31 +00:00
akr 09c7eba7b1 * include/ruby/encoding.h (rb_str_transcode): make 3rd argument
rb_econv_option_t*.

* transcode.c (transcode_loop): take rb_econv_option_t* as a argument.
  (str_transcode0): ditto.
  (str_transcode): make rb_econv_option_t and call str_transcode0 with
  it.
  (rb_str_transcode): take rb_econv_option_t*.

* io.c (io_fwrite): follow the rb_str_transcode change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18814 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 08:39:09 +00:00
akr f27b74142e * include/ruby/encoding.h (rb_econv_opts): declared.
* transcode.c (rb_econv_opts): defined.

* io.c (rb_io_extract_modeenc): use rb_econv_opts.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18811 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 07:20:21 +00:00
akr 9b8adfed22 * include/ruby/encoding.h (rb_econv_option_t): defined.
(rb_econv_open): 3rd arg changed.
  (rb_econv_open_exc): ditto.

* io.c (make_writeconv): use rb_econv_option_t.
  (make_readconv): ditto.
  (rb_econv_open): take rb_econv_option_t for options.
  (rb_econv_open_exc): ditto.
  (transcode_loop): use rb_econv_option_t.
  (econv_init): use rb_econv_option_t.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18810 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 07:12:44 +00:00
akr b47c46106a * transcode.c (rb_econv_substr_append): associate dst with destination
encoding when dst is created.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18809 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 06:44:21 +00:00
akr c6d291b003 * include/ruby/encoding.h (rb_str_transcode): add ecflags argument.
* transcode.c (econv_opts): extracted from str_transcode.
  (str_transcode_enc_args): extracted from str_transcode.
  (str_transcode0): extracted from str_transcode.
  (str_transcode): use econv_opts, str_transcode_enc_args,
  str_transcode0.
  (rb_str_transcode): call str_transcode0.
  (econv_primitive_insert_output): give the additional argument for
  rb_str_transcode.

* io.c (make_writeconv): use invalid/undef flags.
  (io_fwrite): ditto.
  (rb_scan_open_args): give the additional argument for
  rb_str_transcode.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18808 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 06:25:24 +00:00
akr 51ad3df2ce * transcode.c (str_transcode): check last hash only if 0 < argc.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18807 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 04:28:12 +00:00
akr bc0efbc9cd * include/ruby/encoding.h (rb_econv_substr_append): renamed from
rb_econv_string.
  (rb_econv_str_convert): declared.
  (rb_econv_substr_convert): declared.
  (rb_econv_str_append): declared.

* io.c (io_fwrite): use rb_econv_str_convert instead of
  rb_econv_string.

* transcode.c (rb_econv_substr_append): renamed from rb_econv_string.
  (rb_econv_str_append): new function.
  (rb_econv_substr_convert): ditto.
  (rb_econv_str_convert): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18806 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 03:22:43 +00:00
akr 7c2c7ea315 * transcode.c (rb_econv_open): return NULL on error.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18804 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 03:03:18 +00:00
akr 8f9ed3c464 * include/ruby/encoding.h (rb_econv_open_exc): declared.
* transcode.c (rb_eNoConverter): new exception.
  (rb_econv_open_exc): new function.
  (transcode_loop): use rb_econv_open_exc.

* io.c (make_writeconv): use rb_econv_open_exc.
  (make_readconv): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18803 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-24 02:42:37 +00:00
akr d4fabf0e36 * include/ruby/encoding.h (ECONV_INVALID_MASK): defined.
(ECONV_INVALID_IGNORE): defined.
  (ECONV_INVALID_REPLACE): defined.
  (ECONV_UNDEF_MASK): defined.
  (ECONV_UNDEF_IGNORE): defined.
  (ECONV_UNDEF_REPLACE): defined.

* transcode.c (INVALID_IGNORE): removed.
  (INVALID_REPLACE): removed.
  (UNDEF_IGNORE): removed.
  (UNDEF_REPLACE): removed.
  (rb_econv_convert0): renamed from rb_econv_convert.
  (rb_econv_convert): defined to call rb_econv_convert0 with
  replace/ignore behavior moved from transcode_loop.
  (transcode_loop): replace/ignore behavior removed.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18787 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-23 06:02:58 +00:00
akr c0d3881e0e * include/ruby/io.h (FMODE_TEXTMODE): defined.
* include/ruby/encoding.h (rb_econv_t): new field: flags.
  (rb_econv_binmode): declared.

* io.c (io_unread): text mode hack removed.
  (NEED_NEWLINE_DECODER): defined.
  (NEED_NEWLINE_ENCODER): defined.
  (NEED_READCONV): defined.
  (NEED_WRITECONV): defined.
  (TEXTMODE_NEWLINE_ENCODER): defined for windows.
  (make_writeconv): setup converter with TEXTMODE_NEWLINE_ENCODER for
  text mode.
  (io_fwrite): use NEED_WRITECONV.  character code conversion is
  disabled if fptr->writeconv_stateless is nil.
  (make_readconv): setup converter with
  ECONV_UNIVERSAL_NEWLINE_DECODER for text mode.
  (read_all): use NEED_READCONV.
  (appendline): use NEED_READCONV.
  (rb_io_getline_1): use NEED_READCONV.
  (io_getc): use NEED_READCONV.
  (rb_io_ungetc): use NEED_READCONV.
  (rb_io_binmode): OS-level text mode test removed.  call
  rb_econv_binmode.
  (rb_io_binmode_m): call rb_io_binmode_m with write_io as well.
  (rb_io_flags_mode): return mode string including "t".
  (rb_io_mode_flags): detect "t" for text mode.
  (rb_sysopen): always specify O_BINARY.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize flags.
  (rb_econv_open): if source and destination encoding is
  both empty string, open newline converter.  last_tc will be NULL in
  this case.
  (rb_econv_encoding_to_insert_output): last_tc may be NULL now.
  (rb_econv_string): ditto.
  (output_replacement_character): ditto.
  (transcode_loop): ditto.
  (econv_init): ditto.
  (econv_inspect): ditto.
  (rb_econv_binmode): new function.




git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-22 16:44:00 +00:00
akr 8f3e88c432 * transcode.c (rb_econv_open): fix memory leak.
(rb_econv_close): ditto.
  fixed by shinichiro.h.  [ruby-dev:35880]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18710 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-19 17:32:31 +00:00
akr 035d4816c3 * include/ruby/io.h (rb_io_t): new fields: writeconv,
writeconv_stateless and writeconv_initialized.
  (MakeOpenFile): initialize them.

* include/ruby/encoding.h (rb_econv_stateless_encoding): declared.
  (rb_econv_string): declared.

* io.c (make_writeconv): new function.
  (io_fwrite): use econv.
  (make_readconv): fix error message.
  (finish_writeconv): new function.
  (fptr_finalize): call finish_writeconv.
  (clear_writeconv): new function.
  (clear_codeconv): new function to call both clear_readconv and
  clear_writeconv.
  (rb_io_fptr_finalize): call clear_codeconv instead of
  clear_readconv.
  (mode_enc): ditto.
  (io_set_encoding): ditto.
  (argf_next_argv): ditto.
  (io_encoding_set): ditto.

* gc.c (gc_mark_children): mark writeconv_stateless in T_FILE.

* transcode.c (stateless_encoding_i): new function.
  (rb_econv_stateless_encoding): ditto.
  (rb_econv_string): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18691 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-18 12:06:42 +00:00
akr 8372c16764 * transcode.c (transcode_search_path): prevent infinite loop by
searching a path for self conversion.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18687 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-18 04:22:31 +00:00
akr 128abaede6 * transcode.c (econv_primitive_putback): new method.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18671 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-17 04:40:59 +00:00
akr 0051d4a077 * include/ruby/encoding.h (rb_econv_putbackable): declared.
(rb_econv_putback): ditto.

* transcode.c (rb_econv_putbackable): implemented.
  (rb_econv_putback): ditto.

* io.c (io_getc): put back bytes if possible.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18670 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-17 04:25:56 +00:00
akr 04bbc1df12 * transcode.c (make_econv_exception): add several instance variables
to exception object.
  (ecerr_source_encoding): new method:
  Encoding::ConversionUndefined#source_encoding and 
  Encoding::InvalidByteSequence#source_encoding.
  (ecerr_destination_encoding): new method:
  Encoding::ConversionUndefined#destination_encoding and 
  Encoding::InvalidByteSequence#destination_encoding.
  (econverr_error_char): new method:
  Encoding::ConversionUndefined#error_char.
  (econverr_error_bytes): new method:
  Encoding::ConversionUndefined#error_bytes.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-17 03:05:40 +00:00
akr 6acb0e79c8 * include/ruby/encoding.h (rb_econv_check_error): declared.
* transcode.c (make_econv_exception): new function.
  (transcode_loop): use make_econv_exception.
  (rb_econv_check_error): defined.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18662 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-16 15:04:39 +00:00
akr 4dff8b2428 * include/ruby/encoding.h (rb_econv_elem_t): fields removed: from and
to.
  (rb_econv_t): new fields: source_encoding_name and
  destination_encoding_name.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize the
  new fields.
  (rb_econv_open): set up the new fields.
  (econv_inspect): use the new fields.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18655 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-16 06:25:38 +00:00
akr 7a0bea4fd3 * include/ruby/encoding.h (rb_econv_t): add fields: in_buf_start,
in_data_start, in_data_end, in_buf_end and last_trans_index.
  (rb_econv_output): removed.
  (rb_econv_insert_output): declared.
  (rb_econv_encoding_to_insert_output): declared.

* enc/trans/newline.trans (rb_universal_newline): stateful_type
  changed.

* transcode.c (transcode_restartable0): initialize inchar_start,
  tc->recognized_len and next_table at beginning of the loop.
  (rb_econv_open_by_transcoder_entries): initialize new fields.
  (rb_econv_open): setup last_trans_index.
  (trans_sweep): last out_buf_start can be non-NULL now.
  (rb_econv_convert): check last out_buf_start and in_buf_start at
  first.
  (rb_econv_output_with_destination_encoding): removed.
  (econv_just_convert): removed.
  (rb_econv_output): removed.
  (econv_primitive_output): method removed.
  (rb_econv_encoding_to_insert_output): new function.
  (allocate_converted_string): new function.
  (rb_econv_insert_output): new function.
  (econv_primitive_insert_output): new method.
  (output_replacement_character): use rb_econv_insert_output.  unused
  arguments removed.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18654 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-16 05:32:42 +00:00
akr 3a427c6199 * transcode.c (econv_just_convert): extracted from rb_econv_output.
(rb_econv_output): use econv_just_convert.
  (econv_primitive_output): new method.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18647 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 14:17:11 +00:00
akr fe091e89b7 * include/ruby/encoding.h (rb_econv_output): add str_encoding
argument.

* transcode.c (get_replacement_character): add repl_enc_ptr argument.
  (rb_econv_output_with_destination_encoding): renamed from
  rb_econv_output and make it static.
  (rb_econv_output): convert str and call
  rb_econv_output_with_destination_encoding.
  (output_replacement_character): follow above interface change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18646 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 11:02:07 +00:00
akr 7f0c5beb6f * include/ruby/encoding.h (rb_econv_t): add error_tc in last_error.
* transcode.c (rb_econv_convert): fill error_tc.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18644 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 09:33:59 +00:00
akr 131e194cec * include/ruby/encoding.h (rb_econv_t): new field: last_error.
* transcode.c (rb_trans_conv): new argument: result_position_ptr.
  (rb_econv_convert): fill last_error.
  (econv_result_to_symbol): extracted from econv_primitive_convert.
  (econv_primitive_errinfo): new method.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18643 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 09:12:56 +00:00
akr 96ae1f008f * transcode.c (conv_init): check empty name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18637 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 00:34:16 +00:00
akr 520cd6d186 * transcode.c (econv_init): don't create dummy encoding if
rb_econv_open is failed.
  (make_dummy_encoding): new function extracted from make_encoding.
  (make_encoding): removed.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18634 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 00:05:06 +00:00
akr 95af70c5e7 udpate rdoc.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18632 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 16:10:57 +00:00
akr 8eda80ff2c * transcode.c (econv_primitive_convert): set destination_buffer
encoding.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18630 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 16:06:33 +00:00
akr 74a36d5d1f * include/ruby/encoding.h (rb_econv_output): declared.
* transcode_data.h (rb_transcoder): add resetsize_func field.

* enc/trans/iso2022.trans (iso2022jp_reset_sequence_size): defined.
  (rb_EUC_JP_to_ISO_2022_JP): provede resetsize_func.

* tool/transcode-tblgen.rb: set NULL for resetsize_func.

* transcode.c (rb_econv_output): new function for inserting output.
  (output_replacement_character): use rb_econv_output.
  (transcode_loop): check return value of
  output_replacement_character.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18628 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 15:56:39 +00:00
akr e0e39e0db8 * include/ruby/encoding.h (ECONV_UNIVERSAL_NEWLINE_DECODER): defined.
(ECONV_CRLF_NEWLINE_ENCODER): ditto.
  (ECONV_CR_NEWLINE_ENCODER): ditto.
  (ECONV_PARTIAL_INPUT): ditto.
  (ECONV_OUTPUT_FOLLOWED_BY_INPUT): ditto.

* transcode.c: use ECONV_* defined as above.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18627 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 14:48:41 +00:00
akr ed7d1a25cb * transcode.c: local variable renamed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18626 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 14:35:55 +00:00
akr c82aee31b4 * include/ruby/encoding.h (rb_econv_result_t): moved from
transcode_data.h.
  (rb_econv_elem_t): ditto.
  (rb_econv_t): ditto.  source_encoding and destination_encoding field
  is added.
  (rb_econv_open): declared.
  (rb_econv_convert): ditto.
  (rb_econv_close): ditto.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize
  source_encoding and destination_encoding field as NULL.
  (rb_econv_open): make it external linkage.
  (rb_econv_close): ditto.
  (rb_econv_convert): ditto.  renamed from rb_econv_conv.
  (make_encoding): new function.
  (econv_init): use make_encoding and store rb_encoding* in
  rb_econv_t.
  (econv_source_encoding): new method
  Encoding::Converter#source_encoding.
  (econv_destination_encoding): new method
  Encoding::Converter#destination_encoding.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18625 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 14:28:10 +00:00
akr 87779b507e * transcode_data.h (rb_econv_result_t): change enumeration
constant's prefix: transcode_ to econv_.

* transcode.c: follow the constant prefix change.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18624 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 13:45:54 +00:00
akr a6daaf16b0 * transcode.c (econv_init): accept Encoding object as source_encoding
and destination_encoding.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18623 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 12:35:19 +00:00
akr db30661aa9 * transcode.c (rb_econv_open_by_transcoder_entries): renamed from
rb_trans_open_by_transcoder_entries.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18621 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 11:59:47 +00:00
akr 5a4844ed6c * transcode.c (econv_max_output): removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18620 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 11:56:55 +00:00
akr bd9bd52954 * transcode_data.h (transcode_invalid_byte_sequence): renamed from
transcode_invalid_input.
  (transcode_destination_buffer_full): renamed from transcode_obuf_full.
  (transcode_source_buffer_empty): renamed from transcode_ibuf_empty.
  (rb_econv_result_t): renamed from rb_trans_result_t.
  (rb_econv_elem_t): renamed from rb_trans_elem_t.
  (rb_econv_t): renamed from rb_trans_t.

* transcode.c (UNIVERSAL_NEWLINE_DECODER): renamed from
  UNIVERSAL_NEWLINE.
  (CRLF_NEWLINE_ENCODER): renamed from CRLF_NEWLINE.
  (CR_NEWLINE_ENCODER): renamed from CR_NEWLINE.
  (rb_econv_open): renamed from rb_trans_open.
  (rb_econv_close): renamed from rb_trans_close.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18618 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 11:31:27 +00:00
akr dd92751104 * transcode.c (load_transcoder): unused function removed.
(rb_transcoding_open): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18617 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 10:42:28 +00:00
akr a99e6c15b8 * transcode.c (rb_econv_conv): new function. it don't consume input
too much, even for multilevel conversion.
  (transcode_loop): use rb_econv_conv.
  (econv_primitive_convert): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18610 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 06:35:33 +00:00
akr bcae240f19 * transcode_data.h (rb_trans_result_t): new enumeration constant:
transcode_output_followed_by_input.

* transcode.c (OUTPUT_FOLLOWED_BY_INPUT): new flag.
  (transcode_restartable0): suspend when output followed by input if
  OUTPUT_FOLLOWED_BY_INPUT is specified.
  (trans_sweep): check OUTPUT_FOLLOWED_BY_INPUT.
  (rb_trans_conv): support OUTPUT_FOLLOWED_BY_INPUT.
  (econv_primitive_convert): return :output_followed_by_input for
  transcode_output_followed_by_input.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 06:12:27 +00:00
nobu 914969a040 * thread.c (rb_thread_priority_set): get rid of C99 feature.
* transcode.c (transcode_search_path): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18606 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 05:56:51 +00:00
nobu 473dbc4e3f * transcode.c (load_transcoder): suppress warning.
* missing/vsnprintf.c (BSD_vfprintf): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18601 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 03:20:38 +00:00
akr ae8522ff7e rdoc updated.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18597 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 23:45:31 +00:00
akr fde26b7d7a * transcode.c (econv_primitive_convert): add output_byteoffset
argument.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18591 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 17:24:42 +00:00
nobu e6eb8f0399 * transcode.c (rb_trans_open): constified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18590 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 15:49:30 +00:00
akr 9adfb3066d * transcode.c (econv_inspect): new method.
+


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18578 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 08:47:57 +00:00
akr 3e7d8bf9f8 * transcode.c (transcode_restartable0): several local variables
removed.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18576 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 08:36:47 +00:00
akr 04f7390ca8 * transcode_data.h (rb_transcoding): add fields: writebuf_off,
writebuf_len and writebuf.
  (TRANSCODING_WRITEBUF): new macro.

* transcode.c (transcode_restartable0): output until output buffer is
  really full.
  (rb_transcoding_open_by_transcoder): initialize writebuf_len,
  writebuf_off and writebuf.
  (rb_transcoding_close): finalize writebuf.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 08:28:39 +00:00
akr 81719aff65 add rdoc.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18569 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 07:32:34 +00:00
akr 4aa251999d * transcode.c (econv_init): make flags argument optional.
(econv_primitive_convert): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 06:08:56 +00:00
akr a14d5eb09b * enc/trans/newline.trans (rb_crlf_newline): new transcoder.
(rb_cr_newline): new transcoder.

* transcode.c (trans_open_i): one more exra room for input newline
  converter.
  (rb_trans_open): crlf newline and cr newline implemented.
  (Init_transcode): Encoding::Converter::CRLF_NEWLINE and
  Encoding::Converter::LF_NEWLINE defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18557 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:48:57 +00:00
akr 74a2a7bdbf * enc/trans/newline.trans: new file.
* transcode_data.h (rb_trans_t): add last_tc field.

* transcode.c (UNIVERSAL_NEWLINE): defined.
  (CRLF_NEWLINE): defined.
  (CR_NEWLINE): defined.
  (rb_trans_open_by_transcoder_entries): initialize last_tc.
  (trans_open_i): allocate one more room for newline converter.
  (rb_trans_open): universal newline implemented.
  (more_output_buffer): take max_output argument instead ts.
  (output_replacement_character): take tc argument instead of ts.
  (transcode_loop): use last_tc field.
  (econv_init): add flags argument for rb_trans_open.
  (Init_transcode): Encoding::Converter::UNIVERSAL_NEWLINE defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:30:42 +00:00
akr e9ebe1a161 * transcode.c (transcode_search_path): return the length of converter
path.
  (load_transcoder_entry): renamed from load_transcoder.
  (load_transcoder): new function for loding transcoder by encoding
  names.
  (rb_transcoding_open_by_transcoder): extracted from
  rb_transcoding_open.
  (rb_transcoding_open): use load_transcoder and
  rb_transcoding_open_by_transcoder.
  (rb_trans_open_by_transcoder_entries): new function.
  (trans_open_i): construct entries array.
  (rb_trans_open): use rb_trans_open_by_transcoder_entries.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18551 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 02:40:24 +00:00
akr 5501d84ceb * transcode.c (econv_primitive_convert): add output_size argument.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 23:03:46 +00:00
akr 9d4bb3b969 * transcode.c (rb_trans_conv): report last transcode_obuf_full.
(econv_max_output): new method Encoding::Converter#max_output.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18547 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 22:43:17 +00:00
naruse fc6484cbf4 * transcode.c (str_transcode): fix error message.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18545 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 17:37:47 +00:00
akr edcd02cb5a * transcode.c (rb_cEncodingConverter): new class Encoding::Converter.
(econv_free): new function.
  (econv_s_allocate): ditto.
  (econv_init): ditto.
  (check_econv): ditto.
  (econv_primitive_convert): new method.
  (Init_transcode): define Encoding::Converter.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18544 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 14:46:18 +00:00
akr 608cb2ed32 * transcode.c (rb_trans_open): free ts before raise.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18535 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 09:51:42 +00:00
akr 6d87cec02c * transcode_data.h (rb_trans_elem_t): new field: from and to.
* transcode.c (trans_open_i): just record from and to.
  (rb_trans_open): load transcodings.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18531 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 09:42:49 +00:00
akr abef7fa890 * transcode.c (trans_open_i): check the result of rb_transcoding_open.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18526 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 08:17:27 +00:00
akr 5ade93542f * transcode_data.h (TRANSCODE_ERROR): removed.
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid
  (but unique to ASCII-8BIT) character.

* transcode.c (rb_eConversionUndefined): new error.
  (rb_eInvalidByteSequence): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 07:20:10 +00:00
akr 94ca2d94de * transcode_data.h (rb_transcoder): add resetstate_func field for
resetting a state of stateful encoding.

* enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify
  finish_eucjp_to_iso2022jp for resetstate_func.

* tool/transcode-tblgen.rb: specify NULL for resetstate_func.

* transcode.c (output_replacement_character): call resetstate_func
  before appending the replacement character.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 22:44:23 +00:00
akr 5f9b877ebe * transcode.c (get_replacement_character): extracted from
output_replacement_character.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 22:21:15 +00:00
akr 425098de96 * transcode.c (rb_trans_conv): find second last error.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 21:49:38 +00:00
akr a2901a7c75 * transcode_data.h (rb_trans_result_t): new type.
(rb_trans_elem_t): new type.
  (rb_trans_t): new type.

* transcode.c (transcode_dispatch_cb): removed.
  (transcode_dispatch): removed.
  (rb_transcoding_result_t): moved to rb_trans_result_t in
  transcode_data.h.
  (transcode_restartable0): goto follow_info when FUNsi.
  (rb_transcoding_open): use get_transcoder_entry.
  (rb_trans_open): new function.
  (rb_trans_conv): ditto.
  (rb_trans_close): ditto.
  (trans_open_i): ditto.
  (trans_sweep): ditto.
  (more_output_buffer): take rb_trans_t instead of rb_transcoding as
  an argument.
  (transcode_loop): take from_encoding and to_encoding instead of tr
  as arguments.  use rb_trans_open/rb_trans_conv/rb_trans_close.
  (str_transcode): don't use transcode_dispatch.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 15:50:42 +00:00
akr bd5bf3726c * transcode_data.h (rb_transcoding): rename fields.
readlen -> recognized_len.
  feedlen -> readagain_len.

* transcode.c: follow the field renaming.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18481 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 01:06:21 +00:00
akr badd9f1efc * transcode.c (transcode_restartable0): invalid handling simplified.
(transcode_restartable): use PARTIAL_INPUT for converting buffered
  input.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18476 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 13:37:44 +00:00
akr 436a136b4e * transcode_data.h (rb_transcoding): new field: flags.
* transcode.c (load_transcoder): extracted from transcode_dispatch_cb.
  (rb_transcoding_result_t): renamed from transcode_result_t.
  (rb_transcoding_open): new function.
  (rb_transcoding_convert): ditto.
  (rb_transcoding_close): ditto.
  (transcode_loop): use rb_transcoding_open, rb_transcoding_convert
  and rb_transcoding_close.
  (str_transcode): don't need rb_transcoding.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 12:26:01 +00:00
akr 2ce7aeca6e * transcode.c: rename my_transcoding to tc and my_transcoder to tr.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18468 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 02:33:05 +00:00
akr e748ed513d * transcode_data.h (rb_transcoding): add feedlen field.
* transcode.c (transcode_restartable0): renamed from
  transcode_restartable.
  save input buffer into feed buffer if next character is started the
  point before input buffer.  for example, "\x00\xd8\x01" then "\x02"
  in UTF-16LE.  \x02 causes invalid and next character is started from
  \x01.
  (transcode_restartable): new function to call
  transcode_restartable0.   if feed buffer is not empty, convert it at
  first.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 02:17:56 +00:00
akr b2718506ad * transcode.c (transcode_char_start): refactored to remove readlen
argument.
  (transcode_restartable): refactored to remove readlen variable.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18464 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 00:23:36 +00:00
akr 94dc1e0b4a r18455 reverted.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18462 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 23:33:37 +00:00
akr 0e3d10173b * transcode.c (transcode_loop): take destination and resize function
as parameters.
  (more_output_buffer): ditto.
  (str_transcoding_resize): argument changed from rb_transcoding* to
  VALUE.
  (str_transcode): call transcode_loop with destination string and its
  resize function.

* transcode_data.h (rb_transcoding): move ruby_string_dest and
  flush_func to transcode_loop parameters.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 13:34:21 +00:00
akr b9d3995310 * transcode.c (transcode_char_start): arguments changed.
(transcode_restartable): arguments changed to avoid *in_pos points
  out of buffer by decreasing *in_pos.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18455 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 10:32:02 +00:00
akr 4bceda1125 * transcode_data.h (rb_transcoding): new fields: next_info and next_byte.
* transcode.c (transcode_restartable): save/restore next_info and
  next_byte.
  sync readlen and in_p when invalid.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 07:35:28 +00:00
akr ca3dd2d89e * transcode.c (transcode_restartable): my_transcoder argument removed.
(transcode_loop): my_transcoder argument removed.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18453 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 06:25:05 +00:00
akr 139234e1a0 * transcode_data.h (rb_transcoding): add fields for restartable
transcoding.
  (rb_transcoder): add max_input field.
  from_unit_length field is renamed to input_unit_length.

* tool/transcode-tblgen.rb: generate max_input field.

* enc/trans/iso2022.erb.c: follow rb_transcoder change.

* enc/trans/utf_16_32.erb.c: ditto.

* transcode.c (PARTIAL_INPUT): new constant.
  (transcode_char_start): new function.
  (transcode_result_t): new type.
  (transcode_restartable): new function.
  (more_output_buffer): new function.
  (transcode_loop): use transcode_restartable.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 06:02:01 +00:00
akr 2833d9f95d * transcode_data.h (rb_transcoder): from_unit_length field added.
from_utf8 field removed.

* tool/transcode-tblgen.rb: generate offsets range.
  follow rb_transcoder change.

* transcode.c (transcode_loop): don't use from_utf8.
  make invalid region from_unit_length wise.

* enc/trans/iso2022.erb.c: follow rb_transcoder and 
  transcode_generate_node change.

* enc/trans/utf_16_32.erb.c: follow rb_transcoder and
  transcode_generate_node change.
  explicit :invalid map removed.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08 15:48:17 +00:00
akr 1504652373 * transcode_data.h (rb_transcoding): new field "stateful".
(rb_transcoder): preprocessor and postprocessor field removed.
  change arguments of func_ii, func_si, func_io and func_so.
  new field "finish_func".

* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
  generatable.

* transcode.c (transcoder_lib_table): removed.
  (transcoder_table): change structure.
  (transcoder_key): removed because the above structure change.
  (make_transcoder_entry): new function.
  (get_transcoder_entry): ditto.
  (rb_register_transcoder): follow the structure change.
  (declare_transcoder): ditto.
  (transcode_search_path): new function for breadth first search to
  find a list of converters.
  (transcode_search_path_i): new function.
  (transcode_dispatch_cb): ditto.
  (transcode_dispatch): use transcode_search_path.
  (transcode_loop): follow the argument change.
  (str_transcode): preprocessor and postprocessor stuff removed.

* enc/trans/iso2022.erb.c: new file.  ISO-2022-JP conversion
  re-implemented.

* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.

nute(23:52:53)% head -40 ChangeLog
Thu Aug  7 23:43:11 2008  Tanaka Akira  <akr@fsij.org>

* transcode_data.h (rb_transcoding): new field "stateful".
  (rb_transcoder): preprocessor and postprocessor field removed.
  change arguments of func_ii, func_si, func_io and func_so.
  new field "finish_func".

* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
  generatable.

* transcode.c (transcoder_lib_table): removed.
  (transcoder_table): change structure.
  (transcoder_key): removed because the above structure change.
  (make_transcoder_entry): new function.
  (get_transcoder_entry): ditto.
  (rb_register_transcoder): follow the structure change.
  (declare_transcoder): ditto.
  (transcode_search_path): new function for breadth first search to
  find a list of converters.
  (transcode_search_path_i): new function.
  (transcode_dispatch_cb): ditto.
  (transcode_dispatch): use transcode_search_path.
  (transcode_loop): follow the argument change.
  (str_transcode): preprocessor and postprocessor stuff removed.

* enc/trans/iso2022.erb.c: new file.  ISO-2022-JP conversion
  re-implemented.

* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.

* enc/trans/utf_16_32.erb.c: follow argument change of FUNso.

[ruby-dev:35798]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-07 14:53:30 +00:00
akr 5adb247914 * tool/transcode-tblgen.rb: distinguish UNDEF and INVALID.
[ruby-dev:35709]

* transcode.c (transcode_loop): don't need rb_enc_mbclen now.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18390 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-06 11:47:14 +00:00
nobu 29732ed8f2 * transcode.c (str_encode): no need to duplicate first.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18379 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05 20:26:45 +00:00
nobu b84d31c524 * transcode_data.h (TRANSCODE_ERROR): common transcode failure
exception, would be changed later.

* enc/trans/japanese.c (UNSUPPORTED_MODE): unsupported mode transition
  exception.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05 03:34:52 +00:00
nobu 3916eb783c * marshal.c (dump_ensure), process.c (run_exec_dup2),
string.c (rb_str_replace), transcode.c (transcode_dispatch): fixed
  memory leaks.  based on patches from shinichiro.h <shinichiro.hamaji
  AT gmail.com> at [ruby-dev:35751].


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18341 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-04 05:08:57 +00:00
naruse 95cd2c11b9 * transcode.c (transcode_loop): undefined character is replaced with
only one character. [ruby-dev:35709]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18312 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-01 14:29:25 +00:00
naruse 00aef398d0 * transcode.c (output_replacement_character):
rename from _get_replacement_character.

* transcode.c (output_replacement_character):
  fix replacement on UTF-32{BE,LE}. [ruby-dev:35705]

* transcode.c (transcode_loop): ditto.

* test/ruby/test_transcode.rb (test_invalid_replace):
  add for above.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18300 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31 20:35:35 +00:00
naruse 931ba3f3b7 * transcode.c (get_replacement_character): use U+FFFD as replacement
character when convert to Unicode.

* test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):
  rename from test_public_review_issue_121.

* test/ruby/test_transcode.rb (test_unicode_public_review_issue_121):
  enable option2.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18294 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31 10:59:39 +00:00
naruse 97fdfbacec * transcode.c (get_replacement_character): fix: invalid byte sequence
is always replaced "\x00?".

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18293 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-31 08:12:08 +00:00
nobu 94ed51b281 * transcode.c (transcode_loop): constified.
* transcode.c (str_transcode): rb_str_set_len() sets a delimiter.

* transcode_data.h (rb_transcoder): constified preprocessor and
  postprocessor input.

* enc/trans/japanese.c: ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-14 09:47:33 +00:00
naruse ef62472e44 * transcode.c (get_replacement_character): temporary function,
get characters for replacement mode.

* transcode.c (transcode_loop): add undef key and replace value.

* transcode.c (str_transcode): ditto.

* transcode.c (Init_transcode): define sym_undef and sym_replace.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18062 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-14 06:27:26 +00:00
nobu f94f8f351e * configure.in (sizeof ptrdiff_t): check for size of ptrdiff_t.
* include/ruby/ruby.h (PRI?PTRDIFF, PRI?SIZE): printf conversion
  specifiers for ptrdiff_t and size_t/ssize_t.

* insns.def (leave), marshal.c (long_toobig), transcode.c
  (str_transcode), vm_dump.c (control_frame_dump, stack_dump_each),
  (debug_print_register, debug_print_pre): t and z length modifiers
  are C99.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-12 13:17:29 +00:00
nobu 0455e8ea9a * io.c (rb_f_open), re.c (rb_reg_search), transcode.c (str_transcode):
suppress warnings.

* util.c (quorem, rv_alloc, nrv_alloc): only used in dtoa().


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16873 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-07 12:38:03 +00:00
naruse 9256f94d9a * insns.def (DEFINE_INSN): subtract of pointers is ptrdiff_t.
this is not int on 64bit system.

* vm_dump.c (control_frame_dump): ditto.

* vm_dump.c (stack_dump_each): ditto.

* vm_dump.c (debug_print_register): ditto.

* vm_dump.c (debug_print_pre): ditto.

* transcode.c (str_transcode): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16735 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-06-01 03:23:50 +00:00
naruse 327515a95b * transcode.c (rb_str_transcode): argc is 1, and argv is &to.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-05-23 23:15:14 +00:00
naruse 95698ff2a4 * transcode.c, include/ruby/encodng.h (rb_str_transcode):
C API of encoding conversion for Ruby object.
  VALUE rb_str_transcode(VALUE str, VALUE to).

* transcode.c (str_encode, str_encode_bang):
  rename from rb_tr_transcode or rb_str_transcode_bang.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@16496 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-05-20 16:37:13 +00:00
nobu 1369cfd16e * encoding.c (enc_init_db): moved to enc/encdb.c.
* transcode.c (init_transcoder_table): moved to enc/trans/transdb.c.

* enc/depend (enc/encdb.o enc/trans/transdb.o): depend on
  corresponding headers.

* common.mk (COMMONOBJS): moved transcode.o from OBJS


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-04-07 06:51:33 +00:00
naruse d2797c33bd * transcode.c (rb_str_transcode_bang): set coderange.
* transcode.c (rb_str_transcode): use rb_str_transcode_bang.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15707 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-06 05:30:48 +00:00
duerst 08631278ad Web Mar 5 17:43:43 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c (transcode_loop): Adjusted detection of invalid
	  (ill-formed) UTF-8 sequences. Fixing potential security issue, see
	  http://www.unicode.org/versions/Unicode5.1.0/#Notable_Changes.

	* test/ruby/test_transcode.rb: Added two tests for above fix.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15692 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-03-05 08:45:51 +00:00
duerst 6d5ef97a32 Thu Feb 21 17:15:15 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c: Added basic support for passing options to String#encode
	  via a hash. Currently only one option, with one value, is supported:
	  invalid: :ignore (dropping invalid byte sequences instead of
	  producing an error). Option naming is not yet stable!

	* test/ruby/test_transcode.rb: Added a single test for invalid: :ignore
	  option. Not more tests because most data does not yet distinguish
	  between INVALID and UNKNOWN.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-02-21 08:42:10 +00:00
naruse 74b254e833 * enc/trans/japanese.c (rb_to_Windows_31J): to 'Windows-31J'.
* common.mk: add rules for transdb.h.

* transcode.c (init_transcoder_table): use transdb.h.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-29 10:05:39 +00:00
duerst 38321fc0eb Mon Jan 21 19:42:42 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb:
	  added UTF-32BE and UTF-32LE conversions.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15156 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 10:41:59 +00:00
nobu 282e828c59 * transcode.c (str_transcode): initialize transcoder in
rb_transcoding.  [ruby-dev:33234]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15153 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 05:36:16 +00:00
nobu f5eb90f3c2 * transcode.c (str_transcode): initialize transcoder in
rb_transcoding.  [ruby-dev:33234]

* transcode_data.h (rb_transcoding): transcoder constified.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 05:32:12 +00:00
nobu 463af63468 * transcode.c (transcode_loop, str_transcoding_resize): use unsigned
char.  [ruby-dev:33232]

* transcode_data.h (rb_transcoding, rb_transcoder): removed callback
  parameters.

* enc/trans/japanese.c: ditto.

* enc/trans/utf_16_32.c: parenthesized bit-or operands.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15150 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 03:35:05 +00:00
nobu a8969e999a * transcode.c (transcode_dispatch): constified return value.
* transcode_data.h (rb_transcoding): include pointer to rb_transcoder
  and auxiliary data.

* transcode_data.h (rb_transcoder): all callback functions shoud have
  their own parameters.

* enc/trans/{japanese,single_byte}.c: constified.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-20 21:40:08 +00:00
duerst a9b15a4e0c Sun Jan 20 20:00:20 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c, enc/trans/utf_16_32.c, test/ruby/test_transcode.rb:
	  added UTF-16LE conversions.

	* fixed changelog for last commit



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15144 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-20 11:00:24 +00:00
duerst 3d0c7bea4d Sun Jan 20 15:08:08 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* enc/trans/utf_16_32.c: new file, currently implementing
	  UTF-16BE conversions only.

	* test/ruby/test_transcode.rb: Added tests for UTF-16BE;
	  made check_both_ways() use force_encoding differently.

	* transcode_data.h, transcode.c: Support for more conversion
	  functions.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-20 06:12:48 +00:00
nobu ce2652c9d4 * include/ruby/intern.h (rb_str_tmp_new, rb_str_shared_replace):
prototype moved.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-16 03:51:32 +00:00
akr 6cdef2dc7e * $Date$ keyword removed to avoid inclusion of locale dependent
string.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-06 15:49:38 +00:00
akr 041e829127 * include/ruby/encoding.h (rb_isascii): defined.
(rb_isalnum): ditto.
  (rb_isalpha): ditto.
  (rb_isblank): ditto.
  (rb_iscntrl): ditto.
  (rb_isdigit): ditto.
  (rb_isgraph): ditto.
  (rb_islower): ditto.
  (rb_isprint): ditto.
  (rb_ispunct): ditto.
  (rb_isspace): ditto.
  (rb_isupper): ditto.
  (rb_isxdigit): ditto.
  (rb_tolower): ditto.
  (rb_toupper): ditto.

* include/ruby/st.h (st_strcasecmp): declared.
  (st_strncasecmp): ditto.

* st.c (type_strcasehash): use st_strcasecmp instead of strcasecmp.
  (st_strcasecmp): defined.
  (st_strncasecmp): ditto.

* include/ruby/ruby.h: include include/ruby/encoding.h.
  (ISASCII): use rb_isascii.
  (ISPRINT): use rb_isprint.
  (ISSPACE): use rb_isspace.
  (ISUPPER): use rb_isupper.
  (ISLOWER): use rb_islower.
  (ISALNUM): use rb_isalnum.
  (ISALPHA): use rb_isalpha.
  (ISDIGIT): use rb_isdigit.
  (ISXDIGIT): use rb_isxdigit.
  (TOUPPER): defined.
  (TOLOWER): ditto.
  (STRCASECMP): ditto.
  (STRNCASECMP): ditto.

* dir.c, encoding.c, file.c, hash.c, process.c, ruby.c, time.c,
  transcode.c, ext/readline/readline.c: use locale insensitive
  functions.  [ruby-core:14662]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14829 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-01 12:24:04 +00:00
duerst 793e9423cd Fri Dec 28 01:55:04 2007 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c (transcode_dispatch): reverted some of the changes
          in r14746.

	* transcode.c, enc/trans/single_byte.c: Added conversions to/from
	  US-ASCII and ASCII-8BIT (using data tables).

	* enc/trans/single_byte.c: Some spacing/ordering changes due to
	  automatic data file generation.

	* transcode_data.h, transcode.c: Preliminary code for using
	  micro-conversion functions.

	* test/ruby/test_transcode.rb: Added some tests for US-ASCII and
	  ASCII-8BIT conversions.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-28 09:26:55 +00:00
nobu 3b83e10790 * transcode.c (transcode_dispatch): allows transcoding from/to
ASCII-8BIT.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14746 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-27 16:55:06 +00:00
akr efd7504d44 * parse.y, transcode_data.h, transcode.c: change "illegal" to
"invalid" in a context which doesn' t against a law.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14735 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-27 08:27:19 +00:00
nobu 3e65d11090 * transcode.c (transcode_dispatch): fix for multistep transcode.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-25 06:21:35 +00:00
nobu b7db9036be * common.mk (COMMONOBJS): transcode_data_*.c moved under enc/trans.
* transcode_data.h (rb_transcoding, rb_transcoder): prefixed.

* transcode.c (rb_register_transcoder, rb_declare_transcoder): split
  declaration and registration.  [ruby-dev:32704]

* transcode.c (transcode_dispatch): autoload pre-declared transcoder.

* transcode.c (str_transcode): use rb_define_dummy_encoding().

* transcode.c (Init_transcode): initialize transcoder tables.

* enc/trans/single_byte.c, enc/trans/japanese.c: moved from top.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14666 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-25 05:57:04 +00:00
duerst e7ac333ba8 Tue Dec 25 12:32:32 2007 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c: Moving a static counter from inside register_transcoder()
	  and register_functional_transcoder() to outside the functions, renaming
	  from n to next_transcoder_position. Fixes 3) in [ruby-dev:32715].



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14651 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-25 03:33:23 +00:00
naruse be86e3de33 * transocode.c: register_functional_transcoder() added.
(init_transcoder_table(: register ISO-2022-JP.
  (str_transcode): add preprocessor and postprocessor.

* transcode_data_japanese.c: add ISO-2022-JP support.

* transcode_data.h: moved transcoder and transcoding difinition from
  transcode.c.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14607 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-24 13:51:19 +00:00
duerst 9c7718ac6b Mon Dec 24 09:45:45 2007 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c, transcode_data_one_byte.c, transcode_data_japanese.c:
	  added rb_ prefix to external data symbols.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14561 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-24 00:47:21 +00:00
akr 5b809a28f8 * include/ruby/encoding.h, encoding.c, re.c, io.c, parse.y, numeric.c,
ruby.c, transcode.c: rename rb_ascii_encoding. to
  rb_ascii8bit_encoding.  rb_ascii_encoding is ambiguous with 
  ASCII-8BIT and US-ASCII.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14504 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-22 23:47:18 +00:00
duerst 5ad8c5566d Sat Dec 22 15:45:45 2007 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode_data_one_byte: slightly optimized

	* transcode_data_japanese: new data file for EUC-JP and SHIFT_JIS
	  (not yet optimized; tests to follow; data from
	   http://nkf.sourceforge.jp/ucm/{SJIS|eucJP}-nkf.ucm)

	* common.mk, transcode.c: Adjusted for transcode_data_japanese



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14472 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-22 06:45:55 +00:00
matz d7cc14d436 * encoding.c (rb_ascii_encoding): renamed from previous
rb_default_encoding().

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14443 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-21 18:55:30 +00:00
nobu d66a188c4a * transcode.c (rb_str_transcode_bang): returns self if no conversion.
[ruby-dev:32662]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14425 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-21 08:49:08 +00:00
nobu 98da73bc96 * transcode.c (rb_str_transcode_bang, rb_str_transcode): set new
encoding even if no conversion is done because of 7bit only.
  [ruby-dev:32591]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14293 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-18 08:04:26 +00:00
matz 5c4cf9bfdf for undefined conversions.
* transcode_data_iso_8859.c: Changed from character constants
  ('\xC2') to integer contants (0xC2) for shorter files and
  better readability; eliminated duplicated tables; changed
  from -1 offset to actual UNDEF entry (not yet distinguishing
  UNDEF and ILLEGAL correctly).

* test/ruby/test_transcode.rb: added a test for UNDEF conversion.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-17 01:28:26 +00:00
matz f2b0dba1cf * transcode.c (str_transcode, transcode_dispatch): added two-step
* trancode.c: some minor formatting fixes

* transcode_data.h, transcode_data_iso_8859.c: Shortened
  extremely frequently used macros to shorten file length.

* test/ruby/test_transcode.rb: Fixed name of test class;
  added setup method to ensure all necessary encodings exist;
  split tests into more test methods; added tests; fixed ordering
  of arguments in assert_equal to have expected result first.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14236 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-15 05:42:25 +00:00
nobu a153fdb6ed * transcode.c (transcode_loop): get rid of SEGV at sequence can not be
converted.

* transcode.c (rb_str_transcode_bang): copy encoding.  [ruby-dev:32532]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14191 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-11 04:57:51 +00:00
nobu 6d7999c132 * transcode.c (str_transcode): allow non-registered encodings.
[ruby-dev:32520]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14182 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-10 12:47:55 +00:00
nobu 3a3bda73dd * string.c (rb_str_tmp_new): creates hidden temporary buffer.
* transcode.c (transcoding): added a pointer to function to flush.

* transcode.c (transcode_loop): do not use string internal.
  [ruby-dev:32512]

* transcode.c (str_transcode): allow Encoding objects.

* transcode_data.h (BYTE_LOOKUP): use actual struct name.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14176 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-10 08:46:06 +00:00
nobu 38b92f838f * transcode*.[ch], test/ruby/test_transcode.rb: set properties.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-10 08:25:01 +00:00
matz 7ded13f54b * transcode.c: new file to provide encoding conversion features.
code contributed by Martin Duerst.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14172 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-10 05:01:47 +00:00