Граф коммитов

54 Коммитов

Автор SHA1 Сообщение Дата
akr 6eb3843b17 * transcode_data.h (STR1_LENGTH): defined.
(makeSTR1LEN): defined.

* tool/transcode-tblgen.rb: use makeSTR1LEN.  generate STR1 for 4 to
  259 bytes.

* transcode.c (rb_transcoding): new field: output_index.
  (transcode_restartable0): use STR1_LENGTH.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19366 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-15 15:42:30 +00:00
akr 0675246ba6 * transcode_data.h (rb_transcoder): resetsize_func and resetstate_func
also returns ssize_t.

* enc/trans/iso2022.trans: follow the type change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19354 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-15 02:11:50 +00:00
akr c767be3039 * transcode_data.h: return output functions ssize_t.
* transcode.c (transcode_restartable0): don't need to cast the result
  of output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19351 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:35:17 +00:00
akr a3c8c0adec * transcode_data.h: output function takes output buffer size.
* transcode.c: give output buffer size for output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:06:20 +00:00
akr 6270ad5b7f * include/ruby/encoding.h (rb_econv_asciicompat_encoding): renamed
from rb_econv_stateless_encoding to apply stateless ASCII
  incompatible encodings such as UTF-16BE.

* io.c (make_writeconv): use rb_econv_asciicompat_encoding.

* transcode_data.h (rb_transcoder_asciicompat_type_t): renamed from
  rb_transcoder_stateful_type_t.
  (rb_transcoder): use rb_transcoder_asciicompat_type_t.

* transcode.c: follow the type change.
  (asciicompat_encoding_i): renamed from stateless_encoding_i.
  (rb_econv_asciicompat_encoding): renamed from
  rb_econv_stateless_encoding.
  (econv_s_asciicompat_encoding): method renamed.

* tool/transcode-tblgen.rb: follow the type change.

* enc/trans/utf_16_32.trans: follow the type change.
  rb_from_UTF_16BE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_16LE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_32BE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_32LE to UTF-8 is asciicompat_decoder.
  UTF-8 to rb_to_UTF_16BE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_16LE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_32BE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_32LE is asciicompat_encoder.

* enc/trans/newline.trans: follow the type change.  universal newline
  decoder is asciicompat_converter.

* enc/trans/escape.trans: follow the type change.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 14:33:17 +00:00
akr c0bec2fae1 * transcode_data.h (STR1): defined for a string up to 255 bytes.
(STR1_BYTEINDEX): defined.
  (makeSTR1): defined.

* tool/transcode-tblgen.rb: generate STR1.

* transcode.c (transcode_restartable0): interpret STR1.

* enc/trans/escape.trans (fun_so_escape_xml_chref): removed.  STR1 is
  used instead.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07 09:13:16 +00:00
akr f549f1ae32 * transcode_data.h (rb_transcoder): rename fields:
from_encoding -> src_encoding, to_encoding -> dst_encoding.

* transcode.c: follow the renaming.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-04 15:46:18 +00:00
akr 0031967d20 * transcode_data.h (PType): defined unconditionaly.
* transcode.c (PType): don't define here.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19127 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-04 11:31:37 +00:00
akr c359b80127 comment updated.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19107 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03 16:31:14 +00:00
akr 1afc1b7bc8 * transcode.c (rb_transcoding): moved from transcode_data.h.
(TRANSCODING_READBUF): ditto.
  (TRANSCODING_WRITEBUF): ditto.
  (TRANSCODING_STATE_EMBED_MAX): ditto.
  (TRANSCODING_STATE): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03 14:34:09 +00:00
akr f6441bf61c * transcode_data.h (rb_transcoding): remove stateful field.
add state field.
  (TRANSCODING_STATE): defined.
  (rb_transcoder): add fields: state_size, state_init_func,
  state_fini_func.
  change rb_transcoding* argument to void*.

* transcode.c (transcode_restartable0): use TRANSCODING_STATE for
  first arguments of transcoder functions.
  (rb_transcoding_open_by_transcoder): initialize state field.
  (rb_transcoding_close): finalize state field.

* tool/transcode-tblgen.rb: provide state size/init/fini.

* enc/trans/newline.trans (universal_newline_init): defined.
  (fun_so_universal_newline): take void* as a state pointer.
  (rb_universal_newline): provide state size/init/fini.
  (rb_crlf_newline): ditto.
  (rb_cr_newline): ditto.

* enc/trans/iso2022.trans (iso2022jp_init): defined.
  (fun_si_iso2022jp_to_eucjp): take void* as a state pointer.
  (fun_so_iso2022jp_to_eucjp): ditto.
  (fun_so_eucjp_to_iso2022jp): ditto.
  (iso2022jp_reset_sequence_size): ditto.
  (finish_eucjp_to_iso2022jp): ditto.
  (rb_ISO_2022_JP_to_EUC_JP): provide state size/init/fini.
  (rb_EUC_JP_to_ISO_2022_JP): ditto.

* enc/trans/utf_16_32.trans (fun_so_from_utf_16be): take void* as a
  state pointer.
  (fun_so_to_utf_16be): ditto.
  (fun_so_from_utf_16le): ditto.
  (fun_so_to_utf_16le): ditto.
  (fun_so_from_utf_32be): ditto.
  (fun_so_to_utf_32be): ditto.
  (fun_so_from_utf_32le): ditto.
  (fun_so_to_utf_32le): ditto.
  (rb_from_UTF_16BE): provide state size/init/fini.
  (rb_to_UTF_16BE): ditto.
  (rb_from_UTF_16LE): ditto.
  (rb_to_UTF_16LE): ditto.
  (rb_from_UTF_32BE): ditto.
  (rb_to_UTF_32BE): ditto.
  (rb_from_UTF_32LE): ditto.
  (rb_to_UTF_32LE): ditto.

* enc/trans/japanese.trans (fun_so_eucjp2sjis): take void* as a state
  pointer.
  (fun_so_sjis2eucjp): ditto.
  (rb_eucjp2sjis): provide state size/init/fini.
  (rb_sjis2eucjp): provide state size/init/fini.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19096 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03 14:12:06 +00:00
akr 6378201f8b * transcode_data.h (WORDINDEX_SHIFT_BITS): defined.
(WORDINDEX2INFO): defined.
  (INFO2WORDINDEX): defined.

* tool/transcode-tblgen.rb: use WORDINDEX2INFO.

* transcode.c: use INFO2WORDINDEX.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19093 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03 12:35:29 +00:00
akr 4277473d5b * transcode_data.h (rb_transcoder): new field: byte_array_length and
word_array_length.

* tool/transcode-tblgen.rb (transcode_generated_code): generate
  byte_array_length and word_array_length.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19088 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-03 11:48:17 +00:00
akr 41e1933fd8 * transcode_data.h (base_element): removed.
(BYTE_LOOKUP): removed.
  (BYTE_LOOKUP_BASE): don't cast.
  (BYTE_LOOKUP_INFO): ditto.
  (PType): unsigned int, instead of uintptr_t.
  (rb_transcoding): change type of next_field, conv_tree_start and
  word_array.

* tool/transcode-tblgen.rb: generate word_array as array of unsigned
  int.

* transcode.c (transcode_restartable0): follow the above type change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19070 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-02 13:26:02 +00:00
akr 784ccb5806 * transcode_data.h (o3): prevent sign extension on 64bit environment.
(o4): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19048 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 19:01:27 +00:00
akr 6058eeaf03 * tool/transcode-tblgen.rb: record infos and BYTE_LOOKUPs as index of
word_array to avoid relocation.

* transcode.c (transcode_restartable0): add word_array to get infos
  and BYTE_LOOKUPs.

* transcode_data.h (BYTE_LOOKUP_INFO): change return type to
  uintptr_t.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 18:06:28 +00:00
akr 7908180df1 * tool/transcode-tblgen.rb: record offsets array as index of
byte_array to avoid relocation.

* transcode.c (transcode_restartable0): add byte_array to get offsets
  array.

* transcode_data.h (BYTE_LOOKUP_BASE): change return type to
  uintptr_t.
  (rb_transcoder): add fields: byte_array, word_array and word_size.

* enc/trans/newline.trans: follow rb_transcoder change.

* enc/trans/iso2022.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 17:40:32 +00:00
akr 752e053a1d * transcode_data.h (BYTE_LOOKUP): change to uintptr_t array.
(BYTE_LOOKUP_BASE): follow the type change.
  (BYTE_LOOKUP_INFO): ditto.
  (PType): ditto.
  (rb_transcoding): ditto.

* tool/transcode-tblgen.rb: follow the type change.

* transcode.c: ditto.

* enc/trans/newline.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 16:22:49 +00:00
akr 2ae7b0328e * transcode_data.h (BYTE_LOOKUP_BASE): abstract accessor for
BYTE_LOOKUP.
  (BYTE_LOOKUP_INFO): ditto.

* transcode.c (transcode_restartable0): use BYTE_LOOKUP_BASE and
  BYTE_LOOKUP_INFO.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19037 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 14:33:56 +00:00
akr a55e167b68 * transcode_data.h (rb_transcoder_stateful_type_t): defined.
(rb_transcoder): add field: stateful_type.

* tool/transcode-tblgen.rb: generate stateful_type field as
  stateless_converter.

* enc/trans/iso2022.trans: follow rb_transcoder change.

* enc/trans/newline.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 23:13:01 +00:00
akr 74a36d5d1f * include/ruby/encoding.h (rb_econv_output): declared.
* transcode_data.h (rb_transcoder): add resetsize_func field.

* enc/trans/iso2022.trans (iso2022jp_reset_sequence_size): defined.
  (rb_EUC_JP_to_ISO_2022_JP): provede resetsize_func.

* tool/transcode-tblgen.rb: set NULL for resetsize_func.

* transcode.c (rb_econv_output): new function for inserting output.
  (output_replacement_character): use rb_econv_output.
  (transcode_loop): check return value of
  output_replacement_character.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18628 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 15:56:39 +00:00
akr c82aee31b4 * include/ruby/encoding.h (rb_econv_result_t): moved from
transcode_data.h.
  (rb_econv_elem_t): ditto.
  (rb_econv_t): ditto.  source_encoding and destination_encoding field
  is added.
  (rb_econv_open): declared.
  (rb_econv_convert): ditto.
  (rb_econv_close): ditto.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize
  source_encoding and destination_encoding field as NULL.
  (rb_econv_open): make it external linkage.
  (rb_econv_close): ditto.
  (rb_econv_convert): ditto.  renamed from rb_econv_conv.
  (make_encoding): new function.
  (econv_init): use make_encoding and store rb_encoding* in
  rb_econv_t.
  (econv_source_encoding): new method
  Encoding::Converter#source_encoding.
  (econv_destination_encoding): new method
  Encoding::Converter#destination_encoding.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18625 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 14:28:10 +00:00
akr 87779b507e * transcode_data.h (rb_econv_result_t): change enumeration
constant's prefix: transcode_ to econv_.

* transcode.c: follow the constant prefix change.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18624 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 13:45:54 +00:00
akr bd9bd52954 * transcode_data.h (transcode_invalid_byte_sequence): renamed from
transcode_invalid_input.
  (transcode_destination_buffer_full): renamed from transcode_obuf_full.
  (transcode_source_buffer_empty): renamed from transcode_ibuf_empty.
  (rb_econv_result_t): renamed from rb_trans_result_t.
  (rb_econv_elem_t): renamed from rb_trans_elem_t.
  (rb_econv_t): renamed from rb_trans_t.

* transcode.c (UNIVERSAL_NEWLINE_DECODER): renamed from
  UNIVERSAL_NEWLINE.
  (CRLF_NEWLINE_ENCODER): renamed from CRLF_NEWLINE.
  (CR_NEWLINE_ENCODER): renamed from CR_NEWLINE.
  (rb_econv_open): renamed from rb_trans_open.
  (rb_econv_close): renamed from rb_trans_close.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18618 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 11:31:27 +00:00
akr bcae240f19 * transcode_data.h (rb_trans_result_t): new enumeration constant:
transcode_output_followed_by_input.

* transcode.c (OUTPUT_FOLLOWED_BY_INPUT): new flag.
  (transcode_restartable0): suspend when output followed by input if
  OUTPUT_FOLLOWED_BY_INPUT is specified.
  (trans_sweep): check OUTPUT_FOLLOWED_BY_INPUT.
  (rb_trans_conv): support OUTPUT_FOLLOWED_BY_INPUT.
  (econv_primitive_convert): return :output_followed_by_input for
  transcode_output_followed_by_input.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-14 06:12:27 +00:00
akr 04f7390ca8 * transcode_data.h (rb_transcoding): add fields: writebuf_off,
writebuf_len and writebuf.
  (TRANSCODING_WRITEBUF): new macro.

* transcode.c (transcode_restartable0): output until output buffer is
  really full.
  (rb_transcoding_open_by_transcoder): initialize writebuf_len,
  writebuf_off and writebuf.
  (rb_transcoding_close): finalize writebuf.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18573 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 08:28:39 +00:00
akr 74a2a7bdbf * enc/trans/newline.trans: new file.
* transcode_data.h (rb_trans_t): add last_tc field.

* transcode.c (UNIVERSAL_NEWLINE): defined.
  (CRLF_NEWLINE): defined.
  (CR_NEWLINE): defined.
  (rb_trans_open_by_transcoder_entries): initialize last_tc.
  (trans_open_i): allocate one more room for newline converter.
  (rb_trans_open): universal newline implemented.
  (more_output_buffer): take max_output argument instead ts.
  (output_replacement_character): take tc argument instead of ts.
  (transcode_loop): use last_tc field.
  (econv_init): add flags argument for rb_trans_open.
  (Init_transcode): Encoding::Converter::UNIVERSAL_NEWLINE defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:30:42 +00:00
akr 6d87cec02c * transcode_data.h (rb_trans_elem_t): new field: from and to.
* transcode.c (trans_open_i): just record from and to.
  (rb_trans_open): load transcodings.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18531 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 09:42:49 +00:00
akr 5ade93542f * transcode_data.h (TRANSCODE_ERROR): removed.
* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid
  (but unique to ASCII-8BIT) character.

* transcode.c (rb_eConversionUndefined): new error.
  (rb_eInvalidByteSequence): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-12 07:20:10 +00:00
akr 94ca2d94de * transcode_data.h (rb_transcoder): add resetstate_func field for
resetting a state of stateful encoding.

* enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify
  finish_eucjp_to_iso2022jp for resetstate_func.

* tool/transcode-tblgen.rb: specify NULL for resetstate_func.

* transcode.c (output_replacement_character): call resetstate_func
  before appending the replacement character.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 22:44:23 +00:00
akr c1361132bc * transcode_data.h (rb_transcoder): typedef at first.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18501 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 22:00:48 +00:00
akr a2901a7c75 * transcode_data.h (rb_trans_result_t): new type.
(rb_trans_elem_t): new type.
  (rb_trans_t): new type.

* transcode.c (transcode_dispatch_cb): removed.
  (transcode_dispatch): removed.
  (rb_transcoding_result_t): moved to rb_trans_result_t in
  transcode_data.h.
  (transcode_restartable0): goto follow_info when FUNsi.
  (rb_transcoding_open): use get_transcoder_entry.
  (rb_trans_open): new function.
  (rb_trans_conv): ditto.
  (rb_trans_close): ditto.
  (trans_open_i): ditto.
  (trans_sweep): ditto.
  (more_output_buffer): take rb_trans_t instead of rb_transcoding as
  an argument.
  (transcode_loop): take from_encoding and to_encoding instead of tr
  as arguments.  use rb_trans_open/rb_trans_conv/rb_trans_close.
  (str_transcode): don't use transcode_dispatch.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 15:50:42 +00:00
akr bd5bf3726c * transcode_data.h (rb_transcoding): rename fields.
readlen -> recognized_len.
  feedlen -> readagain_len.

* transcode.c: follow the field renaming.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18481 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-11 01:06:21 +00:00
akr 436a136b4e * transcode_data.h (rb_transcoding): new field: flags.
* transcode.c (load_transcoder): extracted from transcode_dispatch_cb.
  (rb_transcoding_result_t): renamed from transcode_result_t.
  (rb_transcoding_open): new function.
  (rb_transcoding_convert): ditto.
  (rb_transcoding_close): ditto.
  (transcode_loop): use rb_transcoding_open, rb_transcoding_convert
  and rb_transcoding_close.
  (str_transcode): don't need rb_transcoding.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 12:26:01 +00:00
akr e748ed513d * transcode_data.h (rb_transcoding): add feedlen field.
* transcode.c (transcode_restartable0): renamed from
  transcode_restartable.
  save input buffer into feed buffer if next character is started the
  point before input buffer.  for example, "\x00\xd8\x01" then "\x02"
  in UTF-16LE.  \x02 causes invalid and next character is started from
  \x01.
  (transcode_restartable): new function to call
  transcode_restartable0.   if feed buffer is not empty, convert it at
  first.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-10 02:17:56 +00:00
akr 0e3d10173b * transcode.c (transcode_loop): take destination and resize function
as parameters.
  (more_output_buffer): ditto.
  (str_transcoding_resize): argument changed from rb_transcoding* to
  VALUE.
  (str_transcode): call transcode_loop with destination string and its
  resize function.

* transcode_data.h (rb_transcoding): move ruby_string_dest and
  flush_func to transcode_loop parameters.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 13:34:21 +00:00
akr 4bceda1125 * transcode_data.h (rb_transcoding): new fields: next_info and next_byte.
* transcode.c (transcode_restartable): save/restore next_info and
  next_byte.
  sync readlen and in_p when invalid.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18454 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 07:35:28 +00:00
akr 139234e1a0 * transcode_data.h (rb_transcoding): add fields for restartable
transcoding.
  (rb_transcoder): add max_input field.
  from_unit_length field is renamed to input_unit_length.

* tool/transcode-tblgen.rb: generate max_input field.

* enc/trans/iso2022.erb.c: follow rb_transcoder change.

* enc/trans/utf_16_32.erb.c: ditto.

* transcode.c (PARTIAL_INPUT): new constant.
  (transcode_char_start): new function.
  (transcode_result_t): new type.
  (transcode_restartable): new function.
  (more_output_buffer): new function.
  (transcode_loop): use transcode_restartable.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18452 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-09 06:02:01 +00:00
akr 2833d9f95d * transcode_data.h (rb_transcoder): from_unit_length field added.
from_utf8 field removed.

* tool/transcode-tblgen.rb: generate offsets range.
  follow rb_transcoder change.

* transcode.c (transcode_loop): don't use from_utf8.
  make invalid region from_unit_length wise.

* enc/trans/iso2022.erb.c: follow rb_transcoder and 
  transcode_generate_node change.

* enc/trans/utf_16_32.erb.c: follow rb_transcoder and
  transcode_generate_node change.
  explicit :invalid map removed.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18445 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-08 15:48:17 +00:00
akr 1504652373 * transcode_data.h (rb_transcoding): new field "stateful".
(rb_transcoder): preprocessor and postprocessor field removed.
  change arguments of func_ii, func_si, func_io and func_so.
  new field "finish_func".

* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
  generatable.

* transcode.c (transcoder_lib_table): removed.
  (transcoder_table): change structure.
  (transcoder_key): removed because the above structure change.
  (make_transcoder_entry): new function.
  (get_transcoder_entry): ditto.
  (rb_register_transcoder): follow the structure change.
  (declare_transcoder): ditto.
  (transcode_search_path): new function for breadth first search to
  find a list of converters.
  (transcode_search_path_i): new function.
  (transcode_dispatch_cb): ditto.
  (transcode_dispatch): use transcode_search_path.
  (transcode_loop): follow the argument change.
  (str_transcode): preprocessor and postprocessor stuff removed.

* enc/trans/iso2022.erb.c: new file.  ISO-2022-JP conversion
  re-implemented.

* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.

nute(23:52:53)% head -40 ChangeLog
Thu Aug  7 23:43:11 2008  Tanaka Akira  <akr@fsij.org>

* transcode_data.h (rb_transcoding): new field "stateful".
  (rb_transcoder): preprocessor and postprocessor field removed.
  change arguments of func_ii, func_si, func_io and func_so.
  new field "finish_func".

* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
  generatable.

* transcode.c (transcoder_lib_table): removed.
  (transcoder_table): change structure.
  (transcoder_key): removed because the above structure change.
  (make_transcoder_entry): new function.
  (get_transcoder_entry): ditto.
  (rb_register_transcoder): follow the structure change.
  (declare_transcoder): ditto.
  (transcode_search_path): new function for breadth first search to
  find a list of converters.
  (transcode_search_path_i): new function.
  (transcode_dispatch_cb): ditto.
  (transcode_dispatch): use transcode_search_path.
  (transcode_loop): follow the argument change.
  (str_transcode): preprocessor and postprocessor stuff removed.

* enc/trans/iso2022.erb.c: new file.  ISO-2022-JP conversion
  re-implemented.

* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.

* enc/trans/utf_16_32.erb.c: follow argument change of FUNso.

[ruby-dev:35798]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-07 14:53:30 +00:00
nobu b84d31c524 * transcode_data.h (TRANSCODE_ERROR): common transcode failure
exception, would be changed later.

* enc/trans/japanese.c (UNSUPPORTED_MODE): unsupported mode transition
  exception.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-05 03:34:52 +00:00
nobu 94ed51b281 * transcode.c (transcode_loop): constified.
* transcode.c (str_transcode): rb_str_set_len() sets a delimiter.

* transcode_data.h (rb_transcoder): constified preprocessor and
  postprocessor input.

* enc/trans/japanese.c: ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18066 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-07-14 09:47:33 +00:00
nobu f5eb90f3c2 * transcode.c (str_transcode): initialize transcoder in
rb_transcoding.  [ruby-dev:33234]

* transcode_data.h (rb_transcoding): transcoder constified.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15152 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 05:32:12 +00:00
nobu 463af63468 * transcode.c (transcode_loop, str_transcoding_resize): use unsigned
char.  [ruby-dev:33232]

* transcode_data.h (rb_transcoding, rb_transcoder): removed callback
  parameters.

* enc/trans/japanese.c: ditto.

* enc/trans/utf_16_32.c: parenthesized bit-or operands.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15150 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-21 03:35:05 +00:00
nobu a8969e999a * transcode.c (transcode_dispatch): constified return value.
* transcode_data.h (rb_transcoding): include pointer to rb_transcoder
  and auxiliary data.

* transcode_data.h (rb_transcoder): all callback functions shoud have
  their own parameters.

* enc/trans/{japanese,single_byte}.c: constified.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-20 21:40:08 +00:00
duerst 3d0c7bea4d Sun Jan 20 15:08:08 2008 Martin Duerst <duerst@it.aoyama.ac.jp>
* enc/trans/utf_16_32.c: new file, currently implementing
	  UTF-16BE conversions only.

	* test/ruby/test_transcode.rb: Added tests for UTF-16BE;
	  made check_both_ways() use force_encoding differently.

	* transcode_data.h, transcode.c: Support for more conversion
	  functions.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-20 06:12:48 +00:00
akr 6cdef2dc7e * $Date$ keyword removed to avoid inclusion of locale dependent
string.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14912 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-01-06 15:49:38 +00:00
duerst 793e9423cd Fri Dec 28 01:55:04 2007 Martin Duerst <duerst@it.aoyama.ac.jp>
* transcode.c (transcode_dispatch): reverted some of the changes
          in r14746.

	* transcode.c, enc/trans/single_byte.c: Added conversions to/from
	  US-ASCII and ASCII-8BIT (using data tables).

	* enc/trans/single_byte.c: Some spacing/ordering changes due to
	  automatic data file generation.

	* transcode_data.h, transcode.c: Preliminary code for using
	  micro-conversion functions.

	* test/ruby/test_transcode.rb: Added some tests for US-ASCII and
	  ASCII-8BIT conversions.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-28 09:26:55 +00:00
akr efd7504d44 * parse.y, transcode_data.h, transcode.c: change "illegal" to
"invalid" in a context which doesn' t against a law.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14735 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-27 08:27:19 +00:00
nobu b7db9036be * common.mk (COMMONOBJS): transcode_data_*.c moved under enc/trans.
* transcode_data.h (rb_transcoding, rb_transcoder): prefixed.

* transcode.c (rb_register_transcoder, rb_declare_transcoder): split
  declaration and registration.  [ruby-dev:32704]

* transcode.c (transcode_dispatch): autoload pre-declared transcoder.

* transcode.c (str_transcode): use rb_define_dummy_encoding().

* transcode.c (Init_transcode): initialize transcoder tables.

* enc/trans/single_byte.c, enc/trans/japanese.c: moved from top.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14666 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2007-12-25 05:57:04 +00:00