Граф коммитов

153 Коммитов

Автор SHA1 Сообщение Дата
卜部昌平 6dd60cf114 st_foreach now free from ANYARGS
After 5e86b005c0, I now think ANYARGS is
dangerous and should be extinct.  This commit deletes ANYARGS from
st_foreach.  I strongly believe that this commit should have had come
with b0af0592fd, which added extra
parameter to st_foreach callbacks.
2019-08-27 15:52:26 +09:00
Nobuyoshi Nakada 2f6cc15cdb
Fixed String#grapheme_clusters with wide encodings
* string.c (get_reg_grapheme_cluster): make regexp from properly
  encoded sources fro wide-char encodings.  [Bug #15965]

* regparse.c (node_extended_grapheme_cluster): suppress false
  duplicated range warning for the time being.
2019-06-29 10:10:17 +09:00
duerst 9bb2da28d5 convert check for array length to assertion and comment out
In regparse.c, in function node_extended_grapheme_cluster,
we used a raw if() with exit(1) as a cross-check for our
length calculations for the common node array. Convert this
to an assertion and comment it out because it is not needed
for active code.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-07 07:20:26 +00:00
duerst 7780553c30 remove code duplication and put everything into forward order
In file regparse.c, in function node_extended_grapheme_cluster(),
eliminate code duplication of CRLF and '.' (any character). This
uses the fact that both for Unicode encodings and for non-Unicode
encodings, the first alternative is CRLF, and the last alternative
is '.' (any character). This puts all of the pieces into forward
order (the order of the code follows the order of the syntax
definition).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-07 07:04:00 +00:00
duerst 9d17024095 remove an unused variable
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66240 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-06 09:54:09 +00:00
duerst c114e275c9 make sure all nodes are freed on error in node_extended_grapheme_cluster()
regparse.c: In function node_extended_grapheme_cluster(), use function-global
array node_common and use it for list and alternate construction. This is done
so that in case of error, all nodes that have already been constructed can be
correctly freed in a single for loop. Document the layout structure.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66239 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-06 09:16:43 +00:00
duerst 456a696f72 remove code duplication and streamline identifiers
In regparse.c:

* Reduce coode duplication by merging the almost identical functions
  create_sequence_node and create_alternate_node into a new function
  create_node_from_array, adding a parameter that distinguishes  between
  creating a list and creating an alternative.

* Streamline variable/function naming. Unicode UAX #29 uses 'sequence', but
  the regular expression library uses 'list' for the same concept. Keep
  'sequence' in the ccmments that are taken from UAX #29, but use 'list'
  in variable names.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66234 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-06 04:46:57 +00:00
duerst e824e21beb remove obsolete data from unicode.c
* unicode.c: Remove the arrays onigenc_unicode_GCB_ranges_GAZ,
  onigenc_unicode_GCB_ranges_E_Base, and onigenc_unicode_GCB_ranges_Emoji,
  because they are not needed anymore for Unicode 11.0.0.

* regparse.c: Remove external declarations for above arrays.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66232 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-06 00:05:08 +00:00
duerst 6cf3ada55c remove unused variables in node_extended_grapheme_cluster()
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66218 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-05 10:40:17 +00:00
duerst 6e97c12e49 tweak/remove comments [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66217 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-05 10:11:57 +00:00
duerst 3913092624 adjust some comments in node_extended_grapheme_cluster() [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-05 09:00:40 +00:00
duerst 66a6073859 update to Unicode 11.0.0 (main step, not complete yet)
- common.mk: Change Unicode version to 11.0.0, and Emoji version to 11.0
- test/ruby/enc/test_emoji_breaks.rb: update hard-coded Emoji version
- enc/unicode/11.0.0, enc/unicode/11.0.0/casefold.h, enc/unicode/name2ctype.h:
  Add generated files. Files for Unicode 10.0.0 will be removed once we are
  sure 11.0.0 works.
- lib/unicode_normalize/tables.rb: Updated table.
- regparse.c: Almost completely reimplement grapheme cluster detection in
  function node_extended_grapheme_cluster().


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-05 08:10:24 +00:00
duerst b56e266d64 remove unnecessary settings with NULL_NODE in \X implementation
Remove unnecessary settings of node_array elements to NULL_NODE.
We can do this because we initialize the whole array to NULL_NODEs
and set everything again to NULL_NODEs when creating a sequence or
alternative node.

Also, fix an index error in the initialization of node_array.
(issue #15343)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66139 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 23:28:42 +00:00
duerst 4a8531db5d fix order of declarations and code at start of node_extended_grapheme_cluster()
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66138 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 22:05:36 +00:00
ko1 d80bf2f1e7 fix last commit (r66135)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66137 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 22:01:29 +00:00
duerst f43a2a5a49 make sure all nodes are freed on error in node_extended_grapheme_cluster()
regparse.c: In function node_extended_grapheme_cluster(), introduce function-global
array node_array and use it for sequence and alternate construction. This is done
so that in case of error, all nodes that have already been constructed can be
correctly freed. (issue #15343)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66135 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 21:41:50 +00:00
duerst 1752d13827 expand a small comment [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66132 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 10:00:35 +00:00
duerst 096d362939 add/change some comments in node_extended_grapheme_cluster() [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66123 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 01:33:24 +00:00
duerst d359ee3f55 reformat code [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66122 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-02 01:06:28 +00:00
duerst d1f8694171 remove unnecessary code removing CR/LF from range
Remove code that tries to remove CR and LF from Grapheme_Cluster_Break=Control.
This code is unnecessary because Grapheme_Cluster_Break=Control already excludes
CR and LF.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66116 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 08:26:39 +00:00
svn 36b3e41a57 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66115 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 07:31:35 +00:00
duerst 3feeed6e99 introduce and use create_alternate_node()
Introduce new function create_alternate_node() to create an alternative node
from a list of nodes in one go. Use it once (two more uses expected).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66114 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 07:31:34 +00:00
duerst 1fa7087f10 eliminate a list with only one element
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66113 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 06:07:53 +00:00
duerst c80aeb527e remove two unnecessary variables (np2 and np3)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 07:07:59 +00:00
duerst d3b7a10dcc eliminate intermediate variable in very short block (3 times)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 06:43:57 +00:00
duerst 33c7fa6501 use create_sequence_node() four more times
Four more use of create_sequence_node() in node_extended_grapheme_cluster
(a few more to come).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66070 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 06:21:52 +00:00
duerst b4e39021f2 use create_sequence_node() once more
One more use of create_sequence_node() in node_extended_grapheme_cluster
(several more to come).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66063 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 01:58:35 +00:00
duerst 97a8334cd3 introduce macro R_ERR to reduce repetitive code
Introduce a new preprocessor macro R_ERR to visually reduce repetitive code
checking for return values and going to the err: label at the end of the
function node_extended_grapheme_cluster().

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66057 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 00:53:35 +00:00
duerst 42cb4feda1 reduce number of arguments on quantify_property_node()
There are only four patterns of the last two arguments to quantify_property_node().
By replacing the lower/upper arguments with a single char, we get more expressive
calls, the last argument directly corresponding to the quantifier that we want to
use (except for '2', which means exactly two).

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66052 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-28 00:28:52 +00:00
duerst 7c4a422d83 fix order of subexpressions for Hangul
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66048 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:50:27 +00:00
svn 793c48b4ce * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66047 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:38:55 +00:00
duerst 06bd42a722 introduce two more uses of create_sequence_node in node_extended_grapheme_cluster
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:38:53 +00:00
duerst 99d451f5c5 correctly handle return value from create_sequence_node()
In function node_extended_grapheme_cluster(), store and test
return value from create_sequence_node(). Never forget this!

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66045 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:23:11 +00:00
svn 87cda89ca7 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66044 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:12:31 +00:00
duerst 8186889120 declare array for sequence at start of code creating sequence
In function node_extended_grapheme_cluster(),
move declaration up so that block encompasses all of the regular expression
creation that finally makes up the sequence. Having blocks like this will
be great because it directly shows the extent of code belonging to each
subexpression of the regular expression being created.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:12:29 +00:00
duerst 7086aae378 make sure all nodes are correctly freed in create_property_node()
We make sure that the newly created tree and all remaining nodes passed in
in the node_array are freed.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 21:00:06 +00:00
k0kubun c3fe307808 regparse.c: conform C90
../regparse.c:5908:28: error: initializer for aggregate is not a compile-time constant [-Werror,-Wc99-extensions]
      Node* sequence[] = { np1, np2, np3, ((Node* )0) };
                           ^~~

https://travis-ci.org/ruby/ruby/jobs/460197620

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66034 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 11:24:19 +00:00
duerst 9e2455b756 introduce helper function create_sequence_node()
The new function create_sequence_node() uses its second argument
(an array of Node*, from left to right, ending with NULL_NODE)
to create a sequence of expressions using node_new_list().

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66033 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 11:04:02 +00:00
svn 231930ca7d * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 10:15:46 +00:00
duerst 8f9c00d207 introduce helper function quantify_property_node()
The new function quantify_property_node() combines the functions
create_property_node() and quantify_node(), which frequently appear together.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66031 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 10:15:45 +00:00
duerst 69443998cd introduce helper function quantify_node() to wrap function node_new_quantifier
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66030 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 09:44:35 +00:00
duerst aa44935969 use explicit property name when creating nodes for "Grapheme_Cluster_Break=Extend"
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66021 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 04:13:21 +00:00
duerst b62e466fb5 use 'Regional_Indicator' script property instead of fixed constants
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66020 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 03:56:19 +00:00
duerst 2e07575914 add some comments in function node_extended_grapheme_cluster() [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66014 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-27 02:18:29 +00:00
duerst 9a4d120736 create function create_property_node to extract recurring functionality
Refactoring: In regparse.c, extract creation of a new CClass node and
initialization using a property into a new function create_property_node().


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65972 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-11-25 11:02:41 +00:00
nobu 6e9fc98d19 regparse.c: check the result of propname2ctype
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65094 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16 08:13:58 +00:00
nobu e07c5baf66 unicode.c: moved addtional GCB ranges
* enc/unicode.c: moved additional Grapheme Cluster Break ranges
  which depend on the Unicode version.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-15 13:48:20 +00:00
nobu 179045acaf regparse.c: Suppress duplicated range warning by mere \X
* regparse.c (node_extended_grapheme_cluster): as Unicode 10 has
  added Grapheme_Cluster_Break properties to some characters,
  remove duplicated ranges for Unicode 9.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-15 12:31:25 +00:00
nobu f31c5e72b2 regparse.c: warn all duplicated ranges when debugging
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-15 12:31:22 +00:00
hsbt ef9bc609db Fix typos.
* rememberd -> remembered
  * refered -> referred

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61933 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-18 11:44:10 +00:00