The CLDR plural data uses a different structure for marking locales
which was not picked up by the previous code.
More specifically, for most types of data, data is split in a
different file per locale. This gets picked up by the cldr package.
The plural data, instead, has collections of rules and then lists the
locales for which the rules apply. This is a special case within
CLDR (albeit a very handy one) and thus needs to be handled separately.
This CL adds these locales to compact base languages and compact
language tags.
Change-Id: I1654501024f4d978361431594c72ee32d8c3ab01
Reviewed-on: https://go-review.googlesource.com/19337
Reviewed-by: Nigel Tao <nigeltao@golang.org>
cldr:
- needed to excluded "validity" directory as it contains xml data
not specific to locales.
language and internal:
- CompactIndex now supports *all* locales for which there are data
files and not just the ones which have data. This is necessary to
not drop en-US. Seems better anyway. This increases table size, but
added comment about a trick that could be used to reduce it again in
the future.
- Skipping more match data entries. My suspicion is that this is
supported anyway, but needs to be verified (TODO added).
language/display:
- manually removed support for "private use" script names are their
coverage is spotty and questionable.
- manually override coverage test for az_Arab: not supported by
English, but seems to have good data for languages for which this
may be relevant.
currency:
- minor changes in test code and generation code to adapt to
change data.
- manually added MVP as currency as it doesn't appear to be added
otherwise.
Note that search and collate are still stuck on old revisions.
Change-Id: I2b2b2cba66caaff7c027fcb421f99595430e31dd
Reviewed-on: https://go-review.googlesource.com/17802
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
The cldr package is quite low-level and probably would only for
internal use if it hadn't predated the internal directory convention.
Added doc.go in unicode directory to clarify nature of packages in
this subdirectory.
After this, the package that remains to be moved to unicode is bidi.
Change-Id: Ibff3285ca8cc00980e48fd30754c3f7abc87b2a0
Reviewed-on: https://go-review.googlesource.com/17491
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Simpler map shaves off about 40k of the binary size even though
table entry size is reduced by only 1k.
This is about 10% of what package langauge adds to the imported
packages.
Change-Id: I479a7de13587b77cce5e5db326c469834de071cd
Reviewed-on: https://go-review.googlesource.com/16626
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
Use CLDR variant. Collate already used this. This provides
a little bit more structure and plays nice with collation.
Note that maketables.go contained a bug that happened to work
given the current data in CLDR.
Change-Id: I40d07d7cd2a8615bbe0e223a074df5b701b7b833
Reviewed-on: https://go-review.googlesource.com/14805
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Compensate for the fact that "und" was already taken out.
Also changed the checksum computation to reflect this change.
Change-Id: I2b29df2a41cd186d886628390a55bd19d0175075
Reviewed-on: https://go-review.googlesource.com/14750
Reviewed-by: Nigel Tao <nigeltao@golang.org>
This function provides an alternative to complex indexes as used in
package display, for example. Many of the upcoming packages will need
to use something like this, so this will be useful and simplify
implementations.
The upcoming localization code will also likely need to do a lot of
table lookups based on tags. CompactIndex allows implementations to
easily associate data based on a simple table lookup. Standardizing
on the index among the packages also allows caching it.
Maybe this should have been internal, but as the code and data are
very interwoven with the types in the language package, I don't see
a trivial way of doing this. The construct may be useful to users
that want to implement their own formatters.
Change-Id: I0c62e825be09d19b85e684fa114aa2cb6bec5003
Reviewed-on: https://go-review.googlesource.com/14438
Reviewed-by: Nigel Tao <nigeltao@golang.org>
Marcel van Lohuizen