gecko-dev/intl/unicharutil/tools/format.txt

#
# $Id: format.txt,v 1.1 1999/01/08 00:19:20 ftang%netscape.com Exp $
#

CHARACTER DATA
==============

This package generates some data files that contain character properties useful
for text processing.

CHARACTER PROPERTIES
====================

The first data file is called "ctype.dat" and contains a compressed form of
the character properties found in the Unicode Character Database (UCDB).
Additional properties can be specified in limited UCDB format in another file
to avoid modifying the original UCDB.

The following is a property name and code table to be used with the character
data:

NAME CODE DESCRIPTION
---------------------
Mn   0    Mark, Non-Spacing
Mc   1    Mark, Spacing Combining
Me   2    Mark, Enclosing
Nd   3    Number, Decimal Digit
Nl   4    Number, Letter
No   5    Number, Other
Zs   6    Separator, Space
Zl   7    Separator, Line
Zp   8    Separator, Paragraph
Cc   9    Other, Control
Cf   10   Other, Format
Cs   11   Other, Surrogate
Co   12   Other, Private Use
Cn   13   Other, Not Assigned
Lu   14   Letter, Uppercase
Ll   15   Letter, Lowercase
Lt   16   Letter, Titlecase
Lm   17   Letter, Modifier
Lo   18   Letter, Other
Pc   19   Punctuation, Connector
Pd   20   Punctuation, Dash
Ps   21   Punctuation, Open
Pe   22   Punctuation, Close
Po   23   Punctuation, Other
Sm   24   Symbol, Math
Sc   25   Symbol, Currency
Sk   26   Symbol, Modifier
So   27   Symbol, Other
L    28   Left-To-Right
R    29   Right-To-Left
EN   30   European Number
ES   31   European Number Separator
ET   32   European Number Terminator
AN   33   Arabic Number
CS   34   Common Number Separator
B    35   Block Separator
S    36   Segment Separator
WS   37   Whitespace
ON   38   Other Neutrals
Pi   47   Punctuation, Initial
Pf   48   Punctuation, Final
#
# Implementation specific properties.
#
Cm   39   Composite
Nb   40   Non-Breaking
Sy   41   Symmetric (characters which are part of open/close pairs)
Hd   42   Hex Digit
Qm   43   Quote Mark
Mr   44   Mirroring
Ss   45   Space, Other (controls viewed as spaces in ctype isspace())
Cp   46   Defined character

The actual binary data is formatted as follows:

  Assumptions: unsigned short is at least 16-bits in size and unsigned long
               is at least 32-bits in size.

    unsigned short ByteOrderMark
    unsigned short OffsetArraySize
    unsigned long  Bytes
    unsigned short Offsets[OffsetArraySize + 1]
    unsigned long  Ranges[N], N = value of Offsets[OffsetArraySize]

  The Bytes field provides the total byte count used for the Offsets[] and
  Ranges[] arrays.  The Offsets[] array is aligned on a 4-byte boundary and
  there is always one extra node on the end to hold the final index of the
  Ranges[] array.  The Ranges[] array contains pairs of 4-byte values
  representing a range of Unicode characters.  The pairs are arranged in
  increasing order by the first character code in the range.

  Determining if a particular character is in the property list requires a
  simple binary search to determine if a character is in any of the ranges
  for the property.

  If the ByteOrderMark is equal to 0xFFFE, then the data was generated on a
  machine with a different endian order and the values must be byte-swapped.

  To swap a 16-bit value:
     c = (c >> 8) | ((c & 0xff) << 8)

  To swap a 32-bit value:
     c = ((c & 0xff) << 24) | (((c >> 8) & 0xff) << 16) |
         (((c >> 16) & 0xff) << 8) | (c >> 24)

CASE MAPPINGS
=============

The next data file is called "case.dat" and contains three case mapping tables
in the following order: upper, lower, and title case.  Each table is in
increasing order by character code and each mapping contains 3 unsigned longs
which represent the possible mappings.

The format for the binary form of these tables is:

  unsigned short ByteOrderMark
  unsigned short NumMappingNodes, count of all mapping nodes
  unsigned short CaseTableSizes[2], upper and lower mapping node counts
  unsigned long  CaseTables[NumMappingNodes]

  The starting indexes of the case tables are calculated as following:

    UpperIndex = 0;
    LowerIndex = CaseTableSizes[0] * 3;
    TitleIndex = LowerIndex + CaseTableSizes[1] * 3;

  The order of the fields for the three tables are:

    Upper case
    ----------
    unsigned long upper;
    unsigned long lower;
    unsigned long title;

    Lower case
    ----------
    unsigned long lower;
    unsigned long upper;
    unsigned long title;

    Title case
    ----------
    unsigned long title;
    unsigned long upper;
    unsigned long lower;

  If the ByteOrderMark is equal to 0xFFFE, endian swapping is required in the
  same way as described in the CHARACTER PROPERTIES section.

  Because the tables are in increasing order by character code, locating a
  mapping requires a simple binary search on one of the 3 codes that make up
  each node.

  It is important to note that there can only be 65536 mapping nodes which
  divided into 3 portions allows 21845 nodes for each case mapping table.  The
  distribution of mappings may be more or less than 21845 per table, but only
  65536 are allowed.

DECOMPOSITIONS
==============

The next data file is called "decomp.dat" and contains the decomposition data
for all characters with decompositions containing more than one character and
are *not* compatibility decompositions.  Compatibility decompositions are
signaled in the UCDB format by the use of the <compat> tag in the
decomposition field.  Each list of character codes represents a full
decomposition of a composite character.  The nodes are arranged in increasing
order by character code.

The format for the binary form of this table is:

  unsigned short ByteOrderMark
  unsigned short NumDecompNodes, count of all decomposition nodes
  unsigned long  Bytes
  unsigned long  DecompNodes[(NumDecompNodes * 2) + 1]
  unsigned long  Decomp[N], N = sum of all counts in DecompNodes[]

  If the ByteOrderMark is equal to 0xFFFE, endian swapping is required in the
  same way as described in the CHARACTER PROPERTIES section.

  The DecompNodes[] array consists of pairs of unsigned longs, the first of
  which is the character code and the second is the initial index of the list
  of character codes representing the decomposition.

  Locating the decomposition of a composite character requires a binary search
  for a character code in the DecompNodes[] array and using its index to
  locate the start of the decomposition.  The length of the decomposition list
  is the index in the following element in DecompNode[] minus the current
  index.

COMBINING CLASSES
=================

The fourth data file is called "cmbcl.dat" and contains the characters with
non-zero combining classes.

The format for the binary form of this table is:

  unsigned short ByteOrderMark
  unsigned short NumCCLNodes
  unsigned long  Bytes
  unsigned long  CCLNodes[NumCCLNodes * 3]

  If the ByteOrderMark is equal to 0xFFFE, endian swapping is required in the
  same way as described in the CHARACTER PROPERTIES section.

  The CCLNodes[] array consists of groups of three unsigned longs.  The first
  and second are the beginning and ending of a range and the third is the
  combining class of that range.

  If a character is not found in this table, then the combining class is
  assumed to be 0.

  It is important to note that only 65536 distinct ranges plus combining class
  can be specified because the NumCCLNodes is usually a 16-bit number.

NUMBER TABLE
============

The final data file is called "num.dat" and contains the characters that have
a numeric value associated with them.

The format for the binary form of the table is:

  unsigned short ByteOrderMark
  unsigned short NumNumberNodes
  unsigned long  Bytes
  unsigned long  NumberNodes[NumNumberNodes]
  unsigned short ValueNodes[(Bytes - (NumNumberNodes * sizeof(unsigned long)))
                            / sizeof(short)]

  If the ByteOrderMark is equal to 0xFFFE, endian swapping is required in the
  same way as described in the CHARACTER PROPERTIES section.

  The NumberNodes array contains pairs of values, the first of which is the
  character code and the second an index into the ValueNodes array.  The
  ValueNodes array contains pairs of integers which represent the numerator
  and denominator of the numeric value of the character.  If the character
  happens to map to an integer, both the values in ValueNodes will be the
  same.