freetype2/src/gxvalid

gxvalid: TrueType GX validator
==============================


1. What is this
---------------

  `gxvalid' is a module to  validate TrueType GX tables: a collection of
  additional tables  in TrueType  font which are  used by  `QuickDraw GX
  Text',  Apple Advanced  Typography  (AAT).  In  addition, gxvalid  can
  validates `kern'  tables which have  been extended for AAT.   Like the
  otvalid  module,   gxvalid  uses  Freetype   2's  validator  framework
  (ftvalid).

  You can link gxvalid with your program; before running your own layout
  engine, gxvalid validates a font  file.  As the result, you can remove
  error-checking code  from the layout  engine.  It is also  possible to
  use  gxvalid  as a  stand-alone  font  validator;  the `ftvalid'  test
  program  included  in the  ft2demo  bundle  calls gxvalid  internally.
  A stand-alone font validator may be useful for font developers.

  This documents documents the following issues.

  - supported TrueType GX tables
  - fundamental validation limitations
  - permissive error handling of broken GX tables
  - `kern' table issue.


2. Supported tables
-------------------

  The following GX tables are currently supported.

    bsln
    feat
    just
    kern(*)
    lcar
    mort
    morx
    opbd
    prop
    trak

  The following GX tables are currently unsupported.

    cvar
    fdsc
    fmtx
    fvar
    gvar
    Zapf

  The following GX tables won't be supported.

    acnt(**)
    hsty(***)

  The following undocumented tables in TrueType fonts designed for Apple
  platform aren't handled either.

    addg
    CVTM
    TPNM
    umif


  *)   The `kern'  validator handles both  the classic and the  new kern
       formats;  the former  is supported  on both  Microsoft  and Apple
       platforms, while the latter is supported on Apple platforms.

  **)  `acnt' tables are not supported by currently available Apple font
       tools.

  ***) There  is  one more  Apple  extension,  `hsty',  but  it  is  for
       Newton-OS, not GX  (Newton-OS is a platform by  Apple, but it can
       use  sfnt- housed bitmap  fonts only).   Therefore, it  should be
       excluded  from  `Apple  platform'  in the  context  of  TrueType.
       gxvalid ignores it as Apple font tools do so.


  We have  checked 183  fonts bundled with  MacOS 9.1, MacOS  9.2, MacOS
  10.0, MacOS X 10.1, MSIE  for MacOS, and AppleWorks 6.0.  In addition,
  we have  checked 67 Dynalab fonts  (designed for MacOS)  and 189 Ricoh
  fonts (designed for Windows and  MacOS dual platforms).  The number of
  fonts including TrueType GX tables are as follows.

    bsln:  76
    feat: 191
    just:  84
    kern:  59
    lcar:   4
    mort: 326
    morx:  19
    opbd:   4
    prop: 114
    trak:  16

  Dynalab  and Ricoh fonts  don't have  GX tables  except of  `feat' and
  `mort'.


3. Fundamental validation limitations
-------------------------------------

  TrueType  GX  provides  layout   information  to  libraries  for  font
  rasterizers  and text layout.   gxvalid can  check whether  the layout
  data in  a font is conformant  to the TrueType GX  format specified by
  Apple.  But gxvalid cannot check  a how QuickDraw GX/AAT renderer uses
  the stored information.

  3-1. Validation of State Machine activity
  -----------------------------------------

    QuickDraw GX/AAT uses a `State Machine' to provide `stateful' layout
    features,  and TrueType GX  stores the  state transition  diagram of
    this `State  Machine' in a  `StateTable' data structure.   While the
    State  Machine receives  a series  of glyph  IDs, the  State Machine
    starts with `start  of text' state, walks around  various states and
    generates various  layout informations to the  renderer, and finally
    reaches the `end of text' state.

    gxvalid can check essential errors like:

      - possibility of state transitions to undefined states
      - existence of glyph  IDs that the State Machine  doesn't know how
        to handle
      - the  State Machine  cannot compute  the layout  information from
        given diagram

    These errors  can be  checked within finite  steps, and  without the
    State Machine itself, because these are `expression' errors of state
    transition diagram.

    There  is no  limitation  about  how long  the  State Machine  walks
    around,  so validation  of  the algorithm  in  the state  transition
    diagram requires infinite  steps, even if we had  a State Machine in
    gxvalid.   Therefore, the  following errors  and problems  cannot be
    checked.

      - existence of states which the State Machine never transits to
      - the  possibility that the  State Machine  never reaches  `end of
        text'
      - the possibility of stack underflow/overflow in the State Machine
        (in  ligature  and  contextual  glyph substitutions,  the  State
        Machine can store 16 glyphs onto its stack)

    In addition, gxvalid doesn't check `temporary glyph IDs' used in the
    chained State Machines  (in `mort' and `morx' tables).   If a layout
    feature  is  implemented by  a  single  State  Machine, a  glyph  ID
    converted by the State Machine is passed to the glyph renderer, thus
    it  should not  point to  an undefined  glyph ID.   But if  a layout
    feature is implemented by  chained State Machines, a component State
    Machine  (if it  is  not the  final  one) is  permitted to  generate
    undefined glyph IDs for temporary use, because it is handled by next
    component State Machine and not  by the glyph renderer.  To validate
    such temporary glyph IDs, gxvalid must stack all undefined glyph IDs
    which  can occur in  the output  of the  previous State  Machine and
    search  them in  the  `ClassTable' structure  of  the current  State
    Machine.  It is too complex to  list all possible glyph IDs from the
    StateTable, especially from a ligature substitution table.

  3-2. Validation of relationship between multiple layout features
  ----------------------------------------------------------------

    gxvalid does  not validate the relationship  between multiple layout
    features at all.

    If  multiple layout  features  are defined  in  TrueType GX  tables,
    possible  interactions,  overrides,  and  conflicts  between  layout
    features are implicitly  given in the font too.   For example, there
    are several predefined spacing control features:

      - Text Spacing          (Proportional/Monospace/Half-width/Normal)
      - Number Spacing        (Monospaced-numbers/Proportional-numbers)
      - Kana Spacing          (Full-width/Proportional)
      - Ideographic Spacing   (Full-width/Proportional)
      - CJK Roman Spacing     (Half-width/Proportional/Default-roman
                               /Full-width-roman/Proportional)

    If all  layout features are  independently managed, we  can activate
    inconsistent  typographic rules  like  `Text Spacing=Monospace'  and
    `Ideographic Spacing=Proportional' at the same time.

    The combinations  of layout features  is managed by a  32bit integer
    (one bit each for selector  setting), so we can define relationships
    between  up  to 32  features,  theoretically.   But  if one  feature
    setting  affects  another   feature  setting,  we  need  typographic
    priority  rules to  validate the  relationship.   Unfortunately, the
    TrueType GX format specification does not give such information even
    for predefined features.


4. Permissive error handling of broken GX tables
------------------------------------------------

  When  Apple's font  rendering system  finds an  inconsistency,  like a
  specification  violation or  an  unspecified value  in  a TrueType  GX
  table, it does not always  return error.  In most cases, the rendering
  engine silently  ignores such wrong  values or even whole  tables.  In
  fact, MacOS is shipped with  fonts including broken GX/AAT tables, but
  no harmful  effects due to  `officially broken' fonts are  observed by
  end-users.

  gxvalid  is designed  to continue  the validation  process as  long as
  possible.  When gxvalid find wrong  values, gxvalid warns it at least,
  and takes  a fallback procedure  if possible.  The  fallback procedure
  depends on the debug level.

  We used the following three tools to investigate Apple's error handling.

    - FontValidator  (for MacOS 8.5 - 9.2)  resource fork font
    - ftxvalidator   (for MacOS X 10.1 -)   dfont or naked-sfnt
    - ftxdumperfuser (for MacOS X 10.1 -)   dfont or naked-sfnt

  However, all tests were done on a PowerPC based Macintosh; at present,
  we have not checked those tools on a m68k-based Macintosh.

  In total, we checked 183 fonts  bundled to MacOS 9.1, MacOS 9.2, MacOS
  10.0, MacOS X  10.1, MSIE for MacOS, and  AppleWorks 6.0.  These fonts
  are distributed  officially, but many broken GX/AAT  tables were found
  by Apple's font tools.  In the following, we list typical violation of
  the GX specification, in fonts officially distributed with those Apple
  systems.

  4-1. broken BinSrchHeader (19/183)
  ----------------------------------

    `BinSrchHeader' is  a header of a  data array for  m68k platforms to
    access memory efficiently.  Although  there are only two independent
    parameters  for real  (`unitSize' and  `nUnits'),  BinSrchHeader has
    three additional parameters which  can be calculated from `unitSize'
    and  `nUnits',  for  fast  setup.   Apple  font  tools  ignore  them
    silently, so gxvalid warns if it finds and inconsistency, and always
    continues  validation.    The  additional  parameters   are  ignored
    regardless of the consistency.

      19  fonts include  such  inconsistencies; all  breaks  are in  the
      BinSrchHeader structure of the `kern' table.

  4-2. too-short LookupTable (5/183)
  ----------------------------------

    LookupTable format 0  is a simple array to get a  value from a given
    GID (glyph  ID); the index of  this array is a  GID too.  Therefore,
    the length  of the array is expected  to be same as  the maximum GID
    value defined  in the `maxp' table,  but there are  some fonts whose
    LookupTable format 0 is too  short to cover all GIDs.  FontValidator
    ignores  this error silently,  ftxvalidator and  ftxdumperfuser both
    warn and continue.  Similar problems are found in format 3 subtables
    of `kern'.  gxvalid  warns always and abort if  the validation level
    is set to FT_VALIDATE_PARANOID.

      5 fonts include too-short kern format 0 subtables.
      1 font includes too-short kern format 3 subtable.

  4-3. broken LookupTable format 2 (1/183)
  ----------------------------------------

    LookupTable  format  2,  subformat  4  covers the  GID  space  by  a
    collection  of  segments which  are  specified  by `firstGlyph'  and
    `lastGlyph'.   Some  fonts  store  `firstGlyph' and  `lastGlyph'  in
    reverse order,  so the segment specification is  broken.  Apple font
    tools ignore this error silently;  a broken segment is ignored as if
    it  did not  exist.   gxvalid  warns and  normalize  the segment  at
    FT_VALIDATE_DEFAULT, or ignore  the segment at FT_VALIDATE_TIGHT, or
    abort at FT_VALIDATE_PARANOID.

      1 font includes broken LookupTable format 2, in the `just' table.

    *) It seems  that all fonts manufactured by  ITC for AppleWorks have
       this error.

  4-4. bad bracketing in glyph property (14/183)
  ----------------------------------------------

    GX/AAT defines a  `bracketing' property of the glyphs  in the `prop'
    table,  to control layout  features of  strings enclosed  inside and
    outside  of   brackets.   Some  fonts   give  inappropriate  bracket
    properties  to glyphs.   Apple  font tools  warn  about this  error;
    gxvalid warns too and aborts at FT_VALIDATE_PARANOID.

      14 fonts include wrong bracket properties.


  4-5. invalid feature number (117/183)
  -------------------------------------

    The GX/AAT extension can  include 255 different layout features, but
    popular      layout      features      are      predefined      (see
    http://developer.apple.com/fonts/Registry/index.html).   Some  fonts
    include feature  numbers which are incompatible  with the predefined
    feature registry.

    In our survey, there are 140 fonts including `feat' table.

    a) 67 fonts use a feature number which should not be used.
    b) 117 fonts set the wrong feature range (nSetting).  This is mostly
       found in the `mort' and `morx' tables.

    Apple  font tools give  no warning,  although they  cannot recognize
    what  the feature  is.   At FT_VALIDATE_DEFAULT,  gxvalid warns  but
    continues in both cases (a, b).  At FT_VALIDATE_TIGHT, gxvalid warns
    and aborts for (a), but continues for (b).  At FT_VALIDATE_PARANOID,
    gxvalid warns and aborts in both cases (a, b).

  4-6. invalid prop version (10/183)
  ----------------------------------

    As most TrueType GX tables, the `prop' table must start with a 32bit
    version identifier: 0x00010000,  0x00020000 or 0x00030000.  But some
    fonts  store nonsense binary  data instead.   When Apple  font tools
    find them, they abort the processing immediately, and the data which
    follows is unhandled.  gxvalid does the same.

      10 fonts include broken `prop' version.

    All  of these  fonts are  classic  TrueType fonts  for the  Japanese
    script, manufactured by Apple.

  4-7. unknown resource name (2/183)
  ------------------------------------

    NOTE: THIS IS NOT A TRUETYPE GX ERROR.

    If  a TrueType  font is  stored  in the  resource fork  or in  dfont
    format, the data must be tagged as `sfnt' in the resource fork index
    to invoke TrueType font handler for the data.  But the TrueType font
    data  in   `Keyboard.dfont'  is  tagged   as  `kbd',  and   that  in
    `LastResort.dfont' is tagged as  `lst'.  Apple font tools can detect
    that the data is in  TrueType format and successfully validate them.
    Maybe  this is possible  because they  are known  to be  dfont.  The
    current  implementation  of the  resource  fork  driver of  FreeType
    cannot do that, thus gxvalid cannot validate them.

      2 fonts use an unknown tag for the TrueType font resource.

5. `kern' table issues
----------------------

  In common terminology of TrueType, `kern' is classified as a basic and
  platform-independent table.  But there are Apple extensions of `kern',
  and  there is  an  extension which  requires  a GX  state machine  for
  contextual kerning.   Therefore, gxvalid includes  a special validator
  for  `kern' tables.   Unfortunately, there  is no  exact  algorithm to
  check Apple's extension, so  gxvalid includes a heuristic algorithm to
  find  the proper validation  routines for  all possible  data formats,
  including    the   data    format   for    Microsoft.     By   calling
  classic_kern_validate() instead of gxv_validate(), you can specify the
  `kern' format  explicitly.  However, current  FreeType2 uses Microsoft
  `kern' format  only, others  are ignored (and  should be handled  in a
  library one level higher than FreeType).

  5-1. History
  ------------

    The original  16bit version of `kern'  was designed by  Apple in the
    pre-GX  era, and  it was  also approved  by  Microsoft.  Afterwards,
    Apple designed a  new 32bit version of the  `kern' table.  According
    to  the documentation, the  difference between  the 16bit  and 32bit
    version is only the size of  variables in the `kern' header.  In the
    following,  we call  the original  16bit version  as  `classic', and
    32bit version as `new'.

  5-2. Versions and dialects which should be differentiated
  ---------------------------------------------------------

    The `kern' table  consists of a table header  and several subtables.
    The version number  which identifies a `classic' or  a `new' version
    is  explicitly   written  in  the   table  header,  but   there  are
    undocumented  differences between  Microsoft's and  Apple's formats.
    It is  called a `dialect' in  the following.  There  are three cases
    which  should  be  handled:   the  new  Apple-dialect,  the  classic
    Apple-dialect,  and the classic  Microsoft-dialect.  An  analysis of
    the formats and the auto detection algorithm of gxvalid is described
    in the following.

    5-2-1. Version detection: classic and new kern
    ----------------------------------------------

      According  to Apple  TrueType  specification, there  are only  two
      differences between the classic and the new:

        - The `kern' table header starts with the version number.
          The classic version starts with 0x0000 (16bit),
          the new version starts with 0x00010000 (32bit).

        - In the  `kern' table header,  the number of  subtables follows
          the version number.
          In the classic version, it is stored as a 16bit value.
          In the new version, it is stored as a 32bit value.

      From Apple font tool's output (DumpKERN is also tested in addition
      to  the  three  Apple  font  tools in  above),  there  is  another
      undocumented difference.  In the  new version, the subtable header
      includes a 16bit variable  named `tupleIndex' which does not exist
      in the classic version.

      The new version  can store all subtable formats (0,  1, 2, and 3),
      but the Apple TrueType specification does not mention the subtable
      formats available in the classic version.

    5-2-2. Avaibale subtable formats in classic version
    ---------------------------------------------------

      Although the  Apple TrueType  specification recommends to  use the
      classic version in  the case if the font is  designed for both the
      Apple and Microsoft platforms,  it does not document the available
      subtable formats in the classic version.

      According  to the Microsoft  TrueType specification,  the subtable
      format  assured for  Windows  and OS/2  support  is only  subtable
      format  0.  The  Microsoft TrueType  specification  also describes
      subtable format  2, but does  not mention which  platforms support
      it.  Aubtable formats 1, 3,  and higher are documented as reserved
      for future use.  Therefore, the classic version can store subtable
      formats 0 and 2, at least.  `ttfdump.exe', a font tool provided by
      Microsoft,  ignores the  subtable format  written in  the subtable
      header, and parses the table as if all subtables are in format 0.

      `kern'  subtable format  1  uses  a StateTable,  so  it cannot  be
      utilized without a GX  State Machine.  Therefore, it is reasonable
      to assume  that format 1 (and  3) were introduced  after Apple had
      introduced GX and moved to the new 32bit version.

    5-2-3. Apple and Microsoft dialects
    -----------------------------------

      The  `kern' subtable  has  a 16bit  `coverage'  field to  describe
      kerning attributes, but bit interpretations by Apple and Microsoft
      are different:  For example, Apple  uses bits 0-7 to  identify the
      subtable, while Microsoft uses bits 8-15.

      In  addition, due  to the  output of  DumpKERN  and FontValidator,
      Apple's bit interpretations of coverage in classic and new version
      are  incompatible also.   In  summary, there  are three  dialects:
      classic Apple  dialect, classic  Microsoft dialect, and  new Apple
      dialect.  The classic Microsoft  dialect and the new Apple dialect
      are documented  by each vendors' TrueType  font specification, but
      the documentation for classic Apple dialect is not available.
      
      For example,  in the  new Apple dialect,  bit 15 is  documented as
      `set to  1 if  the kerning  is vertical'.  On  the other  hand, in
      classic Microsoft dialect, bit 1 is documented as `set to 1 if the
      kerning  is  horizontal'.   From   the  outputs  of  DumpKERN  and
      FontValidator, classic  Apple dialect recognizes  15 as `set  to 1
      when  the kerning  is horizontal'.   From the  results  of similar
      experiments, classic Apple dialect  seems to be the Endian reverse
      of the classic Microsoft dialect.

      As a  conclusion it must be  noted that no font  tool can identify
      classic Apple dialect or classic Microsoft dialect automatically.

    5-2-4. gxvalid auto dialect detection algorithm
    -----------------------------------------------

      The first 16  bits of the `kern' table are  enough to identify the
      version:

        - if  the first  16  bits are  0x0000,  the `kern'  table is  in
          classic Apple dialect or classic Microsoft dialect
        - if the first 16 bits are  0x0001, and next 16 bits are 0x0000,
          the kern table is in new Apple dialect.

      If the `kern'  table is a classic one,  the 16bit `coverage' field
      is checked next.   Firstly, the coverage bits are  decoded for the
      classic Apple dialect using the following bit masks (this is based
      on DumpKERN output):

        0x8000: 1=horizontal, 0=vertical
        0x4000: not used
        0x2000: 1=cross-stream, 0=normal
        0x1FF0: reserved
        0x000F: subtable format

      If  any  of  reserved  bits  are  set  or  the  subtable  bits  is
      interpreted as format 1 or 3, we take it as `impossible in classic
      Apple dialect' and retry, using the classic Microsoft dialect.

        The most popular coverage in new Apple-dialect:         0x8000,
        The most popular coverage in classic Apple-dialect:     0x0000,
        The most popular coverage in classic Microsoft dialect: 0x0001.

  5-3. Tested fonts
  -----------------

    We checked  59 fonts  bundled with MacOS  and 38 fonts  bundled with
    Windows, where all font include a `kern' table.
    
      - fonts bundled with MacOS
        * new Apple dialect
          format 0: 18
          format 2:  1
          format 3:  1
        * classic Apple dialect
          format 0: 14
        * classic Microsoft dialect
          format 0: 15

      - fonts bundled with Windows
        * classic Microsoft dialect
          format 0: 38

    It looks strange that classic Microsoft-dialect fonts are bundled to
    MacOS: they come from MSIE for MacOS, except of MarkerFelt.dfont.


  ACKNOWLEDGEMENT
  ---------------

  Some parts of gxvalid are  derived from both the `gxlayout' module and
  the `otvalid'  module.  Development of  gxlayout was supported  by the
  Information-technology Promotion Agency(IPA), Japan.

  The detailed analysis of undefined  glyph ID utilization in `mort' and
  `morx' tables is provided by George Williams.

------------------------------------------------------------------------

Copyright 2004, 2005 by
suzuki toshiya, Masatake YAMATO, Red hat K.K.,
David Turner, Robert Wilhelm, and Werner Lemberg.

This  file is  part  of the  FreeType  project, and  may  only be  used,
modified,  and  distributed under  the  terms  of  the FreeType  project
license, LICENSE.TXT.  By continuing  to use, modify, or distribute this
file  you indicate that  you have  read the  license and  understand and
accept it fully.


--- end of README ---