tech-invite   World Map     

IETF     RFCs     Groups     SIP     ABNFs    |    3GPP     Specs     Gloss.     Arch.     IMS     UICC    |    Misc.    |    search     info

RFC 7940

Proposed STD
Pages: 82
Top     in Index     Prev     Next
in Group Index     No Prev: Lowest Number in Group     No Next: Highest Number in Group     Group: LAGER

Representing Label Generation Rulesets Using XML

Part 1 of 4, p. 1 to 23
None       Next Section

 


Top       ToC       Page 1 
Internet Engineering Task Force (IETF)                         K. Davies
Request for Comments: 7940                                         ICANN
Category: Standards Track                                     A. Freytag
ISSN: 2070-1721                                              ASMUS, Inc.
                                                             August 2016


            Representing Label Generation Rulesets Using XML

Abstract

   This document describes a method of representing rules for validating
   identifier labels and alternate representations of those labels using
   Extensible Markup Language (XML).  These policies, known as "Label
   Generation Rulesets" (LGRs), are used for the implementation of
   Internationalized Domain Names (IDNs), for example.  The rulesets are
   used to implement and share that aspect of policy defining which
   labels and Unicode code points are permitted for registrations, which
   alternative code points are considered variants, and what actions may
   be performed on labels containing those variants.

Status of This Memo

   This is an Internet Standards Track document.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Further information on
   Internet Standards is available in Section 2 of RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at
   http://www.rfc-editor.org/info/rfc7940.

[Page 2] 
Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1. Introduction ....................................................4
   2. Design Goals ....................................................5
   3. Normative Language ..............................................6
   4. LGR Format ......................................................6
      4.1. Namespace ..................................................7
      4.2. Basic Structure ............................................7
      4.3. Metadata ...................................................8
           4.3.1. The "version" Element ...............................8
           4.3.2. The "date" Element ..................................9
           4.3.3. The "language" Element ..............................9
           4.3.4. The "scope" Element ................................10
           4.3.5. The "description" Element ..........................10
           4.3.6. The "validity-start" and "validity-end" Elements ...11
           4.3.7. The "unicode-version" Element ......................11
           4.3.8. The "references" Element ...........................12
   5. Code Points and Variants .......................................13
      5.1. Sequences .................................................14
      5.2. Conditional Contexts ......................................15
      5.3. Variants ..................................................16
           5.3.1. Basic Variants .....................................16
           5.3.2. The "type" Attribute ...............................17
           5.3.3. Null Variants ......................................18
           5.3.4. Variants with Reflexive Mapping ....................19
           5.3.5. Conditional Variants ...............................20
      5.4. Annotations ...............................................22
           5.4.1. The "ref" Attribute ................................22
           5.4.2. The "comment" Attribute ............................23
      5.5. Code Point Tagging ........................................23

Top      ToC       Page 3 
   6. Whole Label and Context Evaluation .............................23
      6.1. Basic Concepts ............................................23
      6.2. Character Classes .........................................25
           6.2.1. Declaring and Invoking Named Classes ...............25
           6.2.2. Tag-Based Classes ..................................26
           6.2.3. Unicode Property-Based Classes .....................26
           6.2.4. Explicitly Declared Classes ........................28
           6.2.5. Combined Classes ...................................29
      6.3. Whole Label and Context Rules .............................30
           6.3.1. The "rule" Element .................................31
           6.3.2. The Match Operators ................................32
           6.3.3. The "count" Attribute ..............................33
           6.3.4. The "name" and "by-ref" Attributes .................34
           6.3.5. The "choice" Element ...............................34
           6.3.6. Literal Code Point Sequences .......................35
           6.3.7. The "any" Element ..................................35
           6.3.8. The "start" and "end" Elements .....................35
           6.3.9. Example Context Rule from IDNA Specification .......36
      6.4. Parameterized Context or When Rules .......................37
           6.4.1. The "anchor" Element ...............................37
           6.4.2. The "look-behind" and "look-ahead" Elements ........38
           6.4.3. Omitting the "anchor" Element ......................40
   7. The "action" Element ...........................................40
      7.1. The "match" and "not-match" Attributes ....................41
      7.2. Actions with Variant Type Triggers ........................41
           7.2.1. The "any-variant", "all-variants", and
                  "only-variants" Attributes .........................41
           7.2.2. Example from Tables in the Style of RFC 3743 .......44
      7.3. Recommended Disposition Values ............................45
      7.4. Precedence ................................................45
      7.5. Implied Actions ...........................................45
      7.6. Default Actions ...........................................46
   8. Processing a Label against an LGR ..............................47
      8.1. Determining Eligibility for a Label .......................47
           8.1.1. Determining Eligibility Using Reflexive
                  Variant Mappings ...................................47
      8.2. Determining Variants for a Label ..........................48
      8.3. Determining a Disposition for a Label or Variant Label ....49
      8.4. Duplicate Variant Labels ..................................50
      8.5. Checking Labels for Collision .............................50
   9. Conversion to and from Other Formats ...........................51
   10. Media Type ....................................................51
   11. IANA Considerations ...........................................52
      11.1. Media Type Registration ..................................52
      11.2. URN Registration .........................................53
      11.3. Disposition Registry .....................................53

Top      ToC       Page 4 
   12. Security Considerations .......................................54
      12.1. LGRs Are Only a Partial Remedy for Problem Space .........54
      12.2. Computational Expense of Complex Tables ..................54
   13. References ....................................................55
      13.1. Normative References .....................................55
      13.2. Informative References ...................................56
   Appendix A. Example Tables ........................................58
   Appendix B. How to Translate Tables Based on RFC 3743 into the
               XML Format ............................................63
   Appendix C. Indic Syllable Structure Example ......................68
      C.1. Reducing Complexity .......................................70
   Appendix D. RELAX NG Compact Schema ...............................71
   Acknowledgements ..................................................82
   Authors' Addresses ................................................82

1.  Introduction

   This document specifies a method of using Extensible Markup Language
   (XML) to describe Label Generation Rulesets (LGRs).  LGRs are
   algorithms used to determine whether, and under what conditions, a
   given identifier label is permitted, based on the code points it
   contains and their context.  These algorithms comprise a list of
   permissible code points, variant code point mappings, and a set of
   rules that act on the code points and mappings.  LGRs form part of an
   administrator's policies.  In deploying Internationalized Domain
   Names (IDNs), they have also been known as IDN tables or variant
   tables.

   There are other kinds of policies relating to labels that are not
   normally covered by LGRs and are therefore not necessarily
   representable by the XML format described here.  These include, but
   are not limited to, policies around trademarks, or prohibition of
   fraudulent or objectionable words.

   Administrators of the zones for top-level domain registries have
   historically published their LGRs using ASCII text or HTML.  The
   formatting of these documents has been loosely based on the format
   used for the Language Variant Table described in [RFC3743].
   [RFC4290] also provides a "model table format" that describes a
   similar set of functionality.  Common to these formats is that the
   algorithms used to evaluate the data therein are implicit or
   specified elsewhere.

   Through the first decade of IDN deployment, experience has shown that
   LGRs derived from these formats are difficult to consistently
   implement and compare, due to their differing formats.  A universal

Top      ToC       Page 5 
   format, such as one using a structured XML format, will assist by
   improving machine readability, consistency, reusability, and
   maintainability of LGRs.

   When used to represent a simple list of permitted code points, the
   format is quite straightforward.  At the cost of some complexity in
   the resulting file, it also allows for an implementation of more
   sophisticated handling of conditional variants that reflects the
   known requirements of current zone administrator policies.

   Another feature of this format is that it allows many of the
   algorithms to be made explicit and machine implementable.  A
   remaining small set of implicit algorithms is described in this
   document to allow commonality in implementation.

   While the predominant usage of this specification is to represent IDN
   label policy, the format is not limited to IDN usage and may also be
   used for describing ASCII domain name label rulesets, or other types
   of identifier labels beyond those used for domain names.

2.  Design Goals

   The following goals informed the design of this format:

   o  The format needs to be implementable in a reasonably
      straightforward manner in software.

   o  The format should be able to be automatically checked for
      formatting errors, so that common mistakes can be caught.

   o  An LGR needs to be able to express the set of valid code points
      that are allowed for registration under a specific administrator's
      policies.

   o  An LGR needs to be able to express computed alternatives to a
      given identifier based on mapping relationships between code
      points, whether one-to-one or many-to-many.  These computed
      alternatives are commonly known as "variants".

   o  Variant code points should be able to be tagged with explicit
      dispositions or categories that can be used to support registry
      policy (such as whether to allocate the computed variant or to
      merely block it from usage or registration).

   o  Variants and code points must be able to be stipulated based on
      contextual information.  For example, some variants may only be
      applicable when they follow a certain code point or when the code
      point is displayed in a specific presentation form.

Top      ToC       Page 6 
   o  The data contained within an LGR must be able to be interpreted
      unambiguously, so that independent implementations that utilize
      the contents will arrive at the same results.

   o  To the largest extent possible, policy rules should be able to be
      specified in the XML format without relying on hidden or built-in
      algorithms in implementations.

   o  LGRs should be suitable for comparison and reuse, such that one
      could easily compare the contents of two or more to see the
      differences, to merge them, and so on.

   o  As many existing IDN tables as practicable should be able to be
      migrated to the LGR format with all applicable interpretation
      logic retained.

   These requirements are partly derived from reviewing the existing
   corpus of published IDN tables, plus the requirements of ICANN's work
   to implement an LGR for the DNS root zone [LGR-PROCEDURE].  In
   particular, Section B of that document identifies five specific
   requirements for an LGR methodology.

   The syntax and rules in [RFC5892] and [RFC3743] were also reviewed.

   It is explicitly not the goal of this format to stipulate what code
   points should be listed in an LGR by a zone administrator.  Which
   registration policies are used for a particular zone are outside the
   scope of this memo.

3.  Normative Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

4.  LGR Format

   An LGR is expressed as a well-formed XML document [XML] that conforms
   to the schema defined in Appendix D.

   As XML is case sensitive, an LGR must be authored with the correct
   casing.  For example, the XML element names MUST be in lowercase as
   described in this specification, and matching of attribute values is
   only performed in a case-sensitive manner.

   A document that is not well-formed, is non-conforming, or violates
   other constraints specified in this specification MUST be rejected.

Top      ToC       Page 7 
4.1.  Namespace

   The XML Namespace URI is "urn:ietf:params:xml:ns:lgr-1.0".

   See Section 11.2 for more information.

4.2.  Basic Structure

   The basic XML framework of the document is as follows:

       <?xml version="1.0"?>
       <lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
           ...
       </lgr>

   The "lgr" element contains up to three sub-elements or sections.
   First is an optional "meta" element that contains all metadata
   associated with the LGR, such as its authorship, what it is used for,
   implementation notes, and references.  This is followed by a required
   "data" element that contains the substantive code point data.
   Finally, an optional "rules" element contains information on rules
   for evaluating labels, if any, along with "action" elements providing
   for the disposition of labels and computed variant labels.

       <?xml version="1.0"?>
       <lgr xmlns="urn:ietf:params:xml:ns:lgr-1.0">
           <meta>
               ...
           </meta>
           <data>
               ...
           </data>
           <rules>
               ...
           </rules>
       </lgr>

   A document MUST contain exactly one "lgr" element.  Each "lgr"
   element MUST contain zero or one "meta" element, exactly one "data"
   element, and zero or one "rules" element; and these three elements
   MUST be in that order.

   Some elements that are direct or nested child elements of the "rules"
   element MUST be placed in a specific relative order to other elements
   for the LGR to be valid.  An LGR that violates these constraints MUST
   be rejected.  In other cases, changing the ordering would result in a
   valid, but different, specification.

Top      ToC       Page 8 
   In the following descriptions, required, non-repeating elements or
   attributes are generally not called out explicitly, in contrast to
   "OPTIONAL" ones, or those that "MAY" be repeated.  For attributes
   that take lists as values, the elements MUST be space-separated.

4.3.  Metadata

   The "meta" element expresses metadata associated with the LGR, and
   the element SHOULD be included so that the associated metadata are
   available as part of the LGR and cannot become disassociated.  The
   following subsections describe elements that may appear within the
   "meta" element.

   The "meta" element can be used to identify the author or relevant
   contact person, explain the intended usage of the LGR, and provide
   implementation notes as well as references.  Detailed metadata allow
   the LGR document to become self-documenting -- for example, if
   rendered in a human-readable format by an appropriate tool.

   Providing metadata pertaining to the date and version of the LGR is
   particularly encouraged to make it easier for interoperating
   consumers to ensure that they are using the correct LGR.

   With the exception of the "unicode-version" element, the data
   contained within is not required by software consuming the LGR in
   order to calculate valid labels or to calculate variants.  If
   present, the "unicode-version" element MUST be used by a consumer of
   the table to identify that it has the correct Unicode property data
   to perform operations on the table.  This ensures that possible
   differences in code point properties between editions of the Unicode
   Standard do not impact the product of calculations utilizing an LGR.

4.3.1.  The "version" Element

   The "version" element is OPTIONAL.  It is used to uniquely
   identify each version of the LGR.  No specific format is required,
   but it is RECOMMENDED that it be the decimal representation of a
   single positive integer, which is incremented with each revision of
   the file.

   An example of a typical first edition of a document:

       <version>1</version>

   The "version" element may have an OPTIONAL "comment" attribute.

       <version comment="draft">1</version>

Top      ToC       Page 9 
4.3.2.  The "date" Element

   The OPTIONAL "date" element is used to identify the date the LGR was
   posted.  The contents of this element MUST be a valid ISO 8601
   "full-date" string as described in [RFC3339].

   Example of a date:

       <date>2009-11-01</date>

4.3.3.  The "language" Element

   Each OPTIONAL "language" element identifies a language or script for
   which the LGR is intended.  The value of the "language" element MUST
   be a valid language tag as described in [RFC5646].  The tag may refer
   to a script plus undefined language if the LGR is not intended for a
   specific language.

   Example of an LGR for the English language:

       <language>en</language>

   If the LGR applies to a script rather than a specific language, the
   "und" language tag SHOULD be used followed by the relevant script
   subtag from [RFC5646].  For example, for a Cyrillic script LGR:

       <language>und-Cyrl</language>

   If the LGR covers a set of multiple languages or scripts, the
   "language" element MAY be repeated.  However, for cases of a
   script-specific LGR exhibiting insignificant admixture of code points
   from other scripts, it is RECOMMENDED to use a single "language"
   element identifying the predominant script.  In the exceptional case
   of a multi-script LGR where no script is predominant, use Zyyy
   (Common):

       <language>und-Zyyy</language>

Top      ToC       Page 10 
4.3.4.  The "scope" Element

   This OPTIONAL element refers to a scope, such as a domain, to which
   this policy is applied.  The "type" attribute specifies the type of
   scope being defined.  A type of "domain" means that the scope is a
   domain that represents the apex of the DNS zone to which the LGR is
   applied.  For that type, the content of the "scope" element MUST be a
   domain name written relative to the root zone, in presentation format
   with no trailing dot.  However, in the unique case of the DNS root
   zone, it is represented as ".".

       <scope type="domain">example.com</scope>

   There may be multiple "scope" tags used -- for example, to reflect a
   list of domains to which the LGR is applied.

   No other values of the "type" attribute are defined by this
   specification; however, this specification can be used for
   applications other than domain names.  Implementers of LGRs for
   applications other than domain names SHOULD define the scope
   extension grammar in an IETF specification or use XML namespaces to
   distinguish their scoping mechanism distinctly from the base LGR
   namespace.  An explanation of any custom usage of the scope in the
   "description" element is RECOMMENDED.

       <scope xmlns="http://example.com/ns/scope/1.0">
           ... content per alternate namespace ...
       </scope>

4.3.5.  The "description" Element

   The "description" element is an OPTIONAL, free-form element that
   contains any additional relevant description that is useful for the
   user in its interpretation.  Typically, this field contains
   authorship information, as well as additional context on how the LGR
   was formulated and how it applies, such as citations and references
   that apply to the LGR as a whole.

   This field should not be relied upon for providing instructions on
   how to parse or utilize the data contained elsewhere in the
   specification.  Authors of tables should expect that software
   applications that parse and use LGRs will not use the "description"
   element to condition the application of the LGR's data and rules.

Top      ToC       Page 11 
   The element has an OPTIONAL "type" attribute, which refers to the
   Internet media type [RFC2045] of the enclosed data.  Typical types
   would be "text/plain" or "text/html".  The attribute SHOULD be a
   valid media type.  If supplied, it will be assumed that the contents
   are of that media type.  If the description lacks a "type" value, it
   will be assumed to be plain text ("text/plain").

4.3.6.  The "validity-start" and "validity-end" Elements

   The "validity-start" and "validity-end" elements are OPTIONAL
   elements that describe the time period from which the contents of the
   LGR become valid (are used in registry policy) and time when the
   contents of the LGR cease to be used, respectively.

   The dates MUST conform to the "full-date" format described in
   Section 5.6 of [RFC3339].

       <validity-start>2014-03-12</validity-start>

4.3.7.  The "unicode-version" Element

   Whenever an LGR depends on character properties from a given version
   of the Unicode Standard, the version number used in creating the LGR
   MUST be listed in the form x.y.z, where x, y, and z are positive
   decimal integers (see [Unicode-Versions]).  If any software
   processing the table does not have access to character property data
   of the requisite version, it MUST NOT perform any operations relating
   to whole-label evaluation relying on Unicode character properties
   (Section 6.2.3).

   The value of a given Unicode character property may change between
   versions of the Unicode Character Database [UAX44], unless such
   change has been explicitly disallowed in [Unicode-Stability].  It is
   RECOMMENDED to only reference properties defined as stable or
   immutable.  As an alternative to referencing the property, the
   information can be presented explicitly in the LGR.

       <unicode-version>6.3.0</unicode-version>

   It is not necessary to include a "unicode-version" element for LGRs
   that do not make use of Unicode character properties; however, it is
   RECOMMENDED.

Top      ToC       Page 12 
4.3.8.  The "references" Element

   An LGR may define a list of references that are used to associate
   various individual elements in the LGR to one or more normative
   references.  A common use for references is to annotate that code
   points belong to an externally defined collection or standard or to
   give normative references for rules.

   References are specified in an OPTIONAL "references" element
   containing one or more "reference" elements, each with a unique "id"
   attribute.  It is RECOMMENDED that the "id" attribute be a zero-based
   integer; however, in addition to digits 0-9, it MAY contain uppercase
   letters A-Z, as well as a period, hyphen, colon, or underscore.  The
   value of each "reference" element SHOULD be the citation of a
   standard, dictionary, or other specification in any suitable format.
   In addition to an "id" attribute, a "reference" element MAY have a
   "comment" attribute for an optional free-form annotation.

       <references>
         <reference id="0">The Unicode Consortium.  The Unicode
           Standard, Version 8.0.0, (Mountain View, CA: The Unicode
           Consortium, 2015.  ISBN 978-1-936213-10-8)
           http://www.unicode.org/versions/Unicode8.0.0/</reference>
         <reference id="1">Big-5: Computer Chinese Glyph and Character
            Code Mapping Table, Technical Report C-26, 1984</reference>
         <reference id="2" comment="synchronized with Unicode 6.1">
            ISO/IEC
            10646:2012 3rd edition</reference>
         ...
       </references>
       ...
       <data>
         <char cp="0620" ref="0 2" />
         ...
       </data>

   A reference is associated with an element by using its id as part of
   an optional "ref" attribute (see Section 5.4.1).  The "ref" attribute
   may be used with many kinds of elements in the "data" or "rules"
   sections of the LGR, most notably those defining code points,
   variants, and rules.  However, a "ref" attribute may not occur in
   certain kinds of elements, including references to named character
   classes or rules.  See below for the description of these elements.

Top      ToC       Page 13 
5.  Code Points and Variants

   The bulk of an LGR is a description of which set of code points is
   eligible for a given label.  For rulesets that perform operations
   that result in potential variants, the code point-level relationships
   between variants need to also be described.

   The code point data is collected within the "data" element.  Within
   this element, a series of "char" and "range" elements describe
   eligible code points or ranges of code points, respectively.
   Collectively, these are known as the repertoire.

   Discrete permissible code points or code point sequences (see
   Section 5.1) are declared with a "char" element.  Here is a minimal
   example declaration for a single code point, with the code point
   value given in the "cp" attribute:

       <char cp="002D"/>

   As described below, a full declaration for a "char" element, whether
   or not it is used for a single code point or for a sequence (see
   Section 5.1), may have optional child elements defining variants.
   Both the "char" and "range" elements can take a number of optional
   attributes for conditional inclusion, commenting, cross-referencing,
   and character tagging, as described below.

   Ranges of permissible code points may be declared with a "range"
   element, as in this minimal example:

       <range first-cp="0030" last-cp="0039"/>

   The range is inclusive of the first and last code points.  Any
   additional attributes defined for a "range" element act as if applied
   to each code point within.  A "range" element has no child elements.

   It is always possible to substitute a list of individually specified
   code points for a "range" element.  The reverse is not necessarily
   the case.  Whenever such a substitution is possible, it makes no
   difference in processing the data.  Tools reading or writing the LGR
   format are free to aggregate sequences of consecutive code points of
   the same properties into "range" elements.

   Code points MUST be represented according to the standard Unicode
   convention but without the prefix "U+": they are expressed in
   uppercase hexadecimal and are zero-padded to a minimum of 4 digits.

Top      ToC       Page 14 
   The rationale for not allowing other encoding formats, including
   native Unicode encoding in XML, is explored in [UAX42].  The XML
   conventions used in this format, such as element and attribute names,
   mirror this document where practical and reasonable to do so.  It is
   RECOMMENDED to list all "char" elements in ascending order of the
   "cp" attribute.  Not doing so makes it unnecessarily difficult for
   authors and reviewers to check for errors, such as duplications, or
   to review and compare against listing of code points in other
   documents and specifications.

   All "char" elements in the "data" section MUST have distinct "cp"
   attributes.  The "range" elements MUST NOT specify code point ranges
   that overlap either another range or any single code point "char"
   elements.  An LGR that defines the same code point more than once by
   any combination of "char" or "range" elements MUST be rejected.

5.1.  Sequences

   A sequence of two or more code points may be specified in an LGR --
   for example, when defining the source for n:m variant mappings.
   Another use of sequences would be in cases when the exact sequence of
   code points is required to occur in order for the constituent
   elements to be eligible, such as when some code point is only
   eligible when preceded or followed by a certain code point.  The
   following would define the eligibility of the MIDDLE DOT (U+00B7)
   only when both preceded and followed by the LATIN SMALL LETTER L
   (U+006C):

       <char cp="006C 00B7 006C" comment="Catalan middle dot"/>

   All sequences defined this way must be distinct, but sub-sequences
   may be defined.  Thus, the sequence defined here may coexist with
   single code point definitions such as:

       <char cp="006C" />

   As an alternative to using sequences to define a required context, a
   "char" or "range" element may specify a conditional context using an
   optional "when" attribute as described below in Section 5.2.  Using a
   conditional context is more flexible because a context is not limited
   to a specific sequence of code points.  In addition, using a context
   allows the choice of specifying either a prohibited or a required
   context.

Top      ToC       Page 15 
5.2.  Conditional Contexts

   A conditional context is specified by a rule that must be satisfied
   (or, alternatively, must not be satisfied) for a code point in a
   given label, often at a particular location in a label.

   To specify a conditional context, either a "when" or "not-when"
   attribute may be used.  The value of each "when" or "not-when"
   attribute is a context rule as described below in Section 6.3.  This
   rule can be a rule evaluating the whole label or a parameterized
   context rule.  The context condition is met when the rule specified
   in the "when" attribute is matched or when the rule in the "not-when"
   attribute fails to match.  It is an error to reference a rule that is
   not actually defined in the "rules" element.

   A parameterized context rule (see Section 6.4) defines the context
   immediately surrounding a given code point; unlike a sequence, the
   context is not limited to a specific fixed code point but, for
   example, may designate any member of a certain character class or a
   code point that has a certain Unicode character property.

   Given a suitable definition of a parameterized context rule named
   "follows-virama", this example specifies that a ZERO WIDTH JOINER
   (U+200D) is restricted to immediately follow any of several code
   points classified as virama:

       <char cp="200D" when="follows-virama" />

   For a complete example, see Appendix A.

   In contrast, a whole label rule (see Section 6.3) specifies a
   condition to be met by the entire label -- for example, that it must
   contain at least one code point from a given script anywhere in the
   label.  In the following example, no digit from either range may
   occur in a label that mixes digits from both ranges:

       <data>
          <range first-cp="0660" last-cp="0669" not-when="mixed-digits"
                 tag="arabic-indic-digits" />
          <range first-cp="06F0" last-cp="06F9" not-when="mixed-digits"
                 tag="extended-arabic-indic-digits" />
       </data>

   (See Section 6.3.9 for an example of the "mixed-digits" rule.)

Top      ToC       Page 16 
   The OPTIONAL "when" or "not-when" attributes are mutually exclusive.
   They MAY be applied to both "char" and "range" elements in the "data"
   element, including "char" elements defining sequences of code points,
   as well as to "var" elements (see Section 5.3.5).

   If a label contains one or more code points that fail to satisfy a
   conditional context, the label is invalid (see Section 7.5).  For
   variants, the conditional context restricts the definition of the
   variant to the case where the condition is met.  Outside the
   specified context, a variant is not defined.

5.3.  Variants

   Most LGRs typically only determine simple code point eligibility, and
   for them, the elements described so far would be the only ones
   required for their "data" section.  Others additionally specify a
   mapping of code points to other code points, known as "variants".
   What constitutes a variant code point is a matter of policy and
   varies for each implementation.  The following examples are intended
   to demonstrate the syntax; they are not necessarily typical.

5.3.1.  Basic Variants

   Variant code points are specified using one of more "var" elements as
   children of a "char" element.  The target mapping is specified using
   the "cp" attribute.  Other, optional attributes for the "var" element
   are described below.

   For example, to map LATIN SMALL LETTER V (U+0076) as a variant of
   LATIN SMALL LETTER U (U+0075):

       <char cp="0075">
           <var cp="0076"/>
       </char>

   A sequence of multiple code points can be specified as a variant of a
   single code point.  For example, the sequence of LATIN SMALL LETTER O
   (U+006F) then LATIN SMALL LETTER E (U+0065) might hypothetically be
   specified as a variant for a LATIN SMALL LETTER O WITH DIAERESIS
   (U+00F6) as follows:

       <char cp="00F6">
           <var cp="006F 0065"/>
       </char>

   The source and target of a variant mapping may both be sequences but
   not ranges.

Top      ToC       Page 17 
   If the source of one mapping is a prefix sequence of the source for
   another, both variant mappings will be considered at the same
   location in the input label when generating permuted variant labels.
   If poorly designed, an LGR containing such an instance of a prefix
   relation could generate multiple instances of the same variant label
   for the same original label, but with potentially different
   dispositions.  Any duplicate variant labels encountered MUST be
   treated as an error (see Section 8.4).

   The "var" element specifies variant mappings in only one direction,
   even though the variant relation is usually considered symmetric;
   that is, if A is a variant of B, then B should also be a variant of
   A.  The format requires that the inverse of the variant be given
   explicitly to fully specify symmetric variant relations in the LGR.
   This has the beneficial side effect of making the symmetry explicit:

       <char cp="006F 0065">
           <var cp="00F6"/>
       </char>

   Variant relations are normally not only symmetric but also
   transitive.  If A is a variant of B and B is a variant of C, then A
   is also a variant of C.  As with symmetry, these transitive relations
   are only part of the LGR if spelled out explicitly.  Implementations
   that require an LGR to be symmetric and transitive should verify this
   mechanically.

   All variant mappings are unique.  For a given "char" element, all
   "var" elements MUST have a unique combination of "cp", "when", and
   "not-when" attributes.  It is RECOMMENDED to list the "var" elements
   in ascending order of their target code point sequence.  (For "when"
   and "not-when" attributes, see Section 5.3.5.)

5.3.2.  The "type" Attribute

   Variants may be tagged with an OPTIONAL "type" attribute.  The value
   of the "type" attribute may be any non-empty value not starting with
   an underscore and not containing spaces.  This value is used to
   resolve the disposition of any variant labels created using a given
   variant.  (See Section 7.2.)

   By default, the values of the "type" attribute directly describe the
   target policy status (disposition) for a variant label that was
   generated using a particular variant, with any variant label being
   assigned a disposition corresponding to the most restrictive variant
   type.  Several conventional disposition values are predefined below
   in Section 7.  Whenever these values can represent the desired
   policy, they SHOULD be used.

Top      ToC       Page 18 
       <char cp="767C">
           <var cp="53D1" type="allocatable"/>
           <var cp="5F42" type="blocked"/>
           <var cp="9AEA" type="blocked"/>
           <var cp="9AEE" type="blocked"/>
       </char>

   By default, if a variant label contains any instance of one of the
   variants of type "blocked", the label would be blocked, but if it
   contained only instances of variants to be allocated, it could be
   allocated.  See the discussion about implied actions in Section 7.6.

   The XML format for the LGR makes the relation between the values of
   the "type" attribute on variants and the resulting disposition of
   variant labels fully explicit.  See the discussion in Section 7.2.
   Making this relation explicit allows a generalization of the "type"
   attribute from directly reflecting dispositions to a more
   differentiated intermediate value that is then used in the resolution
   of label disposition.  Instead of the default action of applying the
   most restrictive disposition to the entire label, such a generalized
   resolution can be used to achieve additional goals, such as limiting
   the set of allocatable variant labels or implementing other policies
   found in existing LGRs (see, for example, Appendix B).

   Because variant mappings MUST be unique, it is not possible to define
   the same variant for the same "char" element with different "type"
   attributes (however, see Section 5.3.5).

5.3.3.  Null Variants

   A null variant is a variant string that maps to no code point.  This
   is used when a particular code point sequence is considered
   discretionary in the context of a whole label.  To specify a null
   variant, use an empty "cp" attribute.  For example, to mark a string
   with a ZERO WIDTH NON-JOINER (U+200C) to the same string without the
   ZERO WIDTH NON-JOINER:

       <char cp="200C">
           <var cp=""/>
       </char>

   This is useful in expressing the intent that some code points in a
   label are to be mapped away when generating a canonical variant of
   the label.  However, in tables that are designed to have symmetric
   variant mappings, this could lead to combinatorial explosion if not
   handled carefully.

Top      ToC       Page 19 
   The symmetric form of a null variant is expressed as follows:

       <char cp="">
           <var cp="200C" type="invalid" />
       </char>

   A "char" element with an empty "cp" attribute MUST specify at least
   one variant mapping.  It is strongly RECOMMENDED to use a type of
   "invalid" or equivalent when defining variant mappings from null
   sequences, so that variant mappings from null sequences are removed
   in variant label generation (see Section 5.3.2).

5.3.4.  Variants with Reflexive Mapping

   At first glance, there seems to be no call for adding variant
   mappings for which source and target code points are the same -- that
   is, for which the mapping is reflexive, or, in other words, an
   identity mapping.  Yet, such reflexive mappings occur frequently in
   LGRs that follow [RFC3743].

   Adding a "var" element allows both a type and a reference id to be
   specified for it.  While the reference id is not used in processing,
   the type of the variant can be used to trigger actions.  In permuting
   the label to generate all possible variants, the type associated with
   a reflexive variant mapping is applied to any of the permuted labels
   containing the original code point.

   In the following example, let's assume that the goal is to allocate
   only those labels that contain a variant that is considered
   "preferred" in some way.  As defined in the example, the code point
   U+3473 exists both as a variant of U+3447 and as a variant of itself
   (reflexive mapping).  Assuming an original label of "U+3473 U+3447",
   the permuted variant "U+3473 U+3473" would consist of the reflexive
   variant of U+3473 followed by a variant of U+3447.  Given the variant
   mappings as defined here, the types for both of the variant mappings
   used to generate that particular permutation would have the value
   "preferred":

       <char cp="3447" ref="0">
         <var cp="3473" type="preferred" ref="1 3" />
       </char>
       <char cp="3473" ref="0">
         <var cp="3447" type="blocked" ref="1 3" />
         <var cp="3473" type="preferred" ref="0" />
       </char>

Top      ToC       Page 20 
   Having established the variant types in this way, a set of actions
   could be defined that return a disposition of "allocatable" or
   "activated" for a label consisting exclusively of variants with type
   "preferred", for example.  (For details on how to define actions
   based on variant types, see Section 7.2.1.)

   In general, using reflexive variant mappings in this manner makes it
   possible to calculate disposition values using a uniform approach for
   all labels, whether they consist of mapped variant code points,
   original code points, or a mixture of both.  In particular, the
   dispositions for two otherwise identical labels may differ based on
   which variant mappings were executed in order to generate each of
   them.  (For details on how to generate variants and evaluate
   dispositions, see Section 8.)

   Another useful convention that uses reflexive variants is described
   below in Section 7.2.1.

5.3.5.  Conditional Variants

   Fundamentally, variants are mappings between two sequences of code
   points.  However, in some instances, for a variant relationship to
   exist, some context external to the code point sequence must also be
   considered.  For example, a positional context may determine whether
   two code point sequences are variants of each other.

   An example of that are Arabic code points, which can have different
   forms based on position, with some code points sharing forms, thus
   making them variants in the positions corresponding to those forms.
   Such positional context cannot be solely derived from the code point
   by itself, as the code point would be the same for the various forms.

   As described in Section 5.2, an OPTIONAL "when" or "not-when"
   attribute may be given for any "var" element to specify required or
   prohibited contextual conditions under which the variant is defined.

   Assuming that the "rules" element contains suitably defined rules for
   "arabic-isolated" and "arabic-final", the following example shows how
   to mark ARABIC LETTER ALEF WITH WAVY HAMZA BELOW (U+0673) as a
   variant of ARABIC LETTER ALEF WITH HAMZA BELOW (U+0625), but only
   when it appears in its isolated or final forms:

       <char cp="0625">
           <var cp="0673" when="arabic-isolated"/>
           <var cp="0673" when="arabic-final"/>
       </char>

Top      ToC       Page 21 
   While a "var" element MUST NOT contain multiple conditions (it is
   only allowed a single "when" or "not-when" attribute), multiple "var"
   elements using the same mapping MAY be specified with different
   "when" or "not-when" attributes.  The combination of mapping and
   conditional context defines a unique variant.

   For each variant label, care must be taken to ensure that at most one
   of the contextual conditions is met for variants with the same
   mapping; otherwise, duplicate variant labels would be created for the
   same input label.  Any such duplicate variant labels MUST be treated
   as an error; see Section 8.4.

   Two contexts may be complementary, as in the following example, which
   shows ARABIC LETTER TEH MARBUTA (U+0629) as a variant of ARABIC
   LETTER HEH (U+0647), but with two different types.

       <char cp="0647" >
         <var cp="0629" not-when="arabic-final" type="blocked" />
         <var cp="0629" when="arabic-final" type="allocatable" />
       </char>

   The intent is that a label that uses U+0629 instead of U+0647 in a
   final position should be considered essentially the same label and,
   therefore, allocatable to the same entity, while the same
   substitution in a non-final position leads to labels that are
   different, but considered confusable, so that either one, but not
   both, should be delegatable.

   For symmetry, the reverse mappings must exist and must agree in their
   "when" or "not-when" attributes.  However, symmetry does not apply to
   the other attributes.  For example, these are potential reverse
   mappings for the above:

       <char cp="0629" >
         <var cp="0647" not-when="arabic-final" type="allocatable" />
         <var cp="0647" when="arabic-final" type="allocatable" />
       </char>

   Here, both variants have the same "type" attribute.  While it is
   tempting to recognize that, in this instance, the "when" and
   "not-when" attributes are complementary; therefore, between them they
   cover every single possible context, it is strongly RECOMMENDED to
   use the format shown in the example that makes the symmetry easily
   verifiable by parsers and tools.  (The same applies to entries
   created for transitivity.)

Top      ToC       Page 22 
   Arabic is an example of a script for which such conditional variants
   have been implemented based on the joining contexts for Arabic code
   points.  The mechanism defined here supports other forms of
   conditional variants that may be required by other scripts.

5.4.  Annotations

   Two attributes, the "ref" and "comment" attributes, can be used to
   annotate individual elements in the LGR.  They are ignored in
   machine-processing of the LGR.  The "ref" attribute is intended for
   formal annotations and the "comment" attribute for free-form
   annotations.  The latter can be applied more widely.

5.4.1.  The "ref" Attribute

   Reference information MAY optionally be specified by a "ref"
   attribute consisting of a space-delimited sequence of reference
   identifiers (see Section 4.3.8).

       <char cp="5220" ref="0">
           <var cp="5220" ref="5"/>
           <var cp="522A" ref="2 3"/>
       </char>

   This facility is typically used to give source information for code
   points or variant relations.  This information is ignored when
   machine-processing an LGR.  If applied to a range, the "ref"
   attribute applies to every code point in the range.  All reference
   identifiers MUST be from the set declared in the "references" element
   (see Section 4.3.8).  It is an error to repeat a reference identifier
   in the same "ref" attribute.  It is RECOMMENDED that identifiers be
   listed in ascending order.

   In addition to "char", "range", and "var" elements in the "data"
   section, a "ref" attribute may be present for a number of element
   types contained in the "rules" element as described below: actions
   and literals ("char" inside a rule), as well as for definitions of
   rules and classes, but not for references to named character classes
   or rules using the "by-ref" attribute defined below.  (The use of the
   "by-ref" and "ref" attributes is mutually exclusive.)  None of the
   elements in the metadata take a "ref" attribute; to provide
   additional information, use the "description" element instead.

Top      ToC       Page 23 
5.4.2.  The "comment" Attribute

   Any "char", "range", or "variant" element in the "data" section may
   contain an OPTIONAL "comment" attribute.  The contents of a "comment"
   attribute are free-form plain text.  Comments are ignored in machine
   processing of the table.  "comment" attributes MAY also be placed on
   all elements in the "rules" section of the document, such as actions
   and match operators, as well as definitions of classes and rules, but
   not on child elements of the "class" element.  Finally, in the
   metadata, only the "version" and "reference" elements MAY have
   "comment" attributes (to match the syntax in [RFC3743]).

5.5.  Code Point Tagging

   Typically, LGRs are used to explicitly designate allowable code
   points, where any label that contains a code point not explicitly
   listed in the LGR is considered an ineligible label according to the
   ruleset.

   For more-complex registry rules, there may be a need to discern one
   or more subsets of code points.  This can be accomplished by applying
   an OPTIONAL "tag" attribute to "char" or "range" elements that are
   child elements of the "data" element.  By collecting code points that
   share the same tag value, character classes may be defined (see
   Section 6.2.2) that can then be used in parameterized context or
   whole label rules (see Section 6.3.2).

   Each "tag" attribute MAY contain multiple values separated by
   white space.  A tag value is an identifier that may also include
   certain punctuation marks, such as a colon.  Formally, it MUST
   correspond to the XML 1.0 Nmtoken (Name token) production (see [XML]
   Section 2.3).  It is an error to duplicate a value within the same
   "tag" attribute.  A "tag" attribute for a "range" element applies to
   all code points in the range.  Because code point sequences are not
   proper members of a set of code points, a "tag" attribute MUST NOT be
   present in a "char" element defining a code point sequence.



(page 23 continued on part 2)

Next Section