Internet Engineering Task Force (IETF) H. Alvestrand, Ed. Request for Comments: 5893 Google Category: Standards Track C. Karp ISSN: 2070-1721 Swedish Museum of Natural History August 2010 Right-to-Left Scripts for Internationalized Domain Names for Applications (IDNA)
AbstractThe use of right-to-left scripts in Internationalized Domain Names (IDNs) has presented several challenges. This memo provides a new Bidi rule for Internationalized Domain Names for Applications (IDNA) labels, based on the encountered problems with some scripts and some shortcomings in the 2003 IDNA Bidi criterion. Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 5741. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc5893. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Purpose and Applicability . . . . . . . . . . . . . . . . 2 1.2. Background and History . . . . . . . . . . . . . . . . . . 3 1.3. Structure of the Rest of This Document . . . . . . . . . . 3 1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 2. The Bidi Rule . . . . . . . . . . . . . . . . . . . . . . . . 6 3. The Requirement Set for the Bidi Rule . . . . . . . . . . . . 6 4. Examples of Issues Found with RFC 3454 . . . . . . . . . . . . 9 4.1. Dhivehi . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.2. Yiddish . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.3. Strings with Numbers . . . . . . . . . . . . . . . . . . . 12 5. Troublesome Situations and Guidelines . . . . . . . . . . . . 12 6. Other Issues in Need of Resolution . . . . . . . . . . . . . . 13 7. Compatibility Considerations . . . . . . . . . . . . . . . . . 14 7.1. Backwards Compatibility Considerations . . . . . . . . . . 14 7.2. Forward Compatibility Considerations . . . . . . . . . . . 15 8. Security Considerations . . . . . . . . . . . . . . . . . . . 15 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 16 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 16 10.1. Normative References . . . . . . . . . . . . . . . . . . . 16 10.2. Informative References . . . . . . . . . . . . . . . . . . 17 RFC5891]. When labels satisfy the rule, and when certain other conditions are satisfied, there is only a minimal chance of these labels being displayed in a confusing way by the Unicode bidirectional display algorithm. The other normative documents in the IDNA2008 document set establish criteria for valid labels, including listing the permitted characters. This document establishes additional validity criteria for labels in scripts normally written from right to left. This specification is not intended to place any requirements on domain names that do not contain characters from such scripts.
RFC3454], part of IDNA2003, made the following statement in its Section 6 on the Bidi algorithm: 3) If a string contains any RandALCat character, a RandALCat character MUST be the first character of the string, and a RandALCat character MUST be the last character of the string. (A RandALCat character is a character with unambiguously right-to-left directionality.) The reasoning behind this prohibition was to ensure that every component of a displayed domain name has an unambiguously preferred direction. However, this made certain words in languages written with right-to-left scripts invalid as IDN labels, and in at least one case (Dhivehi) meant that all the words of an entire language were forbidden as IDN labels. This is illustrated below with examples taken from the Dhivehi and Yiddish languages, as written with the Thaana and Hebrew scripts, respectively. RFC 3454 did not explicitly state the requirement to be fulfilled. Therefore, it is impossible to determine whether a simple relaxation of the rule would continue to fulfill the requirement. While this document specifies rules quite different from RFC 3454, most reasonable labels that were allowed under RFC 3454 will also be allowed under this specification (the most important example of non-permitted labels being labels that mix Arabic and European digits (AN and EN) inside an RTL label, and labels that use AN in an LTR label -- see Section 1.4 for terminology), so the operational impact of using the new rule in the updated IDNA specification is limited. Section 2 defines a rule, the "Bidi rule", which can be used on a domain name label to check how safe it is to use in a domain name of possibly mixed directionality. The primary initial use of this rule is as part of the IDNA2008 protocol [RFC5891]. Section 3 sets out the requirements for defining the Bidi rule. Section 4 gives detailed examples that serve as justification for the new rule.
Section 5 to Section 8 describe various situations that can occur when dealing with domain names with characters of different directionality. Only Section 1.4 and Section 2 are normative. RFC5890]. The terminology used for the Bidi properties of Unicode characters is taken from the Unicode Standard [Unicode52]. The Unicode Standard specifies a Bidi property for each character. That property controls the character's behavior in the Unicode bidirectional algorithm [Unicode-UAX9]. For reference, here are the values that the Unicode Bidi property can have: o L - Left to right - most letters in LTR scripts o R - Right to left - most letters in non-Arabic RTL scripts o AL - Arabic letters - most letters in the Arabic script o EN - European Number (0-9, and Extended Arabic-Indic numbers) o ES - European Number Separator (+ and -) o ET - European Number Terminator (currency symbols, the hash sign, the percent sign and so on) o AN - Arabic Number; this encompasses the Arabic-Indic numbers, but not the Extended Arabic-Indic numbers o CS - Common Number Separator (. , / : et al) o NSM - Nonspacing Mark - most combining accents o BN - Boundary Neutral - control characters (ZWNJ, ZWJ, and others) o B - Paragraph Separator o S - Segment Separator o WS - Whitespace, including the SPACE character o ON - Other Neutrals, including @, &, parentheses, MIDDLE DOT
o LRE, LRO, RLE, RLO, PDF - these are "directional control characters" and are not used in IDNA labels. In this memo, we use "network order" to describe the sequence of characters as transmitted on the wire or stored in a file; the terms "first", "next", "previous", "beginning", "end", "before", and "after" are used to refer to the relationship of characters and labels in network order. We use "display order" to talk about the sequence of characters as imaged on a display medium; the terms "left" and "right" are used to refer to the relationship of characters and labels in display order. Most of the time, the examples use the abbreviations for the Unicode Bidi classes to denote the directionality of the characters; the example string CS L consists of one character of class CS and one character of class L. In some examples, the convention that uppercase characters are of class R or AL, and lowercase characters are of class L is used -- thus, the example string ABC.abc would consist of three right-to-left characters and three left-to-right characters. The directionality of such examples is determined by context -- for instance, in the sentence "ABC.abc is displayed as CBA.abc", the first example string is in network order, the second example string is in display order. The term "paragraph" is used in the sense of the Unicode Bidi specification [Unicode-UAX9]. It means "a block of text that has an overall direction, either left to right or right to left", approximately; see the "Unicode Bidirectional Algorithm" [Unicode-UAX9] for details. "RTL" and "LTR" are abbreviations for "right to left" and "left to right", respectively. An RTL label is a label that contains at least one character of type R, AL, or AN. An LTR label is any label that is not an RTL label. A "Bidi domain name" is a domain name that contains at least one RTL label. (Note: This definition includes domain names containing only dots and right-to-left characters. Providing a separate category of "RTL domain names" would not make this specification simpler, so it has not been done.)
Section 3. All of the conditions must be satisfied for the rule to be satisfied. 1. The first character must be a character with Bidi property L, R, or AL. If it has the R or AL property, it is an RTL label; if it has the L property, it is an LTR label. 2. In an RTL label, only characters with the Bidi properties R, AL, AN, EN, ES, CS, ET, ON, BN, or NSM are allowed. 3. In an RTL label, the end of the label must be a character with Bidi property R, AL, EN, or AN, followed by zero or more characters with Bidi property NSM. 4. In an RTL label, if an EN is present, no AN may be present, and vice versa. 5. In an LTR label, only characters with the Bidi properties L, EN, ES, CS, ET, ON, BN, or NSM are allowed. 6. In an LTR label, the end of the label must be a character with Bidi property L or EN, followed by zero or more characters with Bidi property NSM. The following guarantees can be made based on the above: o In a domain name consisting of only labels that satisfy the rule, the requirements of Section 3 are satisfied. Note that even LTR labels and pure ASCII labels have to be tested. o In a domain name consisting of only LDH labels (as defined in the Definitions document [RFC5890]) and labels that satisfy the rule, the requirements of Section 3 are satisfied as long as a label that starts with an ASCII digit does not come after a right-to-left label. No guarantee is given for other combinations. RFC3454], provides an explicit justification for the Bidi rule, and states a set of requirements for which it is possible to test whether or not the modified rule fulfills the requirement.
All the text in this document assumes that text containing the labels under consideration will be displayed using the Unicode bidirectional algorithm [Unicode-UAX9]. The requirements proposed are these: o Label Uniqueness: No two labels, when presented in display order in the same paragraph, should have the same sequence of characters without also having the same sequence of characters in network order, both when the paragraph has LTR direction and when the paragraph has RTL direction. (This is the criterion that is explicit in RFC 3454). (Note that a label displayed in an RTL paragraph may display the same as a different label displayed in an LTR paragraph and still satisfy this criterion.) o Character Grouping: When displaying a string of labels, using the Unicode Bidi algorithm to reorder the characters for display, the characters of each label should remain grouped between the characters delimiting the labels, both when the string is embedded in a paragraph with LTR direction and when it is embedded in a paragraph with RTL direction. Several stronger statements were considered and rejected, because they seem to be impossible to fulfill within the constraints of the Unicode bidirectional algorithm. These include: o The appearance of a label should be unaffected by its embedding context. This proved impossible even for ASCII labels; the label "123-A" will have a different display order in an RTL context than in an LTR context. (This particular example is, however, disallowed anyway.) o The sequence of labels should be consistent with network order. This proved impossible -- a domain name consisting of the labels (in network order) L1.R2.R3.L4 will be displayed as L1.R3.R2.L4 in an LTR context. (In an RTL context, it will be displayed as L4.R3.R2.L1). o No two domain names should be displayed the same, even under differing directionality. This was shown to be unsound, since the domain name (in network order) ABC.abc will have display order CBA.abc in an LTR context and abc.CBA in an RTL context, while the domain name (network) abc.ABC will have display order abc.CBA in an LTR context and CBA.abc in an RTL context.
One possible requirement was thought to be problematic, but turned out to be satisfied by a string that obeys the proposed rules: o The Character Grouping requirement should be satisfied when directional controls (LRE, RLE, RLO, LRO, PDF) are used in the same paragraph (outside of the labels). Because these controls affect presentation order in non-obvious ways, by affecting the "sor" and "eor" properties of the Unicode Bidi algorithm, the conditions above require extra testing in order to figure out whether or not they influence the display of the domain name. Testing found that for the strings allowed under the rule presented in this document, directional controls do not influence the display of the domain name. This is still not stated as a requirement, since it did not seem as important as the stated requirements, but it is useful to know that Bidi domain names where the labels satisfy the rule have this property. In the following descriptions, first-level bullets are used to indicate rules or normative statements; second-level bullets are commentary. The Character Grouping requirement can be more formally stated as: o Let "Delimiterchars" be a set of characters with the Unicode Bidi properties CS, WS, ON. (These are commonly used to delimit labels -- both the FULL STOP and the space are included. They are not allowed in domain labels.) * ET, though it commonly occurs next to domain names in practice, is problematic: the context R CS L EN ET (for instance A.a1%) makes the label L EN not satisfy the character grouping requirement. * ES commonly occurs in labels as HYPHEN-MINUS, but could also be used as a delimiter (for instance, the plus sign). It is left out here. o Let "unproblematic label" be a label that either satisfies the requirements or does not contain any character with the Bidi properties R, AL, or AN and does not begin with a character with the Bidi property EN. (Informally, "it does not start with a number".)
A label X satisfies the Character Grouping requirement when, for any Delimiter Character D1 and D2, and for any label S1 and S2 that is an unproblematic label or an empty string, the following holds true: If the string formed by concatenating S1, D1, X, D2, and S2 is reordered according to the Bidi algorithm, then all the characters of X in the reordered string are between D1 and D2, and no other characters are between D1 and D2, both if the overall paragraph direction is LTR and if the overall paragraph direction is RTL. Note that the definition is self-referential, since S1 and S2 are constrained to be "legal" by this definition. This makes testing changes to proposed rules a little complex, but does not create problems for testing whether or not a given proposed rule satisfies the criterion. The "zero-length" case represents the case where a domain name is next to something that isn't a domain name, separated by a delimiter character. Note about the position of BN: The Unicode bidirectional algorithm specifies that a BN has an effect on the adjoining characters in network order, not in display order, and are therefore treated as if removed during Bidi processing ([Unicode-UAX9], Section 3.3.2, rule X9 and Section 5.3). Therefore, the question of "what position does a BN have after reordering" is not meaningful. It has been ignored while developing the rules here. The Label Uniqueness requirement can be formally stated as: If two non-identical labels X and Y, embedded as for the test above, displayed in paragraphs with the same directionality, are reordered by the Bidi algorithm into the same sequence of code points, the labels X and Y cannot both be legal.
The word for "computer", which is romanized as "konpeetaru", is written with the following sequence of Unicode code points: U+0786 THAANA LETTER KAAFU (AL) U+07AE THAANA OBOFILI (NSM) U+0782 THAANA LETTER NOONU (AL) U+07B0 THAANA SUKUN (NSM) U+0795 THAANA LETTER PAVIYANI (AL) U+07A9 THAANA LETTER EEBEEFILI (AL) U+0793 THAANA LETTER TAVIYANI (AL) U+07A6 THAANA ABAFILI (NSM) U+0783 THAANA LETTER RAA (AL) U+07AA THAANA UBUFILI (NSM) The directionality class of U+07AA in the Unicode database [Unicode52] is NSM (Nonspacing Mark), which is not R or AL; a conformant implementation of the IDNA2003 algorithm will say that "this is not in RandALCat" and refuse to encode the string.
Yiddish Orthography: Rules of Yiddish Spelling" [SYO], and are taken as normatively descriptive of modern Standard Yiddish in any context where that notion is deemed relevant. They have been applied exclusively in all formal Yiddish dictionaries published since their establishment, and are similarly dominant in academic and bibliographic regards. It therefore appears appropriate for this repertoire also to be supported fully by IDNA. This presents no difficulty with characters in initial and medial positions, but pointed characters are regularly used in final position as well. All of the characters in the SYO repertoire appear in both marked and unmarked form with one exception: the HEBREW LETTER PE (U+05E4). The SYO only permits this with a HEBREW POINT DAGESH (U+05BC), providing the Yiddish equivalent to the Latin letter "p", or a HEBREW POINT RAFE (U+05BF), equivalent to the Latin letter "f". There is, however, a separate unpointed allograph, the HEBREW LETTER FINAL PE (U+05E3), for the latter character when it appears in final position. The constraint on the use of the SYO repertoire resulting from the proscription of combining marks at the end of RTL strings thus reduces to nothing more, or less, than the equivalent of saying that a string of Latin characters cannot end with the letter "p". It must also be noted that the HEBREW LETTER PE with the HEBREW POINT DAGESH is characteristic of almost all traditional Yiddish orthographies that predate (or remain in use in parallel to) the SYO, being the first pointed character to appear in any of them. A more general instantiation of the basic problem can be seen in the representation of the YIVO acronym. This acronym is written with the Hebrew letters YOD YOD HIRIQ VAV VAV ALEF QAMATS, where HIRIQ and QAMATS are combining points. The Unicode code points are: U+05D9 HEBREW LETTER YOD (R) U+05B4 HEBREW POINT HIRIQ (NSM) U+05D5 HEBREW LETTER VAV (R) U+05D0 HEBREW LETTER ALEF (R) U+05B8 HEBREW POINT QAMATS (NSM) The directionality class of U+05B8 HEBREW POINT QAMATS in the Unicode database is NSM, which again causes the IDNA2003 algorithm to reject the string.
It may also be noted that all of the combined characters mentioned above exist in precomposed form at separate positions in the Unicode chart. However, by invoking Stringprep, the IDNA2003 algorithm also rejects those code points, for reasons not discussed here. RFC3454] prohibited a string containing right-to-left characters from ending with a number. Consider the strings ALEF 5 (HEBREW LETTER ALEF + DIGIT FIVE) and 5 ALEF. Displayed in an LTR context, the first one will be displayed from left to right as 5 ALEF (with the 5 being considered right to left because of the leading ALEF), while 5 ALEF will be displayed in exactly the same order (5 taking the direction from context). Clearly, only one of those should be permitted as a registered label, but barring them both seems unnecessary.
o Even if a label is registered under a "safe" label, there may be a DNAME [RFC2672] with an "unsafe" label that points to the "safe" label, thus creating seemingly valid names that would not satisfy the criterion. o Wildcards create the odd situation where a label is "valid" (can be looked up successfully) without the zone owner knowing that this label exists. So an owner of a zone whose name starts with a digit and contains a wildcard has no way of controlling whether or not names with RTL labels in them are looked up in his zone. Rather than trying to suggest rules that disallow all such undesirable situations, this document merely warns about the possibility, and leaves it to application developers to take whatever measures they deem appropriate to avoid problematic situations.
displayed in the order 2R.1R.ltr in an LTR context, which might surprise someone expecting to see labels displayed in hierarchical order. People used to working with text that mixes LTR and RTL strings might not be so surprised by this. Again, this memo does not attempt to suggest a solution to this problem. RFC 3454, some labels will not be allowed, and domain names containing those labels will remain inaccessible. o An old program is asked to display the newly allowed label, and checks it against RFC 3454 before displaying. The program will perform some kind of fallback, most likely displaying the label in A-label form. o An old program tries to display the newly allowed label. If the old program has code for displaying the last character of a label that is different from the code used to display the characters in the middle of the label, the display may be inconsistent and cause confusion. One particular example of the last case is if a program chooses to examine the last character (in network order) of a string in order to determine its directionality, rather than its first. If it finds an NSM character and tries to display the string as if it was a left-to-right string, the resulting display may be interesting, but not useful. The editors believe that these cases will have a less harmful impact in practice than continuing to deny the use of words from the languages for which these strings are necessary as IDN labels. This specification does not forbid using leading European digits in ASCII-only labels, since this would conflict with a large installed base of such labels, and would increase the scope of the specification from RTL labels to all labels. The harm resulting from this limitation of scope is described in Section 5. Registries and private zone managers can check for this particular condition before they allow registration of any RTL label. Generally, it is best to
disallow registration of any right-to-left strings in a zone where the label at the level above begins with a digit. RFC 3454 IDNA prohibitions, it is possible that the potential problems noted under "Backwards Compatibility Considerations" could cause new kinds of confusion.
Section 2 was suggested by Matitiahu Allouche. The team wishes, in particular, to thank Roozbeh Pournader for calling its attention to the issue with the Thaana script, Paul Hoffman for pointing out the need to be explicit about backwards compatibility considerations, Ken Whistler for suggesting the basis of the formalized "Character Grouping" requirement, Mark Davis for commentary, Erik van der Poel for careful review, comments, and verification of the rulesets, Marcos Sanz, Andrew Sullivan, and Pete Resnick for reviews, and Vint Cerf for chairing the working group and contributing massively to getting the documents finished. [RFC5890] Klensin, J., "Internationalized Domain Names for Applications (IDNA): Definitions and Document Framework", RFC 5890, August 2010. [Unicode-UAX9] The Unicode Consortium, "Unicode Standard Annex #9: Unicode Bidirectional Algorithm", September 2009, <http://www.unicode.org/reports/tr9/>. [Unicode52] The Unicode Consortium. The Unicode Standard, Version 5.2.0, defined by: "The Unicode Standard, Version 5.2.0", (Mountain View, CA: The Unicode Consortium, 2009. ISBN 978-1-936213-00-9). <http://www.unicode.org/versions/Unicode5.2.0/>.
[RFC2672] Crawford, M., "Non-Terminal DNS Name Redirection", RFC 2672, August 1999. [RFC3454] Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, December 2002. [RFC5891] Klensin, J., "Internationalized Domain Names in Applications (IDNA): Protocol", RFC 5891, August 2010. [SYO] "The Standardized Yiddish Orthography: Rules of Yiddish Spelling, 6th ed., New York, ISBN 0-914512-25-0", 1999.