IDNA2008 mapping: Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008
How to normalize the characters a user types before they reach the IDNA2008 protocol.
Resnick, P. and P. Hoffman, "Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008", RFC 5895, DOI 10.17487/RFC5895, September 2010, <https://www.rfc-editor.org/info/rfc5895>.
@misc{rfc5895,
series = {Request for Comments},
number = 5895,
howpublished = {RFC 5895},
publisher = {RFC Editor},
doi = {10.17487/RFC5895},
url = {https://www.rfc-editor.org/info/rfc5895},
author = {P. Resnick and P. Hoffman},
title = {{Mapping Characters for Internationalized Domain Names in Applications (IDNA) 2008}},
pagetotal = 7,
year = 2010,
month = sep,
} RFC text
Contents
Independent Submission P. Resnick
Request for Comments: 5895 Qualcomm Incorporated
Category: Informational P. Hoffman
ISSN: 2070-1721 VPN Consortium
September 2010
Mapping Characters for
Internationalized Domain Names in Applications (IDNA) 2008
Abstract
In the original version of the Internationalized Domain Names in
Applications (IDNA) protocol, any Unicode code points taken from user
input were mapped into a set of Unicode code points that "made
sense", and then encoded and passed to the domain name system (DNS).
The IDNA2008 protocol (described in RFCs 5890, 5891, 5892, and 5893)
presumes that the input to the protocol comes from a set of
"permitted" code points, which it then encodes and passes to the DNS,
but does not specify what to do with the result of user input. This
document describes the actions that can be taken by an implementation
between receiving user input and passing permitted code points to the
new IDNA protocol.
Status of This Memo
This document is not an Internet Standards Track specification; it is
published for informational purposes.
This is a contribution to the RFC Series, independently of any other
RFC stream. The RFC Editor has chosen to publish this document at
its discretion and makes no statement about its value for
implementation or deployment. Documents approved for publication by
the RFC Editor are not a candidate for any level of Internet
Standard; see Section 2 of RFC 5741.
Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc5895.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document.
1. Introduction
This document describes the operations that can be applied to user
input in order to get it into a form that is acceptable by the
Internationalized Domain Names in Applications (IDNA) protocol
[IDNA2008protocol]. It includes a general implementation procedure
for mapping.
It should be noted that this document does not specify the behavior
of a protocol that appears "on the wire". It describes an operation
that is to be applied to user input in order to prepare that user
input for use in an "on the network" protocol. As unusual as this
may be for a document concerning Internet protocols, it is necessary
to describe this operation for implementors who may have designed
around the original IDNA protocol (herein referred to as IDNA2003),
which conflates this user-input operation into the protocol.
It is very important to note that there are many potential valid
mappings of characters from user input. The mapping described in
this document is the basis for other mappings, and is not likely to
be useful without modification. Any useful mapping will have
features designed to reduce the surprise for users and is likely to
be slightly (or sometimes radically) different depending on the
locale of the user, the type of input being used (such as typing,
copy-and-paste, voice, and so on), the type of application used, etc.
Although most common mappings will probably produce similar results
for the same input, there will be subtle differences between
applications.
1.1. The Dividing Line between User Interface and Protocol
The user interface to applications is much more complicated than most
network implementers think. When we say "the user enters an
internationalized domain name in the application", we are talking
about a very complex process that encompasses everything from the
user formulating the name and deciding which symbols to use to
express that name, to the user entering the symbols into the computer
using some input method (be it a keyboard, a stylus, or even a voice
recognition program), to the computer interpreting that input (be it
keyboard scan codes, a graphical representation, or digitized sounds)
into some representation of those symbols, through finally
normalizing those symbols into a particular character repertoire in
an encoding recognizable to IDNA processes and the domain name
system.
Considerations for a user interface for internationalized domain
names involves taking into account culture, context, and locale for
any given user. A simple and well-known example is the lowercasing
of the letter LATIN CAPITAL LETTER I (U+0049) when it is used in the
Turkish and other languages. A capital "I" in Turkish is properly
lowercased to a LATIN SMALL LETTER DOTLESS I (U+0131), not to a LATIN
SMALL LETTER I (U+0069). This lowercasing is clearly dependent on
the locale of the system and/or the locale of the user. Using a
single context-free mapping without considering the user interface
properties has the potential of doing exactly the wrong thing for the
user.
The original version of IDNA conflated user interface processing and
protocol. It took whatever characters the user produced in whatever
encoding the application used, assumed some conversion to Unicode
code points, and then without regard to context, locale, or anything
about the user's intentions, mapped them into a particular set of
other characters, and then re-encoded them in Punycode, in order to
have the entire operation be contained within the protocol. Ignoring
context, locale, and user preference in the IDNA protocol made life
significantly less complicated for the application developer, but at
the expense of violating the principle of "least user surprise" for
consumers and producers of domain names.
In IDNA2008, the dividing line between "user interface" and
"protocol" is clear. The IDNA2008 specification defines the protocol
part of IDNA: it explicitly does not deal with the user interface.
Mappings such as the one described in this document explicitly deal
with the user interface and not the protocol. That is, a mapping is
only to be applied before a string of characters is treated as a
domain name (in the "user interface") and is never to be applied
during domain name processing (in the "protocol").
1.2. The Design of This Mapping
The user interface mapping in this document is a set of expansions to
IDNA2008 that are meant to be sensible and friendly and mostly
obvious to people throughout the world when using typical
applications with domain names that are entered by hand. It is also
designed to let applications be mostly backwards compatible with
IDNA2003. By definition, it cannot meet all of those design goals
for all people, and in fact is known to fail on some of those goals
for quite large populations of people.
A good mapping in the real world might use the "sensible and friendly
and mostly obvious" design goal but come up with a different
algorithm. Many algorithms will have results that are close to what
is described here, but will differ in assumptions about the users'
way of thinking or typing. Having said that, it is likely that some
mappings will be significantly different. For example, a mapping
might apply to a spoken user interface instead of a typed one.
Another example is that a mapping might be different for users that
are typing than for users that are copying-and-pasting from different
applications. Yet another example is that a user interface that
allows typed input that is transliterated from Latin characters could
have very different mappings than one that applies to typing in other
character sets; this would be typical in a Pinyin input method for
Chinese characters.
2. The General Procedure
This section defines a general algorithm that applications ought to
implement in order to produce Unicode code points that will be valid
under the IDNA protocol. An application might implement the full
mapping as described below, or it can choose a different mapping.
This mapping is very general and was designed to be acceptable to the
widest user community, but as stated above, it does not take into
account any particular context, culture, or locale.
The general algorithm that an application (or the input method
provided by an operating system) ought to use is relatively
straightforward:
1. Uppercase characters are mapped to their lowercase equivalents by
using the algorithm for mapping case in Unicode characters. This
step was chosen because the output will behave more like ASCII
host names behave.
2. Fullwidth and halfwidth characters (those defined with
Decomposition Types <wide> and <narrow>) are mapped to their
decomposition mappings as shown in the Unicode character
database. This step was chosen because many input mechanisms,
particularly in Asia, do not allow you to easily enter characters
in the form used by IDNA2008. Even if they do allow the correct
character form, the user might not know which form they are
entering.
3. All characters are mapped using Unicode Normalization Form C
(NFC). This step was chosen because it maps combinations of
combining characters into canonical composed form. As with the
fullwidth/halfwidth mapping, users are not generally aware of the
particular form of characters that they are entering, and
IDNA2008 requires that only the canonical composed forms from NFC
be used.
4. [IDNA2008protocol] is specified such that the protocol acts on
the individual labels of the domain name. If an implementation
of this mapping is also performing the step of separation of the
parts of a domain name into labels by using the FULL STOP
character (U+002E), the IDEOGRAPHIC FULL STOP character (U+3002)
can be mapped to the FULL STOP before label separation occurs.
There are other characters that are used as "full stops" that one
could consider mapping as label separators, but their use as such
has not been investigated thoroughly. This step was chosen
because some input mechanisms do not allow the user to easily
enter proper label separators. Only the IDEOGRAPHIC FULL STOP
character (U+3002) is added in this mapping because the authors
have not fully investigated the applicability of other characters
and the environments where they should and should not be
considered domain name label separators.
Note that the steps above are ordered.
Definitions for the rules in this algorithm can be found in
[Unicode52]. Specifically:
o Unicode Normalization Form C can be found in Annex #15 of
[Unicode-UAX15].
o In order to map uppercase characters to their lowercase
equivalents (defined in Section 3.13 of [Unicode52]), first map
characters to the "Lowercase_Mapping" property (the "<lower>"
entry in the second column) in
<http://www.unicode.org/Public/UNIDATA/SpecialCasing.txt>, if any.
Then, map characters to the "Simple_Lowercase_Mapping" property
(the fourteenth column) in
<http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>, if any.
o In order to map fullwidth and halfwidth characters to their
decomposition mappings, map any character whose
"Decomposition_Type" (contained in the first part of the sixth
column) in <http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>
is either "<wide>" or "<narrow>" to the "Decomposition_Mapping" of
that character (contained in the second part of the sixth column)
in <http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>.
o The Unicode Character Database [TR44] has useful descriptions of
the contents of these files.
If the mappings in this document are applied to versions of Unicode
later than Unicode 5.2, the later versions of the Unicode Standard
should be consulted.
These form a minimal set of mappings that an application should
strongly consider doing. Of course, there are many others that might
be done.
3. Implementing This Mapping
If you are implementing a mapping for an application or operating
system by using exactly the four steps in Section 2, the authors of
this document have a request: please don't. We mean it. Section 2
does not describe a universal mapping algorithm because, as we said,
there is no universally-applicable mapping algorithm.
If you read the material in Section 2 without reading Section 1, go
back and carefully read all of Section 1; in many ways, Section 1 is
more important than Section 2. Further, you can probably think of
user interface considerations that we did not list in Section 1. If
you did read Section 1 but somehow decided that the algorithm in
Section 2 is completely correct for the intended users of your
application or operating system, you are probably not thinking hard
enough about your intended users.
4. Security Considerations
This document suggests creating mappings that might cause confusion
for some users while alleviating confusion in other users. Such
confusion is not covered in any depth in this document (nor in the
other IDNA-related documents).
5. Acknowledgements
This document is the product of many contributions from numerous
people in the IETF.
6. Normative References
[IDNA2008protocol] Klensin, J., "Internationalized Domain Names in
Applications (IDNA): Protocol", RFC 5891,
August 2010.
[TR44] The Unicode Consortium, "Unicode Technical Report
#44: Unicode Character Database", September 2009,
<http://www.unicode.org/reports/tr44/
tr44-4.html>.
[Unicode-UAX15] The Unicode Consortium, "Unicode Standard Annex
#15: Unicode Normalization Forms, Revision 31",
September 2009, <http://www.unicode.org/reports/
tr15/tr15-31.html>.
[Unicode52] The Unicode Consortium. The Unicode Standard,
Version 5.2.0, defined by: "The Unicode Standard,
Version 5.2.0", (Mountain View, CA: The Unicode
Consortium, 2009. ISBN 978-1-936213-00-9).
<http://www.unicode.org/versions/Unicode5.2.0/>.
Authors' Addresses
Peter W. Resnick
Qualcomm Incorporated
5775 Morehouse Drive
San Diego, CA 92121-1714
US
Phone: +1 858 651 4478
EMail: presnick@qualcomm.com
URI: http://www.qualcomm.com/~presnick/
Paul Hoffman
VPN Consortium
127 Segre Place
Santa Cruz, CA 95060
US
Phone: 1-831-426-9827
EMail: paul.hoffman@vpnc.org
ISO Plain Language
On this page
§1What This Document Does and Does Not Cover
In the original version of the Internationalized Domain Names in Applications (IDNA) protocol, any Unicode code points taken from user input were mapped into a set of Unicode code points that made sense, then encoded and passed to the Domain Name System (DNS). The IDNA2008 protocol, described in RFCs 5890, 5891, 5892 and 5893, presumes that the input already comes from a set of permitted code points, which it then encodes and passes to the DNS, but does not specify what to do with the result of user input. This document describes the actions an implementation can take between receiving user input and passing permitted code points to the new protocol.
This is not a protocol that appears on the wire. It describes an operation applied to user input to prepare that input for use in an on-the-network protocol. The document says this is unusual for an Internet protocol document, and explains that it is needed for implementors who may have designed around the original IDNA protocol, referred to here as IDNA2003, which folds this user-input operation into the protocol itself.
Many valid mappings of characters from user input are possible. The mapping described here is the basis for other mappings and is not likely to be useful without modification. A useful mapping has features designed to reduce the surprise for users, and is likely to differ depending on the locale of the user, the type of input being used, such as typing, copy-and-paste or voice, and the type of application used. Most common mappings will probably produce similar results for the same input, but there will be subtle differences between applications.
IDNA2008 only accepts characters from an approved list, and it leaves it to the application to get user input onto that list. This memo describes how to do that, and is clear that the description is a starting point rather than a finished answer, because people type in different ways and expect different results.
§1.1The Line Between User Interface and Protocol
The user interface of an application is more complicated than most network implementers think. When a user enters an internationalized domain name, that covers the whole chain: the user forming the name and choosing symbols to express it, entering those symbols with a keyboard, stylus or voice recognition program, the computer interpreting the input as scan codes, a graphical representation or digitized sounds, and finally normalizing those symbols into a character repertoire in an encoding that IDNA processes and the DNS recognise.
A user interface for internationalized domain names has to take account of culture, context and locale. The document gives a well-known example: lowercasing the letter LATIN CAPITAL LETTER I (U+0049). In Turkish and other languages a capital I lowercases to LATIN SMALL LETTER DOTLESS I (U+0131), not to LATIN SMALL LETTER I (U+0069). That lowercasing depends on the locale of the system or of the user, so a single context-free mapping has the potential of doing exactly the wrong thing.
The original IDNA conflated user interface processing and protocol. It took whatever characters the user produced in whatever encoding the application used, assumed a conversion to Unicode code points, and then, without regard to context, locale or the user's intentions, mapped them into a particular set of other characters and re-encoded them in Punycode, so the whole operation sat inside the protocol. Ignoring context, locale and user preference made life simpler for the application developer, at the cost of violating the principle of least user surprise for consumers and producers of domain names.
In IDNA2008 the dividing line is clear. The specification defines the protocol part and explicitly does not deal with the user interface. A mapping such as the one here deals with the user interface and not the protocol. A mapping is only applied before a string of characters is treated as a domain name, in the user interface, and is never applied during domain name processing, in the protocol.
Typing a name is not one step. The person chooses the symbols, an input method turns them into characters, and the application then has to tidy the result. Where that tidying happens matters: doing it inside the protocol, as the original IDNA did, forces one rule on every language and every input method, and the wrong rule is as likely as the right one.
§2The Four Ordered Mapping Steps
The general algorithm an application, or the input method provided by an operating system, ought to use has four steps, and they are applied in order.
First, uppercase characters are mapped to their lowercase equivalents using the algorithm for mapping case in Unicode characters. The step was chosen because the output behaves more like ASCII host names behave.
Second, fullwidth and halfwidth characters, those defined with Decomposition Types <wide> and <narrow>, are mapped to their decomposition mappings as shown in the Unicode character database. The step was chosen because many input mechanisms, particularly in Asia, do not allow the user to enter characters in the form IDNA2008 requires, and because even where they do, the user may not know which form they are entering.
Third, all characters are mapped using Unicode Normalization Form C (NFC). The step was chosen because it maps combinations of combining characters into canonical composed form. Users are not generally aware of the particular form of the characters they are entering, and IDNA2008 requires that only the canonical composed forms from NFC be used.
Fourth, IDNA2008 acts on the individual labels of the domain name. An implementation of this mapping that also performs the separation of the parts of a domain name into labels using the FULL STOP character (U+002E) can map the IDEOGRAPHIC FULL STOP character (U+3002) to the FULL STOP before label separation occurs. Other characters used as full stops could be considered for mapping as label separators, but their use as such has not been investigated thoroughly. Only U+3002 is added, because the authors have not fully investigated the applicability of other characters and the environments in which they should and should not be considered label separators. The step was chosen because some input mechanisms do not allow the user to enter proper label separators.
Definitions for the rules in the algorithm are found in the Unicode Standard, Version 5.2.0. Unicode Normalization Form C is in Annex 15 of the Unicode Standard Annex #15. To map uppercase characters to their lowercase equivalents, first map characters to the Lowercase_Mapping property if any, then map characters to the Simple_Lowercase_Mapping property if any. To map fullwidth and halfwidth characters to their decomposition mappings, map any character whose Decomposition_Type is either <wide> or <narrow> to the Decomposition_Mapping of that character. If the mappings in this document are applied to versions of Unicode later than 5.2, the later versions of the Unicode Standard should be consulted. These form a minimal set of mappings an application should strongly consider doing, and there are many others that might be done.
Four steps, in this order: fold capitals to lowercase, widen out fullwidth and halfwidth characters, put combining marks back into one composed character, and treat the ideographic full stop as an ordinary dot before the name is split into labels. Each step exists because a real input method produces a form the protocol would otherwise reject, and the user cannot tell which form they typed.
- 1Uppercase to lowercase — So the output behaves more like ASCII host names behave.
- 2Fullwidth and halfwidth to decomposition — Many input mechanisms do not let the user enter the form IDNA2008 requires.
- 3Normalization Form C — Combines combining characters into canonical composed form, which IDNA2008 requires.
- 4Ideographic full stop to full stop — Some input mechanisms do not let the user enter a proper label separator. Applied before label separation.
§3Implementing the Mapping in an Application
If you are implementing a mapping for an application or operating system by using exactly the four steps above, the authors have a request: please do not. They mean it. The four steps do not describe a universal mapping algorithm, because there is no universally applicable mapping algorithm.
If you read the material in the algorithm without reading the introduction, the authors ask you to go back and read it carefully, because in many ways the introduction is more important. Further, you can probably think of user interface considerations that the document did not list. And if you read the introduction and decided that the algorithm is completely correct for the intended users of your application or operating system, the authors say you are probably not thinking hard enough about your intended users.
The mapping described here was designed to be sensible and friendly and mostly obvious to people throughout the world when using typical applications with domain names entered by hand, and to let applications be mostly backwards compatible with IDNA2003. By definition it cannot meet all of those design goals for all people, and it is known to fail on some of them for quite large populations of people.
A good mapping in the real world might use the same design goal but a different algorithm. Many algorithms will produce results close to what is described, but will differ in assumptions about how users think and type. A mapping might apply to a spoken user interface rather than a typed one. A mapping might be different for users typing than for users copying and pasting from other applications. And a user interface that accepts typed input transliterated from Latin characters could have very different mappings from one that applies to typing in other character sets, which would be typical of a Pinyin input method for Chinese characters.
The authors are unusually blunt: do not implement these four steps as they stand. The steps are a floor, not a ceiling. An application has to know who its users are, because the same typed characters mean different things in different languages, and a copy-and-paste action delivers a different string than typing does.
§4Security Considerations
This document suggests creating mappings that might cause confusion for some users while alleviating confusion in other users. Such confusion is not covered in any depth in this document, nor in the other IDNA-related documents.
Any mapping trades one kind of confusion for another. Two users who mean the same name can end up with different strings, and two different names can end up looking the same. The document says it does not cover this in depth.