As per Relevance of the word standard, we have this rfc below:
Network Working Group F.
Request for Comments: 2279 Alis
Obsoletes: 2044 January 1998
Category: Standards
UTF-8, a transformation format of ISO 10646
Status of this
This document specifies an Internet standards track protocol for
Internet community, and requests discussion and suggestions
improvements. Please refer to the current edition of the "
Official Protocol Standards" (STD 1) for the standardization
and status of this protocol. Distribution of this memo is unlimited
Copyright
Copyright (C) The Internet Society (1998). All Rights Reserved
ISO/IEC 10646-1 defines a multi-octet character set called
Universal Character Set (UCS) which encompasses most of the world'
writing systems. Multi-octet characters, however, are not
with many current applications and protocols, and this has led to
development of a few so-called UCS transformation formats (UTF),
with different characteristics. UTF-8, the object of this memo,
the characteristic of preserving the full US-ASCII range,
compatibility with file systems, parsers and other software that
on US-ASCII values but are transparent to other values. This
updates and replaces RFC 2044, in particular addressing the
of versions of the relevant standards
1.
ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character
called the Universal Character Set (UCS), which encompasses most
the world's writing systems. Two multi-octet encodings are defined
a four-octet per character encoding called UCS-4 and a two-octet
character encoding called UCS-2, able to address only the first 64
characters of the UCS (the Basic Multilingual Plane, BMP), outside
which there are currently no assignments
It is noteworthy that the same set of characters is defined by
Unicode standard [UNICODE], which further defines
character properties and other application details of great
to implementors, but does not have the UCS-4 encoding. Up to
Yergeau Standards Track [Page 1]
RFC 2279 UTF-8 January 1998
present time, changes in Unicode and amendments to ISO/IEC 10646
tracked each other, so that the character repertoires and code
assignments have remained in sync. The relevant
committees have committed to maintain this very useful synchronism
The UCS-2 and UCS-4 encodings, however, are hard to use in
current applications and protocols that assume 8 or even 7
characters. Even newer systems able to deal with 16 bit
cannot process UCS-4 data. This situation has led to the
of so-called UCS transformation formats (UTF), each with
characteristics
UTF-1 has only historical interest, having been removed from ISO/
10646. UTF-7 has the quality of encoding the full BMP
using only octets with the high-order bit clear (7 bit US-
values, [US-ASCII]), and is thus deemed a mail-safe
([RFC2152]). UTF-8, the object of this memo, uses all bits of
octet, but has the quality of preserving the full US-ASCII range
US-ASCII characters are encoded in one octet having the normal US
ASCII value, and any octet with such a value can only stand for
US-ASCII character, and nothing else
UTF-16 is a scheme for transforming a subset of the UCS-4
into pairs of UCS-2 values from a reserved range. UTF-16
UTF-8 in that UCS-2 values from the reserved range must be
specially in the UTF-8 transformation
UTF-8 encodes UCS-2 or UCS-4 characters as a varying number
octets, where the number of octets, and the value of each, depend
the integer value assigned to the character in ISO/IEC 10646.
transformation format has the following characteristics (all
are in hexadecimal):
- Character values from 0000 0000 to 0000 007F (US-ASCII repertoire
correspond to octets 00 to 7F (7 bit US-ASCII values). A
consequence is that a plain ASCII string is also a valid UTF-8
string
- US-ASCII values do not appear otherwise in a UTF-8
character stream. This provides compatibility with file
or other software (e.g. the printf() function in C libraries)
parse based on US-ASCII values but are transparent to
values
- Round-trip conversion is easy between UTF-8 and either of UCS-4,
UCS-2.
Yergeau Standards Track [Page 2]
RFC 2279 UTF-8 January 1998
- The first octet of a multi-octet sequence indicates the number
octets in the sequence
- The octet values FE and FF never appear
- Character boundaries are easily found from anywhere in an
stream
- The lexicographic sorting order of UCS-4 strings is preserved.
course this is of limited interest since the sort order is
culturally valid in either case
- The Boyer-Moore fast search algorithm can be used with UTF-8 data
- UTF-8 strings can be fairly reliably recognized as such by
simple algorithm, i.e. the probability that a string of
in any other encoding appears as valid UTF-8 is low,
with increasing string length
UTF-8 was originally a project of the X/Open
Internationalization Group XOJIG with the objective to specify a
System Safe UCS Transformation Format [FSS-UTF] that is
with UNIX systems, supporting multilingual text in a single encoding
The original authors were Gary Miller, Greger Leijonhufvud and
Entenmann. Later, Ken Thompson and Rob Pike did significant work
the formal UTF-8.
A description can also be found in Unicode Technical Report #4 and
the Unicode Standard, version 2.0 [UNICODE]. The
reference, including provisions for UTF-16 data within UTF-8,
Annex R of ISO/IEC 10646-1 [ISO-10646].
2. UTF-8
In UTF-8, characters are encoded using sequences of 1 to 6 octets
The only octet of a "sequence" of one has the higher-order bit set
0, the remaining 7 bits being used to encode the character value.
a sequence of n octets, n>1, the initial octet has the n higher-
bits set to 1, followed by a bit set to 0. The remaining bit(s)
that octet contain bits from the value of the character to
encoded. The following octet(s) all have the higher-order bit set
1 and the following bit set to 0, leaving 6 bits in each to
bits from the character to be encoded
The table below summarizes the format of these different octet types
The letter x indicates bits available for encoding bits of the UCS-4
character value
Yergeau Standards Track [Page 3]
RFC 2279 UTF-8 January 1998
UCS-4 range (hex.) UTF-8 octet sequence (binary
0000 0000-0000 007F 0
0000 0080-0000 07FF 110xxxxx 10
0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10
0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10
0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10
0400 0000-7FFF FFFF 1111110x 10xxxxxx ... 10
Encoding from UCS-4 to UTF-8 proceeds as follows
1) Determine the number of octets required from the character
and the first column of the table above. It is important to
that the rows of the table are mutually exclusive, i.e. there
only one valid way to encode a given UCS-4 character
2) Prepare the high-order bits of the octets as per the second
of the table
3) Fill in the bits marked x from the bits of the character value
starting from the lower-order bits of the character value
putting them first in the last octet of the sequence, then
next to last, etc. until all x bits are filled in
The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can
obtained from the above, in principle, by simply extending
UCS-2 character with two zero-valued octets. However, pairs
UCS-2 values between D800 and DFFF (surrogate pairs in
parlance), being actually UCS-4 characters transformed
UTF-16, need special treatment: the UTF-16 transformation must
undone, yielding a UCS-4 character that is then transformed
above
Decoding from UTF-8 to UCS-4 proceeds as follows
1) Initialize the 4 octets of the UCS-4 character with all bits
to 0.
2) Determine which bits encode the character value from the number
octets in the sequence and the second column of the table
(the bits marked x).
3) Distribute the bits from the sequence to the UCS-4 character
first the lower-order bits from the last octet of the sequence
proceeding to the left until no x bits are left
If the UTF-8 sequence is no more than three octets long,
can proceed directly to UCS-2.
Yergeau Standards Track [Page 4]
RFC 2279 UTF-8 January 1998
NOTE -- actual implementations of the decoding algorithm
should protect against decoding invalid sequences.
instance, a naive implementation may (wrongly) decode
invalid UTF-8 sequence C0 80 into the character U+0000,
may have security consequences and/or cause other problems.
the Security Considerations section below
A more detailed algorithm and formulae can be found in [FSS_UTF],
[UNICODE] or Annex R to [ISO-10646].
3. Versions of the
ISO/IEC 10646 is updated from time to time by published amendments
similarly, different versions of the Unicode standard exist: 1.0, 1.1
and 2.0 as of this writing. Each new version obsoletes and
the previous one, but implementations, and more significantly data
are not updated instantly
In general, the changes amount to adding new characters, which
not pose particular problems with old data. Amendment 5 to ISO/
10646, however, has moved and expanded the Korean Hangul block
thereby making any previous data containing Hangul characters
under the new version. Unicode 2.0 has the same difference
Unicode 1.1. The official justification for allowing such
incompatible change was that no implementations and no
containing Hangul existed, a statement that is likely to be true
remains unprovable. The incident has been dubbed the "Korean mess",
and the relevant committees have pledged to never, ever again
such an incompatible change
New versions, and in particular any incompatible changes, have
conseuences regarding MIME character encoding labels, to be
in section 5.
4.
The UCS-2 sequence "A." (0041, 2262, 0391,
002E) may be encoded in UTF-8 as follows
41 E2 89 A2 CE 91 2
The UCS-2 sequence representing the Hangul characters for the
word "hangugo" (D55C, AD6D, C5B4) may be encoded as follows
ED 95 9C EA B5 AD EC 96 B
Yergeau Standards Track [Page 5]
RFC 2279 UTF-8 January 1998
The UCS-2 sequence representing the Han characters for the
word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows
E6 97 A5 E6 9C AC E8 AA 9
5. MIME
This memo is meant to serve as the basis for registration of a
character set parameter (charset) [CHARSET-REG]. The
charset parameter value is "UTF-8". This string labels media
containing text consisting of characters from the repertoire
ISO/IEC 10646 including all amendments at least up to amendment 5
(Korean block), encoded to a sequence of octets using the
scheme outlined above. UTF-8 is suitable for use in MIME
types under the "text" top-level type
It is noteworthy that the label "UTF-8" does not contain a
identification, referring generically to ISO/IEC 10646. This
intentional, the rationale being as follows
A MIME charset label is designed to give just the information
to interpret a sequence of bytes received on the wire into a
of characters, nothing more (see RFC 2045, section 2.2, in [MIME]).
As long as a character set standard does not change incompatibly
version numbers serve no purpose, because one gains nothing
learning from the tag that newly assigned characters may be
that one doesn't know about. The tag itself doesn't teach
about the new characters, which are going to be received anyway
Hence, as long as the standards evolve compatibly, the
advantage of having labels that identify the versions is only that
apparent. But there is a disadvantage to such version-
labels: when an older application receives data accompanied by
newer, unknown label, it may fail to recognize the label and
completely unable to deal with the data, whereas a generic,
label would have triggered mostly correct processing of the data
which may well not contain any new characters
Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an
change, in principle contradicting the appropriateness of a
independent MIME charset label as described above. But
compatibility problem can only appear with data containing
Hangul characters encoded according to Unicode 1.1 (or
ISO/IEC 10646 before amendment 5), and there is arguably no such
to worry about, this being the very reason the incompatible
was deemed acceptable
Yergeau Standards Track [Page 6]
RFC 2279 UTF-8 January 1998
In practice, then, a version-independent label is warranted,
the label is understood to refer to all versions after Amendment 5,
and provided no incompatible change actually occurs.
incompatible changes occur in a later version of ISO/IEC 10646,
MIME charset label defined here will stay aligned with the
version until and unless the IETF specifically decides otherwise
It is also proposed to register the charset parameter
"UNICODE-1-1-UTF-8", for the exclusive purpose of labelling text
containing Hangul syllables encoded to UTF-8 without taking
account Amendment 5 of ISO/IEC 10646 (i.e. using the pre-amendment 5
code point assignments). Any other UTF-8 data SHOULD NOT use
label, in particular data not containing any Hangul syllables, and
is felt important to strongly recommend against creating any
Hangul-containing data without taking Amendment 5 of ISO/IEC 10646
into account
6. Security
Implementors of UTF-8 need to consider the security aspects of
they handle illegal UTF-8 sequences. It is conceivable that in
circumstances an attacker would be able to exploit an
UTF-8 parser by sending it an octet sequence that is not permitted
the UTF-8 syntax
A particularly subtle form of this attack could be carried
against a parser which performs security-critical validity
against the UTF-8 encoded form of its input, but interprets
illegal octet sequences as characters. For example, a parser
prohibit the NUL character when encoded as the single-octet
00, but allow the illegal two-octet sequence C0 80 and interpret
as a NUL character. Another example might be a parser
prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits
illegal octet sequence 2F C0 AE 2E 2F
The following have participated in the drafting and discussion
this memo
James E. Agenbroad Andries
Martin J. D|rst Ned
David Goldsmith Edwin F.
Kent Karlsson Markus
Michael Kung Alain
John Gardiner Myers Murray
Keld Simonsen Arnold
Yergeau Standards Track [Page 7]
RFC 2279 UTF-8 January 1998
[CHARSET-REG] Freed, N., and J. Postel, "IANA Charset
Procedures", BCP 19, RFC 2278, January 1998.
[FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm
22p. pbk. 172g. 4/95, X/Open Company Ltd., "
System Safe UCS Transformation Format (FSS_UTF)",
X/Open Preleminary Specification, Document
P316. Also published in Unicode Technical Report #4.
[ISO-10646] ISO/IEC 10646-1:1993. International Standard --
Information technology -- Universal Multiple-
Coded Character Set (UCS) -- Part 1: Architecture
Basic Multilingual Plane. Five amendments and
technical corrigendum have been published up to now
UTF-8 is described in Annex R, published as
2. UTF-16 is described in Annex Q, published
Amendment 1. 17 other amendments are currently
various stages of standardization
[MIME] Freed, N., and N. Borenstein, "Multipurpose
Mail Extensions (MIME) Part One: Format of
Message Bodies", RFC 2045. N. Freed, N. Borenstein
"Multipurpose Internet Mail Extensions (MIME)
Two: Media Types", RFC 2046. K. Moore, "
(Multipurpose Internet Mail Extensions) Part Three
Message Header Extensions for Non-ASCII Text",
2047. N. Freed, J. Klensin, J. Postel, "
Internet Mail Extensions (MIME) Part Four
Registration Procedures", RFC 2048. N. Freed, N
Borenstein, " Multipurpose Internet Mail
(MIME) Part Five: Conformance Criteria and Examples",
RFC 2049. All November 1996.
[RFC2152] Goldsmith, D., and M. Davis, "UTF-7: A Mail-
Transformation Format of Unicode", RFC 1642,
inc., May 1997. (Obsoletes RFC1642)
[UNICODE] The Unicode Consortium, "The Unicode Standard --
Version 2.0", Addison-Wesley, 1996.
[US-ASCII] Coded Character Set--7-bit American Standard Code
Information Interchange, ANSI X3.4-1986.
Yergeau Standards Track [Page 8]
RFC 2279 UTF-8 January 1998
Author's
Francois
Alis
100, boul. Alexis-
Suite 600
Montreal QC H4M 2P
Phone: +1 (514) 747-2547
Fax: +1 (514) 747-2561
EMail: fyergeau@alis.
Yergeau Standards Track [Page 9]
RFC 2279 UTF-8 January 1998
Full Copyright
Copyright (C) The Internet Society (1998). All Rights Reserved
This document and translations of it may be copied and furnished
others, and derivative works that comment on or otherwise explain
or assist in its implementation may be prepared, copied,
and distributed, in whole or in part, without restriction of
kind, provided that the above copyright notice and this paragraph
included on all such copies and derivative works. However,
document itself may not be modified in any way, such as by
the copyright notice or references to the Internet Society or
Internet organizations, except as needed for the purpose
developing Internet standards in which case the procedures
copyrights defined in the Internet Standards process must
followed, or as required to translate it into languages other
English
The limited permissions granted above are perpetual and will not
revoked by the Internet Society or its successors or assigns
This document and the information contained herein is provided on
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED,
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE
Yergeau Standards Track [Page 10]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
