As per Relevance of the word information, we have this rfc below:
Network Working Group D.
Request for Comments: 2152 Apple Computer, Inc
Obsoletes: RFC 1642 M.
Category: Informational Taligent, Inc
May 1997
UTF-7
A Mail-Safe Transformation Format of
Status of this
This memo provides information for the Internet community. This
does not specify an Internet standard of any kind. Distribution
this memo is unlimited
The Unicode Standard, version 2.0, and ISO/IEC 10646-1:1993(E) (
amended) jointly define a character set (hereafter referred to
Unicode) which encompasses most of the world's writing systems
However, Internet mail (STD 11, RFC 822) currently supports only 7-
bit US ASCII as a character set. MIME (RFC 2045 through 2049)
Internet mail to support different media types and character sets
and thus could support Unicode in mail messages. MIME neither
Unicode as a permitted character set nor specifies how it would
encoded, although it does provide for the registration of
character sets over time
This document describes a transformation format of Unicode
contains only 7-bit ASCII octets and is intended to be readable
humans in the limiting case that the document consists of
from the US-ASCII repertoire. It also specifies how
transformation format is used in the context of MIME and RFC 1641,
"Using Unicode with MIME".
Although other transformation formats of Unicode exist and
conceivably be used in this context (most notably UTF-8, also
as UTF-2 or UTF-FSS), they suffer the disadvantage that they
octets in the range decimal 128 through 255 to encode
characters outside the US-ASCII range. Thus, in the context of mail
those octets must themselves be encoded. This requires putting
through two successive encoding processes, and leads to a
expansion of characters outside the US-ASCII range, putting non
English speakers at a disadvantage. For example, using UTF-8
Goldsmith & Davis Informational [Page 1]
RFC 2152 UTF-7 May 1997
with the Quoted-Printable content transfer encoding of
represents US-ASCII characters in one octet, but other characters
require up to nine octets
UTF-7 encodes Unicode characters as US-ASCII octets, together
shift sequences to encode characters outside that range. For
purpose, one of the characters in the US-ASCII repertoire is
for use as a shift character
Many mail gateways and systems cannot handle the entire US-
character set (those based on EBCDIC, for example), and so UTF-7
contains provisions for encoding characters within US-ASCII in a
that all mail systems can accomodate
UTF-7 should normally be used only in the context of 7
transports, such as mail. In other contexts, straight Unicode
UTF-8 is preferred
See RFC 1641, "Using Unicode with MIME" for the overall
on usage of Unicode transformation formats with MIME
First, the definition of Unicode
The 16 bit character set Unicode is defined by "The
Standard, Version 2.0". This character set is identical with
character repertoire and coding of the international
ISO/IEC 10646-1:1993(E); Coded Representation Form=UCS-2;
Subset=300; Implementation Level=3, including the first 7
amendments to 10646 plus editorial corrections
Note. Unicode 2.0 further specifies the use and interaction
these character codes beyond the ISO standard. However, any
10646 sequence is a valid Unicode sequence, and vice versa
Unicode supplies interpretations of sequences on which the
standard is silent as to interpretation
Next, some handy definitions of US-ASCII character subsets
Set D (directly encoded characters) consists of the
characters (derived from RFC 1521, Appendix B, which no
appears in RFC 2045): the upper and lower case letters A through
and a through z, the 10 digits 0-9, and the following nine
characters (note that "+" and "=" are omitted):
Goldsmith & Davis Informational [Page 2]
RFC 2152 UTF-7 May 1997
Character ASCII & Unicode Value (decimal
' 39
( 40
) 41
, 44
- 45
. 46
/ 47
: 58
? 63
Set O (optional direct characters) consists of the
characters (note that "\" and "~" are omitted):
Character ASCII & Unicode Value (decimal
! 33
" 34
# 35
$ 36
% 37
& 38
* 42
; 59
< 60
= 61
> 62
@ 64
[ 91
] 93
^ 94
_ 95
' 96
{ 123
| 124
} 125
Rationale. The characters "\" and "~" are omitted because they
often redefined in variants of ASCII
Set B (Modified Base 64) is the set of characters in the Base64
alphabet defined in RFC 2045, excluding the pad character "="
(decimal value 61).
Goldsmith & Davis Informational [Page 3]
RFC 2152 UTF-7 May 1997
Rationale. The pad character = is excluded because UTF-7 is
for use within header fields as set forth in RFC 2047. Since the
readable encoding in RFC 2047 is "Q" (based on RFC 2045's Quoted
Printable), the "=" character is not available for use (without a
of escape sequences). This was very unfortunate but unavoidable.
"=" character could otherwise have been used as the UTF-7
character as well (rather than using "+").
Note that all characters in US-ASCII have the same value in
when zero-extended to 16 bits
UTF-7
A UTF-7 stream represents 16-bit Unicode characters using 7-bit US
ASCII octets as follows
Rule 1: (direct encoding) Unicode characters in set D above may
encoded directly as their ASCII equivalents. Unicode characters
Set O may optionally be encoded directly as their
equivalents, bearing in mind that many of these characters
illegal in header fields, or may not pass correctly through
mail gateways
Rule 2: (Unicode shifted encoding) Any Unicode character
may be encoded using a sequence of characters in set B,
preceded by the shift character "+" (US-ASCII character
decimal 43). The "+" signals that subsequent octets are to
interpreted as elements of the Modified Base64 alphabet until
character not in that alphabet is encountered. Such
include control characters such as carriage returns and
feeds; thus, a Unicode shifted sequence always terminates at
of a line. As a special case, if the sequence terminates with
character "-" (US-ASCII decimal 45) then that character
absorbed; other terminating characters are not absorbed and
processed normally
Note that if the first character after the shifted sequence is "-"
then an extra "-" must be present to terminate the
sequence so that the actual "-" is not itself absorbed
Rationale. A terminating character is necessary for cases
the next character after the Modified Base64 sequence is part
character set B or is itself the terminating character. It
also enhance readability by delimiting encoded sequences
Goldsmith & Davis Informational [Page 4]
RFC 2152 UTF-7 May 1997
Also as a special case, the sequence "+-" may be used to
the character "+". A "+" character followed immediately by
character other than members of set B or "-" is an ill-
sequence
Unicode is encoded using Modified Base64 by first
Unicode 16-bit quantities to an octet stream (with the
significant octet first). Surrogate pairs (UTF-16) are
by treating each half of the pair as a separate 16 bit
(i.e., no special treatment). Text with an odd number of octets
ill-formed. ISO 10646 characters outside the range addressable
surrogate pairs cannot be encoded
Rationale. ISO/IEC 10646-1:1993(E) specifies that when
the UCS-2 form are serialized as octets, that the most
octet appear first. This is also in keeping with common
practice of choosing a canonical format for transmission
Rationale. The policy for code point allocation within ISO 10646
and Unicode is that the repertoires be kept synchronized. No
points will be allocated in ISO 10646 outside the
addressable by surrogate pairs
Next, the octet stream is encoded by applying the Base64
transfer encoding algorithm as defined in RFC 2045, modified
omit the "=" pad character. Instead, when encoding, zero bits
added to pad to a Base64 character boundary. When decoding,
bits at the end of the Modified Base64 sequence that do
constitute a complete 16-bit Unicode character are discarded.
such discarded bits are non-zero the sequence is ill-formed
Rationale. The pad character "=" is not used when
Modified Base64 because of the conflict with its use as an
character for the Q content transfer encoding in RFC 2047
fields, as mentioned above
Rule 3: The space (decimal 32), tab (decimal 9), carriage
(decimal 13), and line feed (decimal 10) characters may
directly represented by their ASCII equivalents. However,
that MIME content transfer encodings have rules concerning the
of such characters. Usage that does not conform to
restrictions of RFC 822, for example, would have to be
using MIME content transfer encodings other than 7bit or 8bit
such as quoted-printable, binary, or base64.
Given this set of rules, Unicode characters which may be encoded
rules 1 or 3 take one octet per character, and other
characters are encoded on average with 2 2/3 octets per
Goldsmith & Davis Informational [Page 5]
RFC 2152 UTF-7 May 1997
plus one octet to switch into Modified Base64 and an optional
to switch out
Example. The Unicode sequence "A."
(hexadecimal 0041,2262,0391,002E) may be encoded as follows
A+ImIDkQ
Example. The Unicode sequence "Hi Mom --!"
(hexadecimal 0048, 0069, 0020, 004D, 006F, 006D, 0020, 002D, 263A
002D, 0021) may be encoded as follows
Hi Mom -+Jjo--!
Example. The Unicode sequence representing the Han characters
the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E) may
encoded as follows
+ZeVnLIqe
Use of Character Set UTF-7 Within
Character set UTF-7 is safe for mail transmission and therefore
be used with any content transfer encoding in MIME (except where
length and line break restrictions are violated). Specifically, the 7
bit encoding for bodies and the Q encoding for headers are
acceptable. The MIME character set tag is UTF-7. This signifies
version of Unicode equal to or greater than 2.0.
Example. Here is a text portion of a MIME message containing
Unicode sequence "Hi Mom !" (hexadecimal 0048,
0069, 0020, 004D, 006F, 006D, 0020, 263A, 0021).
Content-Type: text/plain; charset=UTF-7
Hi Mom +Jjo-!
Example. Here is a text portion of a MIME message containing
Unicode sequence representing the Han characters for the
word "nihongo" (hexadecimal 65E5,672C,8A9E).
Content-Type: text/plain; charset=UTF-7
+ZeVnLIqe
Example. Here is a text portion of a MIME message containing
Unicode sequence "A." (
0041,2262,0391,002E).
Goldsmith & Davis Informational [Page 6]
RFC 2152 UTF-7 May 1997
Content-Type: text/plain; charset=utf-7
A+ImIDkQ
Example. Here is a text portion of a MIME message containing
Unicode sequence "Item 3 is 1." (hexadecimal 0049,
0074, 0065, 006D, 0020, 0033, 0020, 0069, 0073, 0020, 00A3, 0031,
002E).
Content-Type: text/plain; charset=UTF-7
Item 3 is +AKM-1.
Note that to achieve the best interoperability with systems that
not support Unicode or MIME, when preparing text for
transmission line breaks should follow Internet conventions.
means that lines should be short and terminated with the proper
CRLF sequence. Unicode LINE SEPARATOR (hexadecimal 2028)
PARAGRAPH SEPARATOR (hexadecimal 2029) should be converted to
line breaks. Ideally, this would be handled transparently by
Unicode-aware user agent
This preparation is not absolutely necessary, since UTF-7 and
appropriate MIME content transfer encoding can handle text that
not follow Internet conventions, but readability by systems
Unicode or MIME will be impaired. See RFC 2045 for a discussion
mail interoperability issues
Lines should never be broken in the middle of a UTF-7
sequence, since such sequences may not cross line breaks. Therefore
UTF-7 encoding should take place after line breaking. If a
containing a shifted sequence is too long after encoding, a
content transfer encoding such as Quoted Printable can be used
encode the text. Another possibility is to perform line breaking
UTF-7 encoding at the same time, so that lines containing
sequences already conform to length restrictions
In this section we will motivate the introduction of UTF-7 as
to the alternative of using the existing transformation formats
Unicode (e.g., UTF-8) with MIME's content transfer encodings.
discussing this, it will be useful to list some assumptions
character frequency within typical natural language text strings
we use to estimate typical storage requirements
1. Most Western European languages use roughly 7/8 of their
from US-ASCII and 1/8 from Latin 1 (ISO-8859-1).
Goldsmith & Davis Informational [Page 7]
RFC 2152 UTF-7 May 1997
2. Most non-Roman alphabet-based languages (e.g., Greek) use
1/6 of their letters from ASCII (since white space is in the 7-
area) and the rest from their alphabets
3. East Asian ideographic-based languages (including Japanese)
essentially all of their characters from the Han or CJK
area
4. Non-directly encoded punctuation characters do not
frequently enough to affect the results
Notice that current 8 bit standards, such as ISO-8859-x, require
of a content transfer encoding. For comparison with the
discussion, the costs break down as follows (note that many of
figures are approximate since they depend on the exact composition
the text):
8859-x in Base64
Text type Average octets/
All 1.33
8859-x in Quoted
Text type Average octets/
US-ASCII 1
Western European 1.25
Other 2.67
Note also that Unicode encoded in Base64 takes a constant 2.67
per character. For purposes of comparison, we will look at UTF-8
Base64 and Quoted Printable, and UTF-7. Also note that fixed
for long strings is relative to 1/n, where n is the encoded
length in octets
UTF-8 in Base64
Text type Average octets/
US-ASCII 1.33
Western European 1.5
Some Alphabetics 2.44
All others 4
Goldsmith & Davis Informational [Page 8]
RFC 2152 UTF-7 May 1997
UTF-8 in Quoted
Text type Average octets/
US-ASCII 1
Western European 1.63
Some Alphabetics 5.17
All others 7-9
UTF-7
Text type Average octets/
Most US-ASCII 1
Western European 1.5
All others 2.67+2/
We feel that the UTF-8 in Quoted Printable option is not viable
to the very large expansion of all text except Western European.
would only be viable in texts consisting of large expanses of US
ASCII or Latin characters with occasional other
interspersed. We would prefer to introduce one encoding that
reasonably well for all users
We also feel that UTF-8 in Base64 has high expansion for non
Western-European users, and is less desirable because it cannot
read directly, even when the content is largely US-ASCII. The
encoding of UTF-7 gives competitive results and is readable for
text
UTF-7 gives results competitive with ISO-8859-x, with access to
of the Unicode character set. We believe this justifies
introduction of a new transformation format of Unicode
Goldsmith & Davis Informational [Page 9]
RFC 2152 UTF-7 May 1997
As an alternative to use of UTF-7, it might be possible to
Unicode characters with other character sets using an existing
mechanism, the multipart/mixed content type, ignoring for the
the issues with line breaks (thanks to Nathaniel Borenstein
suggesting this). For instance (repeating an earlier example):
Content-type: multipart/mixed; boundary=
Content-Disposition:
--
Content-type: text/plain; charset=us-
Hi
--
Content-type: text/plain; charset=UNICODE-2-0
Content-transfer-encoding: base64
Jjo
--
Content-type: text/plain; charset=us-
!
--foo--
Theoretically, this removes the need for UTF-7 in message
(multipart may not be used in header fields). However, we feel
as use of the Unicode character set becomes more widespread
intermittent use of specialized Unicode characters (such as
and mathematical symbols) will occur, and that text will
typically include small snippets from other scripts, such
Cyrillic, Greek, or East Asian languages (anything in the
script is already handled adequately by existing MIME
sets). Although the multipart technique works well for large
of text in alternating character sets, we feel it does not
support the kinds of uses just discussed, and so we still believe
introduction of UTF-7 is justified
The UTF-7 encoding allows Unicode characters to be encoded within
US-ASCII 7 bit character set. It is most effective for
sequences which contain relatively long strings of US-
characters interspersed with either single Unicode characters
strings of Unicode characters, as it allows the US-ASCII portions
be read on systems without direct Unicode support
UTF-7 should only be used with 7 bit transports such as mail.
other contexts, use of straight Unicode or UTF-8 is preferred
Goldsmith & Davis Informational [Page 10]
RFC 2152 UTF-7 May 1997
Many thanks to the following people for their contributions
comments, and suggestions. If we have omitted anyone it was
oversight and not intentionally
Glenn
Harald T.
Nathaniel
Lee
Jim
Dave
Steve
Dana S.
Ned
Kari E.
John H.
John C.
Valdis
Keith
Masataka
Einar
Erik M. van der
Goldsmith & Davis Informational [Page 11]
RFC 2152 UTF-7 May 1997
Appendix A --
Here is a longer example, taken from a document originally in Big
code. It has been condensed for brevity. There are two versions:
first uses optional characters from set O (and so may not
through some mail gateways), and the second does not
Content-type: text/plain; charset=utf-7
Below is the full Chinese text of the Analects (+itaKng-).
The sources for the text are
"The sayings of Confucius," James R. Ware, trans. +U/BTFw-:
+ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation
+Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990.
"The Chinese Classics with a Translation, Critical and
Notes, Prolegomena, and Copius Indexes," James Legge, trans., Taipei
Southern Materials Center Publishing, Inc., 1991. (Chinese text
English translation
Big Five and GB versions of the text are being made
separately
Neither the Big Five nor GB contain all the characters used in
text. Missing characters have been indicated using their Unicode/
10646 code points. "U+-" followed by four hexadecimal
indicates a Unicode/10646 code (e.g., U+-9F08). There is no
solution to the problem of the small size of the Big Five/
character sets; this represents the solution I find personally
satisfactory
(omitted...)
I have tried to minimize this problem by using variant
where they were available and the character actually in the text
not. Only variants listed as such in the +XrdxmVtXUXg- were used
(omitted...)
John H. Jenkins +TpVPXGBG- jenkins@apple.com 5 January 1993
(omitted...)
Content-type: text/plain; charset=utf-7
Below is the full Chinese text of the Analects (+itaKng-).
Goldsmith & Davis Informational [Page 12]
RFC 2152 UTF-7 May 1997
The sources for the text are
+ACI-The sayings of Confucius,+ACI- James R. Ware, trans. +U/BTFw-:
+ZYeB9FH6ckh5Pg-, 1980. (Chinese text with English translation
+Vttm+E6UfZM-, +W4tRQ066bOg-, +UxdOrA-: +Ti1XC2b4Xpc-, 1990.
+ACI-The Chinese Classics with a Translation, Critical and
Notes, Prolegomena, and Copius Indexes,+ACI- James Legge, trans.,
Taipei: Southern Materials Center Publishing, Inc., 1991. (
text with English translation
Big Five and GB versions of the text are being made
separately
Neither the Big Five nor GB contain all the characters used in
text. Missing characters have been indicated using their Unicode/
10646 code points. +ACI-U+-+ACI- followed by four hexadecimal
indicates a Unicode/10646 code (e.g., U+-9F08). There is no
solution to the problem of the small size of the Big Five/
character sets+ADs- this represents the solution I find
most satisfactory
(omitted...)
I have tried to minimize this problem by using variant
where they were available and the character actually in the text
not. Only variants listed as such in the +XrdxmVtXUXg- were used
(omitted...)
John H. Jenkins +TpVPXGBG- jenkins+AEA-apple.com 5 January 1993
(omitted...)
Goldsmith & Davis Informational [Page 13]
RFC 2152 UTF-7 May 1997
Security
Security issues are not discussed in this memo
[UNICODE 2.0] "The Unicode Standard, Version 2.0", The
Consortium, Addison-Wesley, 1996. ISBN 0-201-48345-9.
[ISO 10646] ISO/IEC 10646-1:1993(E) Information Technology--
Multiple-octet Coded Character Set (UCS). See
amendments 1 through 7, plus editorial corrections
[RFC-1641] Goldsmith, D., and M. Davis, "Using Unicode with MIME",
RFC 1641, Taligent, Inc., July 1994.
[US-ASCII] Coded Character Set--7-bit American Standard Code
Information Interchange, ANSI X3.4-1986.
[ISO-8859] Information Processing -- 8-bit Single-Byte Coded
Character Sets -- Part 1: Latin Alphabet No. 1,
8859-1:1987. Part 2: Latin alphabet No. 2, ISO 8859-2,
1987. Part 3: Latin alphabet No. 3, ISO 8859-3, 1988.
Part 4: Latin alphabet No. 4, ISO 8859-4, 1988. Part 5:
Latin/Cyrillic alphabet, ISO 8859-5, 1988. Part 6:
Latin/Arabic alphabet, ISO 8859-6, 1987. Part 7:
Latin/Greek alphabet, ISO 8859-7, 1987. Part 8:
Latin/Hebrew alphabet, ISO 8859-8, 1988. Part 9:
alphabet No. 5, ISO 8859-9, 1990.
[RFC822] Crocker, D., "Standard for the Format of ARPA
Text Messages", STD 11, RFC 822, UDEL, August 1982.
[MIME] Borenstein N., N. Freed, K. Moore, J. Klensin, and J
Postel, "MIME (Multipurpose Internet Mail Extensions
Parts One through Five", RFC 2045, 2046, 2047, 2048,
2049, November 1996.
Authors'
David
Apple Computer, Inc
2 Infinite Loop, MS: 302-2
Cupertino, CA 95014
Phone: 408-974-1957
Fax: 408-862-4566
EMail: goldsmith@apple.
Goldsmith & Davis Informational [Page 14]
RFC 2152 UTF-7 May 1997
Mark
Taligent, Inc
10201 N. DeAnza Blvd
Cupertino, CA 95014-2233
Phone: 408-777-5116
Fax: 408-777-5081
EMail: mark_davis@taligent.
Goldsmith & Davis Informational [Page 15]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
