busybox/docs/unicode_UTF-8-test.txt
<<
>>
Prefs
   1UTF-8 decoder capability and stress test
   2----------------------------------------
   3
   4Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
   5
   6This test file can help you examine, how your UTF-8 decoder handles
   7various types of correct, malformed, or otherwise interesting UTF-8
   8sequences. This file is not meant to be a conformance test. It does
   9not prescribes any particular outcome and therefore there is no way to
  10"pass" or "fail" this test file, even though the texts suggests a
  11preferable decoder behaviour at some places. The aim is instead to
  12help you think about and test the behaviour of your UTF-8 on a
  13systematic collection of unusual inputs. Experience so far suggests
  14that most first-time authors of UTF-8 decoders find at least one
  15serious problem in their decoder by using this file.
  16
  17The test lines below cover boundary conditions, malformed UTF-8
  18sequences as well as correctly encoded UTF-8 sequences of Unicode code
  19points that should never occur in a correct UTF-8 file.
  20
  21According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
  22receiving UTF-8 shall interpret a "malformed sequence in the same way
  23that it interprets a character that is outside the adopted subset" and
  24"characters that are not within the adopted subset shall be indicated
  25to the user" by a receiving device. A quite commonly used approach in
  26UTF-8 decoders is to replace any malformed UTF-8 sequence by a
  27replacement character (U+FFFD), which looks a bit like an inverted
  28question mark, or a similar symbol. It might be a good idea to
  29visually distinguish a malformed UTF-8 sequence from a correctly
  30encoded Unicode character that is just not available in the current
  31font but otherwise fully legal, even though ISO 10646-1 doesn't
  32mandate this. In any case, just ignoring malformed sequences or
  33unavailable characters does not conform to ISO 10646, will make
  34debugging more difficult, and can lead to user confusion.
  35
  36Please check, whether a malformed UTF-8 sequence is (1) represented at
  37all, (2) represented by exactly one single replacement character (or
  38equivalent signal), and (3) the following quotation mark after an
  39illegal UTF-8 sequence is correctly displayed, i.e. proper
  40resynchronization takes place immageately after any malformed
  41sequence. This file says "THE END" in the last line, so if you don't
  42see that, your decoder crashed somehow before, which should always be
  43cause for concern.
  44
  45All lines in this file are exactly 79 characters long (plus the line
  46feed). In addition, all lines end with "|", except for the two test
  47lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
  48U+0000 and U+007F. If you display this file with a fixed-width font,
  49these "|" characters should all line up in column 79 (right margin).
  50This allows you to test quickly, whether your UTF-8 decoder finds the
  51correct number of characters in every line, that is whether each
  52malformed sequences is replaced by a single replacement character.
  53
  54Note that as an alternative to the notion of malformed sequence used
  55here, it is also a perfectly acceptable (and in some situations even
  56preferable) solution to represent each individual byte of a malformed
  57sequence by a replacement character. If you follow this strategy in
  58your decoder, then please ignore the "|" column.
  59
  60
  61Here come the tests:                                                          |
  62                                                                              |
  631  Some correct UTF-8 text                                                    |
  64                                                                              |
  65You should see the Greek word 'kosme':       "κόσμε"                          |
  66                                                                              |
  672  Boundary condition test cases                                              |
  68                                                                              |
  692.1  First possible sequence of a certain length                              |
  70                                                                              |
  712.1.1  1 byte  (U-00000000):        "�"                                        
  722.1.2  2 bytes (U-00000080):        "€"                                       |
  732.1.3  3 bytes (U-00000800):        "ࠀ"                                       |
  742.1.4  4 bytes (U-00010000):        "𐀀"                                       |
  752.1.5  5 bytes (U-00200000):        "øˆ€€€"                                       |
  762.1.6  6 bytes (U-04000000):        "ü„€€€€"                                       |
  77                                                                              |
  782.2  Last possible sequence of a certain length                               |
  79                                                                              |
  802.2.1  1 byte  (U-0000007F):        ""                                        
  812.2.2  2 bytes (U-000007FF):        "ß¿"                                       |
  822.2.3  3 bytes (U-0000FFFF):        "ï¿¿"                                       |
  832.2.4  4 bytes (U-001FFFFF):        "÷¿¿¿"                                       |
  842.2.5  5 bytes (U-03FFFFFF):        "û¿¿¿¿"                                       |
  852.2.6  6 bytes (U-7FFFFFFF):        "ý¿¿¿¿¿"                                       |
  86                                                                              |
  872.3  Other boundary conditions                                                |
  88                                                                              |
  892.3.1  U-0000D7FF = ed 9f bf = "퟿"                                            |
  902.3.2  U-0000E000 = ee 80 80 = ""                                            |
  912.3.3  U-0000FFFD = ef bf bd = "�"                                            |
  922.3.4  U-0010FFFF = f4 8f bf bf = "􏿿"                                         |
  932.3.5  U-00110000 = f4 90 80 80 = "ô€€"                                         |
  94                                                                              |
  953  Malformed sequences                                                        |
  96                                                                              |
  973.1  Unexpected continuation bytes                                            |
  98                                                                              |
  99Each unexpected continuation byte should be separately signalled as a         |
 100malformed sequence of its own.                                                |
 101                                                                              |
 1023.1.1  First continuation byte 0x80: "€"                                      |
 1033.1.2  Last  continuation byte 0xbf: "¿"                                      |
 104                                                                              |
 1053.1.3  2 continuation bytes: "€¿"                                             |
 1063.1.4  3 continuation bytes: "€¿€"                                            |
 1073.1.5  4 continuation bytes: "€¿€¿"                                           |
 1083.1.6  5 continuation bytes: "€¿€¿€"                                          |
 1093.1.7  6 continuation bytes: "€¿€¿€¿"                                         |
 1103.1.8  7 continuation bytes: "€¿€¿€¿€"                                        |
 111                                                                              |
 1123.1.9  Sequence of all 64 possible continuation bytes (0x80-0xbf):            |
 113                                                                              |
 114   "€‚ƒ„…†‡ˆ‰Š‹ŒŽ                                                          |
 115    ‘’“”•–—˜™š›œžŸ                                                          |
 116     ¡¢£¤¥¦§¨©ª«¬­®¯                                                          |
 117    °±²³´µ¶·¸¹º»¼½¾¿"                                                         |
 118                                                                              |
 1193.2  Lonely start characters                                                  |
 120                                                                              |
 1213.2.1  All 32 first bytes of 2-byte sequences (0xc0-0xdf),                    |
 122       each followed by a space character:                                    |
 123                                                                              |
 124   "À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï                                           |
 125    Ð Ñ Ò Ó Ô Õ Ö × Ø Ù Ú Û Ü Ý Þ ß "                                         |
 126                                                                              |
 1273.2.2  All 16 first bytes of 3-byte sequences (0xe0-0xef),                    |
 128       each followed by a space character:                                    |
 129                                                                              |
 130   "à á â ã ä å æ ç è é ê ë ì í î ï "                                         |
 131                                                                              |
 1323.2.3  All 8 first bytes of 4-byte sequences (0xf0-0xf7),                     |
 133       each followed by a space character:                                    |
 134                                                                              |
 135   "ð ñ ò ó ô õ ö ÷ "                                                         |
 136                                                                              |
 1373.2.4  All 4 first bytes of 5-byte sequences (0xf8-0xfb),                     |
 138       each followed by a space character:                                    |
 139                                                                              |
 140   "ø ù ú û "                                                                 |
 141                                                                              |
 1423.2.5  All 2 first bytes of 6-byte sequences (0xfc-0xfd),                     |
 143       each followed by a space character:                                    |
 144                                                                              |
 145   "ü ý "                                                                     |
 146                                                                              |
 1473.3  Sequences with last continuation byte missing                            |
 148                                                                              |
 149All bytes of an incomplete sequence should be signalled as a single           |
 150malformed sequence, i.e., you should see only a single replacement            |
 151character in each of the next 10 tests. (Characters as in section 2)          |
 152                                                                              |
 1533.3.1  2-byte sequence with last byte missing (U+0000):     "À"               |
 1543.3.2  3-byte sequence with last byte missing (U+0000):     "à€"               |
 1553.3.3  4-byte sequence with last byte missing (U+0000):     "ð€€"               |
 1563.3.4  5-byte sequence with last byte missing (U+0000):     "ø€€€"               |
 1573.3.5  6-byte sequence with last byte missing (U+0000):     "ü€€€€"               |
 1583.3.6  2-byte sequence with last byte missing (U-000007FF): "ß"               |
 1593.3.7  3-byte sequence with last byte missing (U-0000FFFF): "ï¿"               |
 1603.3.8  4-byte sequence with last byte missing (U-001FFFFF): "÷¿¿"               |
 1613.3.9  5-byte sequence with last byte missing (U-03FFFFFF): "û¿¿¿"               |
 1623.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "ý¿¿¿¿"               |
 163                                                                              |
 1643.4  Concatenation of incomplete sequences                                    |
 165                                                                              |
 166All the 10 sequences of 3.3 concatenated, you should see 10 malformed         |
 167sequences being signalled:                                                    |
 168                                                                              |
 169   "Àà€ð€€ø€€€ü€€€€ßï¿÷¿¿û¿¿¿ý¿¿¿¿"                                                               |
 170                                                                              |
 1713.5  Impossible bytes                                                         |
 172                                                                              |
 173The following two bytes cannot appear in a correct UTF-8 string               |
 174                                                                              |
 1753.5.1  fe = "þ"                                                               |
 1763.5.2  ff = "ÿ"                                                               |
 1773.5.3  fe fe ff ff = "þþÿÿ"                                                   |
 178                                                                              |
 1794  Overlong sequences                                                         |
 180                                                                              |
 181The following sequences are not malformed according to the letter of          |
 182the Unicode 2.0 standard. However, they are longer then necessary and         |
 183a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8         |
 184decoder" should reject them just like malformed sequences for two             |
 185reasons: (1) It helps to debug applications if overlong sequences are         |
 186not treated as valid representations of characters, because this helps        |
 187to spot problems more quickly. (2) Overlong sequences provide                 |
 188alternative representations of characters, that could maliciously be          |
 189used to bypass filters that check only for ASCII characters. For              |
 190instance, a 2-byte encoded line feed (LF) would not be caught by a            |
 191line counter that counts only 0x0a bytes, but it would still be               |
 192processed as a line feed by an unsafe UTF-8 decoder later in the              |
 193pipeline. From a security point of view, ASCII compatibility of UTF-8         |
 194sequences means also, that ASCII characters are *only* allowed to be          |
 195represented by ASCII bytes in the range 0x00-0x7f. To ensure this             |
 196aspect of ASCII compatibility, use only "safe UTF-8 decoders" that            |
 197reject overlong UTF-8 sequences for which a shorter encoding exists.          |
 198                                                                              |
 1994.1  Examples of an overlong ASCII character                                  |
 200                                                                              |
 201With a safe UTF-8 decoder, all of the following five overlong                 |
 202representations of the ASCII character slash ("/") should be rejected         |
 203like a malformed UTF-8 sequence, for instance by substituting it with         |
 204a replacement character. If you see a slash below, you do not have a          |
 205safe UTF-8 decoder!                                                           |
 206                                                                              |
 2074.1.1 U+002F = c0 af             = "À¯"                                        |
 2084.1.2 U+002F = e0 80 af          = "à€¯"                                        |
 2094.1.3 U+002F = f0 80 80 af       = "ð€€¯"                                        |
 2104.1.4 U+002F = f8 80 80 80 af    = "ø€€€¯"                                        |
 2114.1.5 U+002F = fc 80 80 80 80 af = "ü€€€€¯"                                        |
 212                                                                              |
 2134.2  Maximum overlong sequences                                               |
 214                                                                              |
 215Below you see the highest Unicode value that is still resulting in an         |
 216overlong sequence if represented with the given number of bytes. This         |
 217is a boundary test for safe UTF-8 decoders. All five characters should        |
 218be rejected like malformed UTF-8 sequences.                                   |
 219                                                                              |
 2204.2.1  U-0000007F = c1 bf             = "Á¿"                                   |
 2214.2.2  U-000007FF = e0 9f bf          = "àŸ¿"                                   |
 2224.2.3  U-0000FFFF = f0 8f bf bf       = "ð¿¿"                                   |
 2234.2.4  U-001FFFFF = f8 87 bf bf bf    = "ø‡¿¿¿"                                   |
 2244.2.5  U-03FFFFFF = fc 83 bf bf bf bf = "üƒ¿¿¿¿"                                   |
 225                                                                              |
 2264.3  Overlong representation of the NUL character                             |
 227                                                                              |
 228The following five sequences should also be rejected like malformed           |
 229UTF-8 sequences and should not be treated like the ASCII NUL                  |
 230character.                                                                    |
 231                                                                              |
 2324.3.1  U+0000 = c0 80             = ""                                       |
 2334.3.2  U+0000 = e0 80 80          = "à€€"                                       |
 2344.3.3  U+0000 = f0 80 80 80       = "ð€€€"                                       |
 2354.3.4  U+0000 = f8 80 80 80 80    = "ø€€€€"                                       |
 2364.3.5  U+0000 = fc 80 80 80 80 80 = "ü€€€€€"                                       |
 237                                                                              |
 2385  Illegal code positions                                                     |
 239                                                                              |
 240The following UTF-8 sequences should be rejected like malformed               |
 241sequences, because they never represent valid ISO 10646 characters and        |
 242a UTF-8 decoder that accepts them might introduce security problems           |
 243comparable to overlong UTF-8 sequences.                                       |
 244                                                                              |
 2455.1 Single UTF-16 surrogates                                                  |
 246                                                                              |
 2475.1.1  U+D800 = ed a0 80 = "í €"                                                |
 2485.1.2  U+DB7F = ed ad bf = "í­¿"                                                |
 2495.1.3  U+DB80 = ed ae 80 = "í®€"                                                |
 2505.1.4  U+DBFF = ed af bf = "í¯¿"                                                |
 2515.1.5  U+DC00 = ed b0 80 = "í°€"                                                |
 2525.1.6  U+DF80 = ed be 80 = "í¾€"                                                |
 2535.1.7  U+DFFF = ed bf bf = "í¿¿"                                                |
 254                                                                              |
 2555.2 Paired UTF-16 surrogates                                                  |
 256                                                                              |
 2575.2.1  U+D800 U+DC00 = ed a0 80 ed b0 80 = "𐀀"                               |
 2585.2.2  U+D800 U+DFFF = ed a0 80 ed bf bf = "𐏿"                               |
 2595.2.3  U+DB7F U+DC00 = ed ad bf ed b0 80 = "󯰀"                               |
 2605.2.4  U+DB7F U+DFFF = ed ad bf ed bf bf = "í­¿í¿¿"                               |
 2615.2.5  U+DB80 U+DC00 = ed ae 80 ed b0 80 = "󰀀"                               |
 2625.2.6  U+DB80 U+DFFF = ed ae 80 ed bf bf = "󰏿"                               |
 2635.2.7  U+DBFF U+DC00 = ed af bf ed b0 80 = "􏰀"                               |
 2645.2.8  U+DBFF U+DFFF = ed af bf ed bf bf = "􏿿"                               |
 265                                                                              |
 2665.3 Other illegal code positions                                              |
 267                                                                              |
 2685.3.1  U+FFFE = ef bf be = "￾"                                                |
 2695.3.2  U+FFFF = ef bf bf = "ï¿¿"                                                |
 270                                                                              |
 271THE END                                                                       |
 272