| 1 | UTF-8 decoder capability and stress test
|
| 2 | ----------------------------------------
|
| 3 |
|
| 4 | Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2015-08-28 - CC BY 4.0
|
| 5 |
|
| 6 | This test file can help you examine, how your UTF-8 decoder handles
|
| 7 | various types of correct, malformed, or otherwise interesting UTF-8
|
| 8 | sequences. This file is not meant to be a conformance test. It does
|
| 9 | not prescribe any particular outcome. Therefore, there is no way to
|
| 10 | "pass" or "fail" this test file, even though the text does suggest a
|
| 11 | preferable decoder behaviour at some places. Its aim is, instead, to
|
| 12 | help you think about, and test, the behaviour of your UTF-8 decoder on a
|
| 13 | systematic collection of unusual inputs. Experience so far suggests
|
| 14 | that most first-time authors of UTF-8 decoders find at least one
|
| 15 | serious problem in their decoder using this file.
|
| 16 |
|
| 17 | The test lines below cover boundary conditions, malformed UTF-8
|
| 18 | sequences, as well as correctly encoded UTF-8 sequences of Unicode code
|
| 19 | points that should never occur in a correct UTF-8 file.
|
| 20 |
|
| 21 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
|
| 22 | receiving UTF-8 shall interpret a "malformed sequence in the same way
|
| 23 | that it interprets a character that is outside the adopted subset" and
|
| 24 | "characters that are not within the adopted subset shall be indicated
|
| 25 | to the user" by a receiving device. One commonly used approach in
|
| 26 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a
|
| 27 | replacement character (U+FFFD), which looks a bit like an inverted
|
| 28 | question mark, or a similar symbol. It might be a good idea to
|
| 29 | visually distinguish a malformed UTF-8 sequence from a correctly
|
| 30 | encoded Unicode character that is just not available in the current
|
| 31 | font but otherwise fully legal, even though ISO 10646-1 doesn't
|
| 32 | mandate this. In any case, just ignoring malformed sequences or
|
| 33 | unavailable characters does not conform to ISO 10646, will make
|
| 34 | debugging more difficult, and can lead to user confusion.
|
| 35 |
|
| 36 | Please check, whether a malformed UTF-8 sequence is (1) represented at
|
| 37 | all, (2) represented by exactly one single replacement character (or
|
| 38 | equivalent signal), and (3) the following quotation mark after an
|
| 39 | illegal UTF-8 sequence is correctly displayed, i.e. proper
|
| 40 | resynchronization takes place immediately after any malformed
|
| 41 | sequence. This file says "THE END" in the last line, so if you don't
|
| 42 | see that, your decoder crashed somehow before, which should always be
|
| 43 | cause for concern.
|
| 44 |
|
| 45 | All lines in this file are exactly 79 characters long (plus the line
|
| 46 | feed). In addition, all lines end with "|", except for the two test
|
| 47 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
|
| 48 | U+0000 and U+007F. If you display this file with a fixed-width font,
|
| 49 | these "|" characters should all line up in column 79 (right margin).
|
| 50 | This allows you to test quickly, whether your UTF-8 decoder finds the
|
| 51 | correct number of characters in every line, that is whether each
|
| 52 | malformed sequences is replaced by a single replacement character.
|
| 53 |
|
| 54 | Note that, as an alternative to the notion of malformed sequence used
|
| 55 | here, it is also a perfectly acceptable (and in some situations even
|
| 56 | preferable) solution to represent each individual byte of a malformed
|
| 57 | sequence with a replacement character. If you follow this strategy in
|
| 58 | your decoder, then please ignore the "|" column.
|
| 59 |
|
| 60 |
|
| 61 | Here come the tests: |
|
| 62 | |
|
| 63 | 1 Some correct UTF-8 text |
|
| 64 | |
|
| 65 | You should see the Greek word 'kosme': "κόσμε" |
|
| 66 | |
|
| 67 | 2 Boundary condition test cases |
|
| 68 | |
|
| 69 | 2.1 First possible sequence of a certain length |
|
| 70 | |
|
| 71 | 2.1.1 1 byte (U-00000000): " |
| 72 | 2.1.2 2 bytes (U-00000080): "" |
|
| 73 | 2.1.3 3 bytes (U-00000800): "ࠀ" |
|
| 74 | 2.1.4 4 bytes (U-00010000): "𐀀" |
|
| 75 | 2.1.5 5 bytes (U-00200000): "" |
|
| 76 | 2.1.6 6 bytes (U-04000000): "" |
|
| 77 | |
|
| 78 | 2.2 Last possible sequence of a certain length |
|
| 79 | |
|
| 80 | 2.2.1 1 byte (U-0000007F): ""
|
| 81 | 2.2.2 2 bytes (U-000007FF): "߿" |
|
| 82 | 2.2.3 3 bytes (U-0000FFFF): "" |
|
| 83 | 2.2.4 4 bytes (U-001FFFFF): "" |
|
| 84 | 2.2.5 5 bytes (U-03FFFFFF): "" |
|
| 85 | 2.2.6 6 bytes (U-7FFFFFFF): "" |
|
| 86 | |
|
| 87 | 2.3 Other boundary conditions |
|
| 88 | |
|
| 89 | 2.3.1 U-0000D7FF = ed 9f bf = "" |
|
| 90 | 2.3.2 U-0000E000 = ee 80 80 = "" |
|
| 91 | 2.3.3 U-0000FFFD = ef bf bd = "�" |
|
| 92 | 2.3.4 U-0010FFFF = f4 8f bf bf = "" |
|
| 93 | 2.3.5 U-00110000 = f4 90 80 80 = "" |
|
| 94 | |
|
| 95 | 3 Malformed sequences |
|
| 96 | |
|
| 97 | 3.1 Unexpected continuation bytes |
|
| 98 | |
|
| 99 | Each unexpected continuation byte should be separately signalled as a |
|
| 100 | malformed sequence of its own. |
|
| 101 | |
|
| 102 | 3.1.1 First continuation byte 0x80: "" |
|
| 103 | 3.1.2 Last continuation byte 0xbf: "" |
|
| 104 | |
|
| 105 | 3.1.3 2 continuation bytes: "" |
|
| 106 | 3.1.4 3 continuation bytes: "" |
|
| 107 | 3.1.5 4 continuation bytes: "" |
|
| 108 | 3.1.6 5 continuation bytes: "" |
|
| 109 | 3.1.7 6 continuation bytes: "" |
|
| 110 | 3.1.8 7 continuation bytes: "" |
|
| 111 | |
|
| 112 | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |
|
| 113 | |
|
| 114 | " |
|
| 115 | |
|
| 116 | |
|
| 117 | " |
|
| 118 | |
|
| 119 | 3.2 Lonely start characters |
|
| 120 | |
|
| 121 | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |
|
| 122 | each followed by a space character: |
|
| 123 | |
|
| 124 | " |
|
| 125 | " |
|
| 126 | |
|
| 127 | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |
|
| 128 | each followed by a space character: |
|
| 129 | |
|
| 130 | " " |
|
| 131 | |
|
| 132 | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |
|
| 133 | each followed by a space character: |
|
| 134 | |
|
| 135 | " " |
|
| 136 | |
|
| 137 | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |
|
| 138 | each followed by a space character: |
|
| 139 | |
|
| 140 | " " |
|
| 141 | |
|
| 142 | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |
|
| 143 | each followed by a space character: |
|
| 144 | |
|
| 145 | " " |
|
| 146 | |
|
| 147 | 3.3 Sequences with last continuation byte missing |
|
| 148 | |
|
| 149 | All bytes of an incomplete sequence should be signalled as a single |
|
| 150 | malformed sequence, i.e., you should see only a single replacement |
|
| 151 | character in each of the next 10 tests. (Characters as in section 2) |
|
| 152 | |
|
| 153 | 3.3.1 2-byte sequence with last byte missing (U+0000): "" |
|
| 154 | 3.3.2 3-byte sequence with last byte missing (U+0000): "" |
|
| 155 | 3.3.3 4-byte sequence with last byte missing (U+0000): "" |
|
| 156 | 3.3.4 5-byte sequence with last byte missing (U+0000): "" |
|
| 157 | 3.3.5 6-byte sequence with last byte missing (U+0000): "" |
|
| 158 | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" |
|
| 159 | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" |
|
| 160 | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" |
|
| 161 | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" |
|
| 162 | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" |
|
| 163 | |
|
| 164 | 3.4 Concatenation of incomplete sequences |
|
| 165 | |
|
| 166 | All the 10 sequences of 3.3 concatenated, you should see 10 malformed |
|
| 167 | sequences being signalled: |
|
| 168 | |
|
| 169 | "" |
|
| 170 | |
|
| 171 | 3.5 Impossible bytes |
|
| 172 | |
|
| 173 | The following two bytes cannot appear in a correct UTF-8 string |
|
| 174 | |
|
| 175 | 3.5.1 fe = "" |
|
| 176 | 3.5.2 ff = "" |
|
| 177 | 3.5.3 fe fe ff ff = "" |
|
| 178 | |
|
| 179 | 4 Overlong sequences |
|
| 180 | |
|
| 181 | The following sequences are not malformed according to the letter of |
|
| 182 | the Unicode 2.0 standard. However, they are longer then necessary and |
|
| 183 | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 |
|
| 184 | decoder" should reject them just like malformed sequences for two |
|
| 185 | reasons: (1) It helps to debug applications if overlong sequences are |
|
| 186 | not treated as valid representations of characters, because this helps |
|
| 187 | to spot problems more quickly. (2) Overlong sequences provide |
|
| 188 | alternative representations of characters, that could maliciously be |
|
| 189 | used to bypass filters that check only for ASCII characters. For |
|
| 190 | instance, a 2-byte encoded line feed (LF) would not be caught by a |
|
| 191 | line counter that counts only 0x0a bytes, but it would still be |
|
| 192 | processed as a line feed by an unsafe UTF-8 decoder later in the |
|
| 193 | pipeline. From a security point of view, ASCII compatibility of UTF-8 |
|
| 194 | sequences means also, that ASCII characters are *only* allowed to be |
|
| 195 | represented by ASCII bytes in the range 0x00-0x7f. To ensure this |
|
| 196 | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that |
|
| 197 | reject overlong UTF-8 sequences for which a shorter encoding exists. |
|
| 198 | |
|
| 199 | 4.1 Examples of an overlong ASCII character |
|
| 200 | |
|
| 201 | With a safe UTF-8 decoder, all of the following five overlong |
|
| 202 | representations of the ASCII character slash ("/") should be rejected |
|
| 203 | like a malformed UTF-8 sequence, for instance by substituting it with |
|
| 204 | a replacement character. If you see a slash below, you do not have a |
|
| 205 | safe UTF-8 decoder! |
|
| 206 | |
|
| 207 | 4.1.1 U+002F = c0 af = "" |
|
| 208 | 4.1.2 U+002F = e0 80 af = "" |
|
| 209 | 4.1.3 U+002F = f0 80 80 af = "" |
|
| 210 | 4.1.4 U+002F = f8 80 80 80 af = "" |
|
| 211 | 4.1.5 U+002F = fc 80 80 80 80 af = "" |
|
| 212 | |
|
| 213 | 4.2 Maximum overlong sequences |
|
| 214 | |
|
| 215 | Below you see the highest Unicode value that is still resulting in an |
|
| 216 | overlong sequence if represented with the given number of bytes. This |
|
| 217 | is a boundary test for safe UTF-8 decoders. All five characters should |
|
| 218 | be rejected like malformed UTF-8 sequences. |
|
| 219 | |
|
| 220 | 4.2.1 U-0000007F = c1 bf = "" |
|
| 221 | 4.2.2 U-000007FF = e0 9f bf = "" |
|
| 222 | 4.2.3 U-0000FFFF = f0 8f bf bf = "" |
|
| 223 | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" |
|
| 224 | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" |
|
| 225 | |
|
| 226 | 4.3 Overlong representation of the NUL character |
|
| 227 | |
|
| 228 | The following five sequences should also be rejected like malformed |
|
| 229 | UTF-8 sequences and should not be treated like the ASCII NUL |
|
| 230 | character. |
|
| 231 | |
|
| 232 | 4.3.1 U+0000 = c0 80 = "" |
|
| 233 | 4.3.2 U+0000 = e0 80 80 = "" |
|
| 234 | 4.3.3 U+0000 = f0 80 80 80 = "" |
|
| 235 | 4.3.4 U+0000 = f8 80 80 80 80 = "" |
|
| 236 | 4.3.5 U+0000 = fc 80 80 80 80 80 = "" |
|
| 237 | |
|
| 238 | 5 Illegal code positions |
|
| 239 | |
|
| 240 | The following UTF-8 sequences should be rejected like malformed |
|
| 241 | sequences, because they never represent valid ISO 10646 characters and |
|
| 242 | a UTF-8 decoder that accepts them might introduce security problems |
|
| 243 | comparable to overlong UTF-8 sequences. |
|
| 244 | |
|
| 245 | 5.1 Single UTF-16 surrogates |
|
| 246 | |
|
| 247 | 5.1.1 U+D800 = ed a0 80 = "" |
|
| 248 | 5.1.2 U+DB7F = ed ad bf = "" |
|
| 249 | 5.1.3 U+DB80 = ed ae 80 = "" |
|
| 250 | 5.1.4 U+DBFF = ed af bf = "" |
|
| 251 | 5.1.5 U+DC00 = ed b0 80 = "" |
|
| 252 | 5.1.6 U+DF80 = ed be 80 = "" |
|
| 253 | 5.1.7 U+DFFF = ed bf bf = "" |
|
| 254 | |
|
| 255 | 5.2 Paired UTF-16 surrogates |
|
| 256 | |
|
| 257 | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" |
|
| 258 | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" |
|
| 259 | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" |
|
| 260 | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" |
|
| 261 | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" |
|
| 262 | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" |
|
| 263 | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" |
|
| 264 | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" |
|
| 265 | |
|
| 266 | 5.3 Noncharacter code positions |
|
| 267 | |
|
| 268 | The following "noncharacters" are "reserved for internal use" by |
|
| 269 | applications, and according to older versions of the Unicode Standard |
|
| 270 | "should never be interchanged". Unicode Corrigendum #9 dropped the |
|
| 271 | latter restriction. Nevertheless, their presence in incoming UTF-8 data |
|
| 272 | can remain a potential security risk, depending on what use is made of |
|
| 273 | these codes subsequently. Examples of such internal use: |
|
| 274 | |
|
| 275 | - Some file APIs with 16-bit characters may use the integer value -1 |
|
| 276 | = U+FFFF to signal an end-of-file (EOF) or error condition. |
|
| 277 | |
|
| 278 | - In some UTF-16 receivers, code point U+FFFE might trigger a |
|
| 279 | byte-swap operation (to convert between UTF-16LE and UTF-16BE). |
|
| 280 | |
|
| 281 | With such internal use of noncharacters, it may be desirable and safer |
|
| 282 | to block those code points in UTF-8 decoders, as they should never |
|
| 283 | occur legitimately in incoming UTF-8 data, and could trigger unsafe |
|
| 284 | behaviour in subsequent processing. |
|
| 285 | |
|
| 286 | Particularly problematic noncharacters in 16-bit applications: |
|
| 287 | |
|
| 288 | 5.3.1 U+FFFE = ef bf be = "" |
|
| 289 | 5.3.2 U+FFFF = ef bf bf = "" |
|
| 290 | |
|
| 291 | Other noncharacters: |
|
| 292 | |
|
| 293 | 5.3.3 U+FDD0 .. U+FDEF = ""|
|
| 294 | |
|
| 295 | 5.3.4 U+nFFFE U+nFFFF (for n = 1..10) |
|
| 296 | |
|
| 297 | " |
|
| 298 | " |
|
| 299 | |
|
| 300 | THE END |
|