Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2002-11-08
DHTML version Simon Montagu <smontagu@netscape.com> - 2003-03-07
My additions to Markus’ original text and testcases appear in blue.
The original version is http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt.
This test text examines, how UTF-8 decoders handle various types of corrupted or otherwise interesting UTF-8 sequences.
According to ISO 10646-1, sections R.7 and 2.3c, a device receiving UTF-8 shall interpret a “malformed sequence in the same way that it interprets a character that is outside the adopted subset”. This means usually that the malformed UTF-8 sequence is replaced by a replacement character (U+FFFD), which looks a bit like an inverted question mark, or a similar symbol. It might be a good idea to visually distinguish a malformed UTF-8 sequence from a correctly encoded Unicode character that is just not available in the current font but otherwise fully legal. For both cases, a clearly recognisable symbol should be used. Just ignoring malformed sequences or unavailable characters will make debugging more difficult and can lead to user confusion.
Test sequences:
Check, whether a malformed UTF-8 sequence is (1) represented at all, (2) represented by exactly one replacement character (or equivalent signal), and (3) the following quotation mark after an illegal UTF-8 sequence is correctly displayed, i.e. proper resynchronization takes place. This file says “THE END” in the last line, so if you don’t see that, your decoder crashed somehow before, which is also not nice.
In the DHTML version the output of the UTF-8 decoder is compared to the “correct” output, encoded as NCRs and including the quotation marks, and marked as PASS or FAIL. “Correctness” here is entirely my personal opinion. Markus has given me permission to reuse his original tests, but I don’t take that as endorsement of my interpretation of the standards.
I accept either “?” (U+003F) or “�” (U+FFFD) for malformed sequences or illegal characters. Allowing “?” is really a sop to IE, which would otherwise fail all the malformed sequence tests.
Some of the tests may fail even if there is no problem with the UTF-8 decoder, because the encoded character is undefined in the SGML character set. These cases are marked with a comment “Undefined character!”. Some others test codepoints beyond U+10FFFF, which are illegal according to Unicode 3.1 and Amendment 1 to ISO/IEC 10646-1:2000(E). These cases are marked with a comment “Out of range!”, and the expected output is defined as the same as for illegal characters.
I omit the last two paragraphs of Markus’ introduction, which are not relevant to this version.
Here come the tests:
You should see the Greek word “kosme”: “κόσμε” “κόσμε”
2.1.1 1 byte (U-00000000): [Test omitted] Undefined character!
2.1.2 2 bytes (U-00000080): “” “€” Undefined character!
2.1.3 3 bytes (U-000000800): “ࠀ” “ࠀ”
2.1.4 4 bytes (U-000010000): “𐀀” “𐀀”
2.1.5 5 bytes (U-00200000): “” “?” Out of range!
2.1.6 6 bytes (U-04000000): “” “?” Out of range!
2.2.1 1 byte (U-0000007F): &ldquo” “” Undefined character!
2.2.2 2 bytes (U-000007FF): “߿” “߿”
2.2.3 3 bytes (U-0000FFFF): “” “” This case seems to contradict case 5.3.2, which gives U+FFFF as an “illegal code position”. See my comment on section 5.3
2.2.4 4 bytes (U-001FFFFF): “” “?” Out of range!
2.2.5 5 bytes (U-03FFFFFF): “” “?” Out of range!
2.2.6 6 bytes (U-7FFFFFFF): “” “?” Out of range!
2.3.1 U-0000D7FF = ed 9f bf: “” “”
2.3.2 U-0000E000 = ee 80 80: “” “”
2.3.3 U-0000FFFD = ef bf bd: “�” “�”
2.3.4 U-0010FFFF = f4 8f bf bf: “” “”
2.3.5 U-00110000 = f4 90 80 80: “” “?” Out of range!
Each unexpected continuation byte should be separately signalled as a malformed sequence of its own.
3.1.1 First continuation byte 0x80: “” “?”
3.1.2 Last continuation byte 0xbf: “” “?”
3.1.3 2 continuation bytes: “” “??”
3.1.4 3 continuation bytes: “” “???”
3.1.5 4 continuation bytes: “” “????”
3.1.6 5 continuation bytes: “” “?????”
3.1.7 6 continuation bytes: “” “??????”
3.1.8 7 continuation bytes: “” “???????”
3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): “” “????????????????????????????????????????????????????????????????”
3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), each followed by a space character: “ ” “? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ”
3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), each followed by a space character: “ ” “? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ”
3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), each followed by a space character: “ ” “? ? ? ? ? ? ? ? ”
3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), each followed by a space character: “ ” “? ? ? ? ”
3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), each followed by a space character: “ ” “? ? ”
All bytes of an incomplete sequence should be signalled as a single malformed sequence, i.e., you should see only a single replacement character in each of the next 10 tests. (Characters as in section 2)
3.3.1 2-byte sequence with last byte missing (U+0000): “” “?”
3.3.2 3-byte sequence with last byte missing (U+0000): “” “?”
3.3.3 4-byte sequence with last byte missing (U+0000): “” “?”
3.3.4 5-byte sequence with last byte missing (U+0000): “” “?”
3.3.5 6-byte sequence with last byte missing (U+0000): “” “?”
3.3.6 2-byte sequence with last byte missing (U-000007FF): “” “?”
3.3.7 3-byte sequence with last byte missing (U-0000FFFF): “” “?”
3.3.8 4-byte sequence with last byte missing (U-001FFFFF): “” “?”
3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): “” “?”
3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): “” “?”
All the 10 previous sequences concatenated, you should see 10 malformed sequences being signalled: “” “??????????”
The following two bytes cannot appear in a correct UTF-8 string
3.5.1 fe: “” “?”
3.5.2 ff: “” “?”
3.5.3 fe fe ff ff: “” “????”
The following sequences are not malformed according to the letter of the Unicode 2.0 standard. However, they are longer then necessary and a correct UTF-8 encoder is not allowed to produce them. A “safe UTF-8 decoder” should reject them just like malformed sequences for two reasons: (1) It helps to debug applications if overlong sequences are not treated as valid representations of characters, because this helps to spot problems more quickly. (2) Overlong sequences provide alternative representations of characters, that could maliciously be used to bypass filters that check only for ASCII characters. For instance, a 2-byte encoded line feed (LF) would not be caught by a line counter that counts only 0x0a bytes, but it would still be processed as a line feed by an unsafe UTF-8 decoder later in the pipeline. From a security point of view, ASCII compatibility of UTF-8 sequences means also, that ASCII characters are only allowed to be represented by ASCII bytes in the range 0x00-0x7f. To ensure this aspect of ASCII compatibility, use only “safe UTF-8 decoders” that reject overlong UTF-8 sequences for which a shorter encoding exists.
With a safe UTF-8 decoder, all of the following five overlong representations of the ASCII character slash (“/”) should be rejected like a malformed UTF-8 sequence, for instance by substituting it with a replacement character. If you see a slash below, you do not have a safe UTF-8 decoder!
4.1.1 U+002F = c0 af: “” “?”
4.1.2 U+002F = e0 80 af: “” “?”
4.1.3 U+002F = f0 80 80 af “” “?”
4.1.4 U+002F = f8 80 80 80 af: “” “?”
4.1.5 U+002F = fc 80 80 80 80 af: “” “?”
Below you see the highest Unicode value that is still resulting in an overlong sequence if represented with the given number of bytes. This is a boundary test for safe UTF-8 decoders. All five characters should be rejected like malformed UTF-8 sequences.
4.2.1 U-0000007F = c1 bf: “” “?”
4.2.2 U-000007FF = e0 9f bf: “” “?”
4.2.3 U-0000FFFF = f0 8f bf bf: “” “?”
4.2.4 U-001FFFFF = f8 87 bf bf bf: “” “?”
4.2.5 U-03FFFFFF = fc 83 bf bf bf bf: “” “?”
4.3.1 U+0000 = c0 80: “” “?”
4.3.2 U+0000 = e0 80 80: “” “?”
4.3.3 U+0000 = f0 80 80 80: “” “?”
4.3.4 U+0000 = f8 80 80 80 80: “” “?”
4.3.5 U+0000 = fc 80 80 80 80 80: “” “?”
The following UTF-8 sequences should be rejected like malformed sequences, because they never represent valid ISO 10646 characters and a UTF-8 decoder that accepts them might introduce security problems comparable to overlong UTF-8 sequences.
5.1.1 U+D800 = ed a0 80: “” “?”
5.1.2 U+DB7F = ed ad bf: “” “?”
5.1.3 U+DB80 = ed ae 80: “” “?”
5.1.4 U+DBFF = ed af bf: “” “?”
5.1.2 U+DC00 = ed b0 80: “” “?”
5.1.3 U+DF80 = ed be 80: “” “?”
5.1.4 U+DFFF = ed bf bf: “” “?”
5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80: “” “??”
5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf: “” “??”
5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80: “” “??”
5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf: “” “??”
5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80: “” “??”
5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf: “” “??”
5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80: “” “??”
5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf: “” “??”
As I understand chapter 3 of version 3.2 of the Unicode standard, these tests should pass, but if they fail the problem is not with the UTF-8 decoder. Paragraph C5 says “A process shall not interpret either U+FFFE or U+FFFF as an abstract character,” but table 3.1B includes them among the “Legal UTF-8 Byte Sequences.”
5.3.1 U+FFFE = ef bf be: “” “?”
5.3.2 U+FFFF = ef bf be: “” “?”