Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Not extremely unlikely. Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.


> Many charsets decode fine as UTF-8 as long as the message happens to fit in ASCII.

At which point the message is effectively ASCII. UTF-8 is a superset of ASCII, so "decoding" ASCII as UTF-8 is fine.

(Yes, I know there are some Japanese text encodings where 0x5c is decoded as "¥" instead of "\". But they're sometimes treated as backslashes even though they look like ¥ symbols so handling them "correctly" is complicated.)


"Fun" fact: some video subtitle formats (ASS specifically) use "text{\b1}bold" to format things — but since they were primarily used to subtitle Japanese anime, this frequently became "text{¥b1}bold". Which is all good and well, except when those subtitles moved to UTF-8 they kept the ¥. So now you have to support ¥ (0xC2 0xA5) as a markup/control character in those subtitles.


I'm guessing they're thinking of Extended ASCII (the 8-bit one that's actually multiple different encodings, but the lower half is shared with 7-bit ASCII and so that part does fit in UTF-8 while the upper half likely won't if the message actually uses it).


Yes. And some test messages will decode fine and then suddenly no.


ISO-2022-JP (sometimes?) disguises perfectly as ASCII:

  $ echo は | iconv -t ISO-2022-JP | hd
  00000000  1b 24 42 24 4f 1b 28 42  0a                       |.$B$O.(B.|
  00000009


The definition GP is using most likely refers to non-ASCII sequences that validly decode as UTF-8, because virtually every major charset in practical use has ASCII as a subset.


ASCII (or ISO646) has local variants [0] that replace certain characters, like $ with £, or \ with ¥. The latter is still in use in Japan. That’s why "ASCII" is sometimes clarified as "US-ASCII".

[0] https://en.wikipedia.org/wiki/ASCII#7-bit_codes


I'm skeptical. Any charset that uses bytes 128-255 as characters is unlikely to successfully decode to UTF-8. Are there really many others that only use 0-127, or most text ends up only using 0-127?


such encodings are also UTF-8 then, are they not?


I think there are a bunch of encodings that just repurposed a few ASCII characters as different characters - someone on this page was giving the example of some Swedish encoding where {}| were replaced with three accented Swedish letters. There are probably a bunch of others. In those cases, the text will decode fine as UTF-8, but it will display the wrong thing.


A distinction without a difference?


I meant, some messages will decode fine as UTF-8, but in some other messages there may be letters which don't fit in 7 bits. So some simple testing, especially with English words, will show it to work fine. But as soon as a non-7-bit characters creeps, it will stop working fine.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: