Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I don't think that's true. Looking at how it's encoded[0], it seems similar to many other country/language-specific encodings: bytes 0-127 are the control chars and latin alphabet and symbols, and is more-or-less ASCII, then 128-255 represent characters specific to the language at hand.

The only way you'd successfully decode Shift-JIS as UTF-8 is if it essentially is just latin-alphabet text (though the yen symbol would incorrectly display as a '\'). If it includes any non-trival amount of Japanese in it, it'll fail to decode as UTF-8.

As for whether or not you can then (after it fails to decode as UTF-8) use statistical analysis to reliably figure out that it's in fact Shift-JIS, and not something else, I can't speak to that.

[0] https://en.wikipedia.org/wiki/Shift_JIS#Shift_JIS_byte_map



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: