Beware of Character Encoding during Cut & Paste of Websites

The issue is very simple;

Do not assume that a website written in “Shift-JIS” will only contain characters that can be represented in “Shift-JIS”.

If you copy any characters that cannot be represented in “Shift-JIS” and paste them to another web page also coded in “Shift-JIS”, it may generate garbled-text (Mojibake).

For example, you may copy text from a website coded in Shift-JIS which contains ™, © or ® to code , © or ®. These characters are not available in Shift-JIS. If you paste them in your webpage which is also in Shift-JIS, you will see garbled-text or a “?” sign (Mojibake).

Another example is if the page has an element that is loaded via Ajax. The Ajax payload will be handled by Javascript, which will handle the payload as Unicode. It is capable of inserting characters into the DOM that are not representable in Shift-JIS.

The thing to keep in mind is that the HTML character set is only a transfer protocol. It does not govern or limit in any way the text that can be displayed on the browser. Hence you cannot assume that all the text that you see on the browser is encodable in a particular encoding except Unicode.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: