alan

unicode

So, here we come to our second part; if you did miss the first, it's there. All of the following discussion is relative to Sun Java 6, but most details – if not every detail – is still valid for Java 7 and 8.

Let's see the way Java handles unicode issues and why it's usually less problematic than Python – or, at least, while it looks less problematic than Python.

String literals in Java, just like in Python 3, are just Unicode objects; so you don't need to think whether it's a “binary string” or a “unicode object”; if it's a binary blob, you'll probably get a byte[], which should result in a 'here be dragons' behaviour about encoding.

Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?

System.out, which is an instance of java.io.PrintStream, internally uses a java.io.OutputStreamWriter which gets initialized using [Charset.defaultCharset()](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset(\)), and such value will be used for conversions between raw bytes and strings:

Encoding.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.nio.charset.Charset;

public class Encoding {
    public static void main(String[] args) throws Exception {
        String myString = "àèìòù";
        System.out.println(System.out.getClass());
        System.out.println(Charset.defaultCharset());
        System.out.println(myString);
        System.out.write(myString.getBytes(Charset.defaultCharset()));
        System.out.println("");
    }
}

result:

class java.io.PrintStream
UTF-8
àèìòù

The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that's discouraged as it can be unpredictable and unreliable. On the contrary, you should always use constructors and methods that accept a charset parameter if you're converting to/from raw bytes, just like String.getBytes(Charset charset) or [OutputStreamWriter(OutputStream outputStream, Charset charset)](http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter\(java.io.OutputStream, java.nio.charset.Charset))

So far, so good. But what happens if the outputstream is unable to handle the chars you'd like it to write?

CantEncode.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.io.FileOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class CantEncode {
    public static void main(String[] args) throws Exception {
        // change default output stream
        OutputStream outputStream = new FileOutputStream("eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output");
        PrintStream printStream = new PrintStream(outputStream, true, "ASCII");
        System.setOut(printStream);

        String myString = "àèìòù";

        System.out.println(System.out.getClass());
        System.out.println(myString);
     }
}

result:

class java.io.PrintStream
?????

ASCII encoding cannot possibly output the accented chars, so they're replaced with question marks instead of throwing the dreaded UnicodeEncodeError.

Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate 'heads up!'. Otherwise if you're just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.

But there's another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you'll use javac -encoding flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; if you don't pass such option, the platform default converter is used

What's the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.

Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That's what happens, in fact:

echo "utf8 compiled:"
javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding
echo "windows-1252 compiled:"
javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding

result:

utf8 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù
windows-1252 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù

As you can see there, it's not that the JVM encoding has any issue at runtime; it's the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There's no way around this!

What I can recommend you is to ALWAYS set the project.build.sourceEncoding property when using Maven to compile, and to ALWAYS set the encoding attribute whenever you're using Ant. This will save you a lot, lot, lot of headaches.

#Ollivander #java #python #unicode #encodings

Before going on with this post, be sure you've read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) - otherwise, I won't be able to solve your issues.

Sometimes Java fans just say that Python Unicode support sucks. While there's a bit of truth in that assertion – for Python 2.x at least, since 3.x solved the problem at its root – the real problem lies in the fact that many programmers don't know what's going on “under the hood”, and that python default behaviour is a bit unforgiving.

First things first: Python 2.x has got two distinct string types: the so-called “byte strings” (str type) are one, and unicode objects (unicode type) are another one. Unicode objects are much like Java strings; they're an internal abstraction of Python, and need to be interpreted to/from byte strings whenever printing, reading/writing from/to a file, etc.

Sometimes such interpreting “just works” (most probably if you're dealing with ascii-only text), but if such conversion fails, errors occur: Python infamous and dreaded UnicodeDecodeError can happen in many places and might sometime just puzzle the programmer:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù” + u”asd”

javapythonunicode$ python unicode_concat.py
current default encoding: ascii
Traceback (most recent call last):
File “unicode_concat.py”, line 6, in <module>
“àèìòù” + u”asd”
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

 This happens when trying to concatenate an unicode string to string object. What really happens here, behind the scenes, is explained in unicode_concat_explained.py:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù”.decode(sys.getdefaultencoding()) + u”asd”

When using a byte string (not a unicode object) python tries to convert that string to unicode before concatenating to a unicode object. Since an encoding wasn't specified, python just uses the default.

So, if you're planning to mix unicode and bytestring objects (which is not a good idea), always remember to explicitly convert string objects to unicode by their decode() method. Or, if you're sure the encoding is always the same, you could add a sitecustomize.py to your interpreter or your project PYTHONPATH, and set such encoding. Beware that setting it system wide may lead to unexpected results when running your project on another machine.

Also, just remember that the coding directive at the begininning of the file does not change unicode encoding/decoding behaviour: that's a directive to the parser, which is used to instruct it about the current encoding of the file. That information is not retained nor used at runtime.


Let's go for another much dreaded error now: UnicodeEncodeError!

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

print u”àèìòù”

results in this output:

javapythonunicode$ python unicode_print.py
current stdout encoding: UTF-8
àèìòù

Everything seems to work (as long as your terminal supports a charset, like UTF-8, or iso-8859-1, which can display accented chars) ! But now let's go for some “black magic”:


That may be puzzling, but the sys.stdout printout should give you an hint.

Whenever output is to a terminal, Python does perform autodetection of the encoding of your terminal, and subsequently sets sys.stdout accordingly. When sys.stdout encoding is set, any unicode object that gets printed is encoded accordingly. When output is NOT to a terminal, python does not autodetect sys.stdout encoding and, when sys.stdout has no encoding set any unicode object that is printed is converted according to sys.getdefaultencoding()

So, what really happens here is:

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

sys.stdout.write(
u”àèìòù”.encode(sys.stdout.encoding or sys.getdefaultencoding())
)

Since accented characters can be encoded to UTF-8 but not to ASCII, such UnicodeEncodeErrors arise.

Also beware of bug 4947 - it may hit you if you're using Python 2.6 or or older.

You can find the second part of this article here

#Ollivander #python #java #unicode #encodings