alan

java

So, here we come to our second part; if you did miss the first, it's there. All of the following discussion is relative to Sun Java 6, but most details – if not every detail – is still valid for Java 7 and 8.

Let's see the way Java handles unicode issues and why it's usually less problematic than Python – or, at least, while it looks less problematic than Python.

String literals in Java, just like in Python 3, are just Unicode objects; so you don't need to think whether it's a “binary string” or a “unicode object”; if it's a binary blob, you'll probably get a byte[], which should result in a 'here be dragons' behaviour about encoding.

Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?

System.out, which is an instance of java.io.PrintStream, internally uses a java.io.OutputStreamWriter which gets initialized using [Charset.defaultCharset()](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset(\)), and such value will be used for conversions between raw bytes and strings:

Encoding.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.nio.charset.Charset;

public class Encoding {
    public static void main(String[] args) throws Exception {
        String myString = "àèìòù";
        System.out.println(System.out.getClass());
        System.out.println(Charset.defaultCharset());
        System.out.println(myString);
        System.out.write(myString.getBytes(Charset.defaultCharset()));
        System.out.println("");
    }
}

result:

class java.io.PrintStream
UTF-8
àèìòù

The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that's discouraged as it can be unpredictable and unreliable. On the contrary, you should always use constructors and methods that accept a charset parameter if you're converting to/from raw bytes, just like String.getBytes(Charset charset) or [OutputStreamWriter(OutputStream outputStream, Charset charset)](http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter\(java.io.OutputStream, java.nio.charset.Charset))

So far, so good. But what happens if the outputstream is unable to handle the chars you'd like it to write?

CantEncode.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.io.FileOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class CantEncode {
    public static void main(String[] args) throws Exception {
        // change default output stream
        OutputStream outputStream = new FileOutputStream("eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output");
        PrintStream printStream = new PrintStream(outputStream, true, "ASCII");
        System.setOut(printStream);

        String myString = "àèìòù";

        System.out.println(System.out.getClass());
        System.out.println(myString);
     }
}

result:

class java.io.PrintStream
?????

ASCII encoding cannot possibly output the accented chars, so they're replaced with question marks instead of throwing the dreaded UnicodeEncodeError.

Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate 'heads up!'. Otherwise if you're just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.

But there's another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you'll use javac -encoding flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; if you don't pass such option, the platform default converter is used

What's the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.

Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That's what happens, in fact:

echo "utf8 compiled:"
javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding
echo "windows-1252 compiled:"
javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding

result:

utf8 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù
windows-1252 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù

As you can see there, it's not that the JVM encoding has any issue at runtime; it's the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There's no way around this!

What I can recommend you is to ALWAYS set the project.build.sourceEncoding property when using Maven to compile, and to ALWAYS set the encoding attribute whenever you're using Ant. This will save you a lot, lot, lot of headaches.

#Ollivander #java #python #unicode #encodings

Before going on with this post, be sure you've read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) - otherwise, I won't be able to solve your issues.

Sometimes Java fans just say that Python Unicode support sucks. While there's a bit of truth in that assertion – for Python 2.x at least, since 3.x solved the problem at its root – the real problem lies in the fact that many programmers don't know what's going on “under the hood”, and that python default behaviour is a bit unforgiving.

First things first: Python 2.x has got two distinct string types: the so-called “byte strings” (str type) are one, and unicode objects (unicode type) are another one. Unicode objects are much like Java strings; they're an internal abstraction of Python, and need to be interpreted to/from byte strings whenever printing, reading/writing from/to a file, etc.

Sometimes such interpreting “just works” (most probably if you're dealing with ascii-only text), but if such conversion fails, errors occur: Python infamous and dreaded UnicodeDecodeError can happen in many places and might sometime just puzzle the programmer:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù” + u”asd”

javapythonunicode$ python unicode_concat.py
current default encoding: ascii
Traceback (most recent call last):
File “unicode_concat.py”, line 6, in <module>
“àèìòù” + u”asd”
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

 This happens when trying to concatenate an unicode string to string object. What really happens here, behind the scenes, is explained in unicode_concat_explained.py:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù”.decode(sys.getdefaultencoding()) + u”asd”

When using a byte string (not a unicode object) python tries to convert that string to unicode before concatenating to a unicode object. Since an encoding wasn't specified, python just uses the default.

So, if you're planning to mix unicode and bytestring objects (which is not a good idea), always remember to explicitly convert string objects to unicode by their decode() method. Or, if you're sure the encoding is always the same, you could add a sitecustomize.py to your interpreter or your project PYTHONPATH, and set such encoding. Beware that setting it system wide may lead to unexpected results when running your project on another machine.

Also, just remember that the coding directive at the begininning of the file does not change unicode encoding/decoding behaviour: that's a directive to the parser, which is used to instruct it about the current encoding of the file. That information is not retained nor used at runtime.


Let's go for another much dreaded error now: UnicodeEncodeError!

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

print u”àèìòù”

results in this output:

javapythonunicode$ python unicode_print.py
current stdout encoding: UTF-8
àèìòù

Everything seems to work (as long as your terminal supports a charset, like UTF-8, or iso-8859-1, which can display accented chars) ! But now let's go for some “black magic”:


That may be puzzling, but the sys.stdout printout should give you an hint.

Whenever output is to a terminal, Python does perform autodetection of the encoding of your terminal, and subsequently sets sys.stdout accordingly. When sys.stdout encoding is set, any unicode object that gets printed is encoded accordingly. When output is NOT to a terminal, python does not autodetect sys.stdout encoding and, when sys.stdout has no encoding set any unicode object that is printed is converted according to sys.getdefaultencoding()

So, what really happens here is:

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

sys.stdout.write(
u”àèìòù”.encode(sys.stdout.encoding or sys.getdefaultencoding())
)

Since accented characters can be encoded to UTF-8 but not to ASCII, such UnicodeEncodeErrors arise.

Also beware of bug 4947 - it may hit you if you're using Python 2.6 or or older.

You can find the second part of this article here

#Ollivander #python #java #unicode #encodings

Heroku Java support is something I longed for, but they apparently don't support deploying a plain, old war file to Heroku.

I don't like that; I've got my own continuous integration setup, and I want to compile my files on my own box – this allows me to keep my own Maven proxy with my own dependencies – which sometimes are patched versions of public ones –  so that I don't need to push everything on a public repo. I don't want to be forced to build everything on Heroku, and I don't want to compile anything I push – I'd usually push to Heroku stable versions only, or maybe integration versions that pass unit, integration and acceptance tests.

So what? Don't worry, deployment is possible.



Setup an heroku-aware git repository and a Cedar stack app, and add the files you see above; your dir structure should look like:

pom.xml
Procfile
src/
     main/
             java/
                    root/
                           StartWebApp.java
             resources/
                             mywebapp.war


Now a simple

git push heroku master


Should do the trick, and let you see your fully deployed war app.


There're still things you need to be aware of; file persistence may not work as expected, since filesystem is ephemeral, and I still need to figure out things related to database configuration, but those should not come at a great expense.

I'll post updates in the future as soon as I get them right.



#Ollivander #java #heroku #war

Small primer for mock javamail, a pretty useful project that will be handy whenever you need to test code that uses POP3/IMAP/SMTP and you wouldn't like to build wrappers for anything you need to test – javamail does not really offer interfaces that can be implemented by mock objects, but different implementations can be used at runtime depending on which jars are on the classpath.

Take a look at the project homepage:

http://java.net/projects/mock-javamail

Be sure you've enabled java.net Maven repo, and add this dependency to your pom.xml:

1  <dependency>

2 <groupId>org.jvnet.mock-javamail</groupId>
3 <artifactId>mock-javamail</artifactId>
4 <version>1.9</version>
5 <scope>test</scope>
6 </dependency>

You're almost ready; now setup your test code this way:

 1 public class IMAPMailRepositoryTest {

2 @Before
3 public void setUp() throws Exception {
4 final Session session = Session.getInstance(
5 System.getProperties());
6
7
MimeMessage msg = new MimeMessage(
8 session);
9 msg.setRecipients(Message.RecipientType.TO,
10 “testuser@mockserver.com“);
11 msg.setSubject(“Some Subject”);
12 msg.setText(“sometext”);
13 Transport.send(msg);
14 }
15
16
@After
17 public void tearDown() throws Exception {
18 Mailbox.clearAll();
19 }



Now you're ready to use your mock javamail:

 1 public class SomeImapClient {

2
3
private static Logger log = Logger.getLogger(SomeImapClient.class);
4
5
public void processMail() {
6 try {
7 Session session = getMailSession();
8 Store store = connect(session);
9 Folder folder = openMailFolder(store);
10 findContent(folder);
11 } catch (MessagingException e) {
12 throw new RuntimeException(e);
13 } catch (IOException e) {
14 throw new RuntimeException(e);
15 }
16
17
}
18
19
public Session getMailSession() {
20 Properties props = System.getProperties();
21 props.setProperty(“mail.store.protocol”, “imaps”);
22 props.setProperty(“mail.imap.partialfetch”, “0”);
23
24
log.debug(“Getting session”);
25 return Session.getDefaultInstance(props, null);
26
27
}
28
29
public Store connect(Session session) throws MessagingException {
30 log.debug(“getting the session for accessing email.”);
31 Store store = session.getStore(“imap”);
32
33
store.connect(“mockserver.com”, “testuser”, “somepassword”);
34 log.debug(“Connection established with IMAP server.”);
35 return store;
36 }
37
38
public Folder openMailFolder(Store store) throws MessagingException {
39 Folder folder = store.getDefaultFolder();
40 folder = folder.getFolder(“inbox”);
41 folder.open(Folder.READ_ONLY);
42 return folder;
43 }
44
45
public void findContent(Folder folder) throws MessagingException, IOException {
46 for (Message m : folder.getMessages()) {
47 log.debug(m.getSubject());
48
49
}
50
51
52
}
53
54
55
}


Output is:

DEBUG 30 Aug 2011 19:25:25 SomeImapClient:32 – Getting session
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:38 – getting the session for accessing email.
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:42 – Connection established with IMAP server.
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:55 – Some Subject


Some small gotchas:

  • your store.connect(“example.com”, “username”, “anything”) must match your “username@example.com” addressee in msg.setRecipients; different addressees will yield multiple mailboxes.
  • Only works for plain imap & pop3 mailboxes. Even though it doesn't make a lot of sense to use SSL on a mock mailbox, if you try getting the “imaps” store you'll get an error as a real implementation of javamail will be employed.
  • I'd always advise to use such tearDown() when doing unit tests, since the Session seems global and different tests could interfere with different contents – unless you use a different fake user/test server url in each of your tests.



#Ollivander #java #javamail #smtp

Apple has announced it will contribute to the OpenJDK project, so there's probably no risk that MacOS-specific pieces of the current JDK will be lost. Probably the next, Oracle-backed JDK will be just fine on Mac and will probably be released on time.

There're two things I don't understand:

  •  why such a bad timing? Apple could have deprecated his Java and announced OpenJDK participation at the same time – and that would have been good news. I don't understand if Apple had already chosen to contribute to OpenJDK or it had been pushed in that direction once it saw what happened in the dev community after their deprecation.
  • It's still not clear why I can't use Java in a software I'd like to distribute within the Mac App Store. If it's Oracle backed it's not a “deprecated technology” anymore, right? The language restriction was lifted even in iOS apps, why should it come back on Mac apps?

#Ollivander #apple #java

I'm not an hardcore Apple user. My first Apple product ever was a Macbook Pro I bought about one year ago.

But I fell in love with the Mac platform, and I think its really strong point is that it's versatile. It's got plenty of useful proprietary apps that sometimes I want to use or I am forced to use, like Microsoft Office or Adobe Photoshop, a pretty decent Adobe Flash implementation, but since it's a BSD derivative it's got a lot of nifty tools I'm used to and that I really like.

Sure, Windows might be faster at many tasks, especially if you're gonna spend as much money as you need to invest in Apple hardware, but my feelings towards Windows are mixed. I liked XP at the beginning, but it grew old and tired; Vista was a mess; Seven seems fine, but Microsoft still needs to conquer back my trust. Also, Windows is fine as a development platform as long as you use Microsoft tools – which are quite good – but it can get squeaky as soon as you adventure into the FLOSS land – I can remember some madness I incurred in while trying to use git under Windows.

What about Linux? I's a great development environment. I simply love my Ubuntu box, it's exactly as I want it. Eclipse, Netbeans, everything is set up as I want, all the key bindings, all the details. But my Ubuntu box is a tower form factor workstation with ethernet connectivity and two monitors – I think Ubuntu fails pretty hard at the common desktop experience, especially on a notebook – wireless connectivity is often an issue, suspend and hibernate are unreliable, the battery doesn't last as long as I'd like, and Flash on full screen is flaky. I don't think it's Canonical's own fault – it's really hard to have an OS work on each and every hardware and giving it away from free – but from the user's point of view this is a failure.  Some key apps are also missing – OpenOffice might not be enough a replacement for Office, and GIMP is not quite as effective as Adobe Photoshop.

But the Mac has it all, and that means it's got a lot of appeal on developers – we usually want the eye candy and the dark console, we like our computer to 'just work' and yet we like to be able to tune every detail. Whenever I go to a conference I can see there're far more Macs, iPads, iPhones than in any other typical business environment. A lot of developers from many communities simply love using Macs for their development work - I simply don't think that developers don't count in Mac market as someone stated. Maybe many of them are not Java developers, but as soon as they're unable to install Eclipse they'll complain as well.

Apple's Java sports its own Cocoa bindings, which are very useful to achieve a native look-and-feel, OS integration and general responsiveness for all java based software. Eclipse is java based and it's a widely used platform, RubyMine is one of the great favourites for Ruby coders, Netbeans is evolving at a great pace and supports Java, Ruby, Python, PHP and many others. Not to mention the fact that while I write those lines it's the only JDK available and working out of the box on Snow Leopard.

Now this is going to change. Java is deprecated on Snow Leopard, won't be included in Lion and can't be used for Mac App Store.

If Mac OS X were to be left with a third-party Java implementation – and let's hope SoyLatte can cope with such burden – we'll probably lose native OS integration, everything that's Swing or SWT based will need X11 to run, Swing will probably use its native Metal widget set while SWT will probably rely on GTK+ bindings; Apple doesn't even seem to be willing to contribute its own patches to the OpenJDK project, even though it would be meaningful if they just didn't want to waste resources on what they think it's a dead technology.

It's not a great scenario, but the real question here is: why?


Why should Apple have done all that? Is it that difficult to update the current JDK to work with  Lion? JDK7 might be a different topic, but hey – they're basically dropping what they've already done! And they're not letting anyone to continue their job easily. I think Apple is perfectly aware that the Mac OS X ecosystem is fairly different from the walled garden of IOS –  Windows is still got a much larger market, and limiting what Mac OS X can do will just feed Microsoft's pockets again.

I don't think there will be an official statement on that, but I bet Oracle has a part on this. Mac JDK seems to have been built by Apple but to have sprung from Sun's own JDK – is Oracle not willing to let this deal to go on? Maybe Apple needed to renegotiate its license and it turned out to be too expensive? Even though Apple may think Java is dead, it has no apparent reason to silently stab it again.

Their decision is causing a lot of noise in the Mac/Java community, let's see if Steve comments on this as well

#Ollivander #apple #java