alan

Ollivander

So, here we come to our second part; if you did miss the first, it's there. All of the following discussion is relative to Sun Java 6, but most details – if not every detail – is still valid for Java 7 and 8.

Let's see the way Java handles unicode issues and why it's usually less problematic than Python – or, at least, while it looks less problematic than Python.

String literals in Java, just like in Python 3, are just Unicode objects; so you don't need to think whether it's a “binary string” or a “unicode object”; if it's a binary blob, you'll probably get a byte[], which should result in a 'here be dragons' behaviour about encoding.

Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?

System.out, which is an instance of java.io.PrintStream, internally uses a java.io.OutputStreamWriter which gets initialized using [Charset.defaultCharset()](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset(\)), and such value will be used for conversions between raw bytes and strings:

Encoding.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.nio.charset.Charset;

public class Encoding {
    public static void main(String[] args) throws Exception {
        String myString = "àèìòù";
        System.out.println(System.out.getClass());
        System.out.println(Charset.defaultCharset());
        System.out.println(myString);
        System.out.write(myString.getBytes(Charset.defaultCharset()));
        System.out.println("");
    }
}

result:

class java.io.PrintStream
UTF-8
àèìòù

The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that's discouraged as it can be unpredictable and unreliable. On the contrary, you should always use constructors and methods that accept a charset parameter if you're converting to/from raw bytes, just like String.getBytes(Charset charset) or [OutputStreamWriter(OutputStream outputStream, Charset charset)](http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter\(java.io.OutputStream, java.nio.charset.Charset))

So far, so good. But what happens if the outputstream is unable to handle the chars you'd like it to write?

CantEncode.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.io.FileOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class CantEncode {
    public static void main(String[] args) throws Exception {
        // change default output stream
        OutputStream outputStream = new FileOutputStream("eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output");
        PrintStream printStream = new PrintStream(outputStream, true, "ASCII");
        System.setOut(printStream);

        String myString = "àèìòù";

        System.out.println(System.out.getClass());
        System.out.println(myString);
     }
}

result:

class java.io.PrintStream
?????

ASCII encoding cannot possibly output the accented chars, so they're replaced with question marks instead of throwing the dreaded UnicodeEncodeError.

Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate 'heads up!'. Otherwise if you're just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.

But there's another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you'll use javac -encoding flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; if you don't pass such option, the platform default converter is used

What's the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.

Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That's what happens, in fact:

echo "utf8 compiled:"
javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding
echo "windows-1252 compiled:"
javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding

result:

utf8 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù
windows-1252 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù

As you can see there, it's not that the JVM encoding has any issue at runtime; it's the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There's no way around this!

What I can recommend you is to ALWAYS set the project.build.sourceEncoding property when using Maven to compile, and to ALWAYS set the encoding attribute whenever you're using Ant. This will save you a lot, lot, lot of headaches.

#Ollivander #java #python #unicode #encodings

Before going on with this post, be sure you've read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) - otherwise, I won't be able to solve your issues.

Sometimes Java fans just say that Python Unicode support sucks. While there's a bit of truth in that assertion – for Python 2.x at least, since 3.x solved the problem at its root – the real problem lies in the fact that many programmers don't know what's going on “under the hood”, and that python default behaviour is a bit unforgiving.

First things first: Python 2.x has got two distinct string types: the so-called “byte strings” (str type) are one, and unicode objects (unicode type) are another one. Unicode objects are much like Java strings; they're an internal abstraction of Python, and need to be interpreted to/from byte strings whenever printing, reading/writing from/to a file, etc.

Sometimes such interpreting “just works” (most probably if you're dealing with ascii-only text), but if such conversion fails, errors occur: Python infamous and dreaded UnicodeDecodeError can happen in many places and might sometime just puzzle the programmer:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù” + u”asd”

javapythonunicode$ python unicode_concat.py
current default encoding: ascii
Traceback (most recent call last):
File “unicode_concat.py”, line 6, in <module>
“àèìòù” + u”asd”
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

 This happens when trying to concatenate an unicode string to string object. What really happens here, behind the scenes, is explained in unicode_concat_explained.py:

# -– coding: utf-8 --
import sys

print “current default encoding: “ + sys.getdefaultencoding()

“àèìòù”.decode(sys.getdefaultencoding()) + u”asd”

When using a byte string (not a unicode object) python tries to convert that string to unicode before concatenating to a unicode object. Since an encoding wasn't specified, python just uses the default.

So, if you're planning to mix unicode and bytestring objects (which is not a good idea), always remember to explicitly convert string objects to unicode by their decode() method. Or, if you're sure the encoding is always the same, you could add a sitecustomize.py to your interpreter or your project PYTHONPATH, and set such encoding. Beware that setting it system wide may lead to unexpected results when running your project on another machine.

Also, just remember that the coding directive at the begininning of the file does not change unicode encoding/decoding behaviour: that's a directive to the parser, which is used to instruct it about the current encoding of the file. That information is not retained nor used at runtime.


Let's go for another much dreaded error now: UnicodeEncodeError!

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

print u”àèìòù”

results in this output:

javapythonunicode$ python unicode_print.py
current stdout encoding: UTF-8
àèìòù

Everything seems to work (as long as your terminal supports a charset, like UTF-8, or iso-8859-1, which can display accented chars) ! But now let's go for some “black magic”:


That may be puzzling, but the sys.stdout printout should give you an hint.

Whenever output is to a terminal, Python does perform autodetection of the encoding of your terminal, and subsequently sets sys.stdout accordingly. When sys.stdout encoding is set, any unicode object that gets printed is encoded accordingly. When output is NOT to a terminal, python does not autodetect sys.stdout encoding and, when sys.stdout has no encoding set any unicode object that is printed is converted according to sys.getdefaultencoding()

So, what really happens here is:

# -– coding: utf-8 --
import sys
print “current stdout encoding: “ + str(sys.stdout.encoding)
print “current default encoding: “ + sys.getdefaultencoding()

sys.stdout.write(
u”àèìòù”.encode(sys.stdout.encoding or sys.getdefaultencoding())
)

Since accented characters can be encoded to UTF-8 but not to ASCII, such UnicodeEncodeErrors arise.

Also beware of bug 4947 - it may hit you if you're using Python 2.6 or or older.

You can find the second part of this article here

#Ollivander #python #java #unicode #encodings

UPDATE:
Most of this is pointless on IDEA >= 11 and PyCharm >= 2, since a Tools -> Create desktop entry menu now exists and will let you do everything my launcher did. You'll need to reinstall it at every update – or just use the symlink trick I reccomended with the standalone launcher and edit the IDEA desktop entry.

I had already elaborated on this, and I still thank Dutchworks for their work, but I was yet to be satisfied with the status of dock awareness for Jetbrains IDEs; I tend to use multiple of those and I was highly dissatisfied with the way the dock grouped all my Java apps – that's caused by the fact that many Java apps based on Swing/AWT share the same WM_CLASS attribute, which is used bye the dock to tell one window from another – but you can read more on that from the Dutchworks blog page above.

The basic problem is:


Here I have launched IDEA and PyCharm at the same time, and they're grouped with the very same icon by docky – you need to right-click the icon and pick the proper subwindow, which is quite an hassle.

My previously created solution required to update the launcher and copy agent.jar at every update.

So what?

I just created a standalone launcher which creates a proper .desktop file in your .local/share/applications directory, does not modify original files and still allows IDEs to have their own distinct icon in the dock:



There I have both IDEA and PyCharm active at the same time, and they both work properly, and, whenever updating, you can just update the symlink (as suggested in the README from the GitHub repo) and you're done!

Multiple projects in the same DIE still share the same icon, but that doesn't seem to be changeable from the outside, it's something that Jetbrains should do; I'll open a ticket for that.

Enjoy!


#Ollivander #jetbrains #ide #launcher

Heroku Java support is something I longed for, but they apparently don't support deploying a plain, old war file to Heroku.

I don't like that; I've got my own continuous integration setup, and I want to compile my files on my own box – this allows me to keep my own Maven proxy with my own dependencies – which sometimes are patched versions of public ones –  so that I don't need to push everything on a public repo. I don't want to be forced to build everything on Heroku, and I don't want to compile anything I push – I'd usually push to Heroku stable versions only, or maybe integration versions that pass unit, integration and acceptance tests.

So what? Don't worry, deployment is possible.



Setup an heroku-aware git repository and a Cedar stack app, and add the files you see above; your dir structure should look like:

pom.xml
Procfile
src/
     main/
             java/
                    root/
                           StartWebApp.java
             resources/
                             mywebapp.war


Now a simple

git push heroku master


Should do the trick, and let you see your fully deployed war app.


There're still things you need to be aware of; file persistence may not work as expected, since filesystem is ephemeral, and I still need to figure out things related to database configuration, but those should not come at a great expense.

I'll post updates in the future as soon as I get them right.



#Ollivander #java #heroku #war

Small primer for mock javamail, a pretty useful project that will be handy whenever you need to test code that uses POP3/IMAP/SMTP and you wouldn't like to build wrappers for anything you need to test – javamail does not really offer interfaces that can be implemented by mock objects, but different implementations can be used at runtime depending on which jars are on the classpath.

Take a look at the project homepage:

http://java.net/projects/mock-javamail

Be sure you've enabled java.net Maven repo, and add this dependency to your pom.xml:

1  <dependency>

2 <groupId>org.jvnet.mock-javamail</groupId>
3 <artifactId>mock-javamail</artifactId>
4 <version>1.9</version>
5 <scope>test</scope>
6 </dependency>

You're almost ready; now setup your test code this way:

 1 public class IMAPMailRepositoryTest {

2 @Before
3 public void setUp() throws Exception {
4 final Session session = Session.getInstance(
5 System.getProperties());
6
7
MimeMessage msg = new MimeMessage(
8 session);
9 msg.setRecipients(Message.RecipientType.TO,
10 “testuser@mockserver.com“);
11 msg.setSubject(“Some Subject”);
12 msg.setText(“sometext”);
13 Transport.send(msg);
14 }
15
16
@After
17 public void tearDown() throws Exception {
18 Mailbox.clearAll();
19 }



Now you're ready to use your mock javamail:

 1 public class SomeImapClient {

2
3
private static Logger log = Logger.getLogger(SomeImapClient.class);
4
5
public void processMail() {
6 try {
7 Session session = getMailSession();
8 Store store = connect(session);
9 Folder folder = openMailFolder(store);
10 findContent(folder);
11 } catch (MessagingException e) {
12 throw new RuntimeException(e);
13 } catch (IOException e) {
14 throw new RuntimeException(e);
15 }
16
17
}
18
19
public Session getMailSession() {
20 Properties props = System.getProperties();
21 props.setProperty(“mail.store.protocol”, “imaps”);
22 props.setProperty(“mail.imap.partialfetch”, “0”);
23
24
log.debug(“Getting session”);
25 return Session.getDefaultInstance(props, null);
26
27
}
28
29
public Store connect(Session session) throws MessagingException {
30 log.debug(“getting the session for accessing email.”);
31 Store store = session.getStore(“imap”);
32
33
store.connect(“mockserver.com”, “testuser”, “somepassword”);
34 log.debug(“Connection established with IMAP server.”);
35 return store;
36 }
37
38
public Folder openMailFolder(Store store) throws MessagingException {
39 Folder folder = store.getDefaultFolder();
40 folder = folder.getFolder(“inbox”);
41 folder.open(Folder.READ_ONLY);
42 return folder;
43 }
44
45
public void findContent(Folder folder) throws MessagingException, IOException {
46 for (Message m : folder.getMessages()) {
47 log.debug(m.getSubject());
48
49
}
50
51
52
}
53
54
55
}


Output is:

DEBUG 30 Aug 2011 19:25:25 SomeImapClient:32 – Getting session
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:38 – getting the session for accessing email.
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:42 – Connection established with IMAP server.
DEBUG 30 Aug 2011 19:25:25 SomeImapClient:55 – Some Subject


Some small gotchas:

  • your store.connect(“example.com”, “username”, “anything”) must match your “username@example.com” addressee in msg.setRecipients; different addressees will yield multiple mailboxes.
  • Only works for plain imap & pop3 mailboxes. Even though it doesn't make a lot of sense to use SSL on a mock mailbox, if you try getting the “imaps” store you'll get an error as a real implementation of javamail will be employed.
  • I'd always advise to use such tearDown() when doing unit tests, since the Session seems global and different tests could interfere with different contents – unless you use a different fake user/test server url in each of your tests.



#Ollivander #java #javamail #smtp

I had some hard time when testing Twisted protocols. Although they should be decoupled from factories, most examples I could find, including those from the official doc, were just too noisy and suggested using a factory.

I don't like that approach – I should not be forced to use a factory – that creates an unnecessary coupling – let's just pretend a factory existed and creates a connections, that's all our protocol should need.

Of course sometimes a protocol is bound to a very specific factory since it needs to invoke methods on it for non protocol-specific data (remember, there's just one factory, while there's one protocol per connection). I don't like that behaviour, since it creates an unnecessary coupling and makes factories huge and not cohesive, and I think protocols should just be injected whatever peers they need during the buildProtocol() invocation – but I'll talk about that in another post. In the meantime, I reccommend just creating a mock factory and setting it manually on the protocol.

So, here it is the solution I baked:



What do you think about this solution?

#Ollivander #python #twisted #testing

This is a multipart post. See part 3

Just a brief conclusion to my post sequence; the three principles for succeeding at community-driven projects are:


  • Make it easy for the people to give you their feedback – it's highly valuable.
  • Listen and reply to people's feedback. Don't do every-possible-requested-thing, but reply on time.
  • Don't take any policy as a dogma. Everything can be changed.

#Ollivander #community #OpenSource

I've found a good way to launch tests using Twisted Trial in a zc.buildout enabled project. I was in fear of needing a full-fledged recipe – which I didn't have the time nor the will to code.

Luckily, zc.recipe.egg seems good enough to satisfy my needs; I wanted a recipe that could launch all unit tests if no parameter was passed, while retaining the ability to choose which specific unit test was launched on command line.


Here it is:



That's filled in for pydenji

#Ollivander #python #twisted #ZcBuildout

UPDATED: IDEA entry.
UPDATED: Automatic agent add script.

I'm starting to like both PyCharm and RubyMine, two IDEs from IntelliJ, along with IDEA

I had some problems while they were both running on Linux using a dock – I like docky – since they're both Java apps and they got “merged” in a single icon, but then I found a solution - pull the agent into bin folder and just edit pycharm.sh, rubymine.sh and idea.sh this way, around the end of the file:




If you're lazy, or you're just tired of readding the agent every time your IDE gets updated, just download the agent and the adder script for IDEA, PyCharm or RubyMine to your bin directory and re-execute it every time your ide gets updated.

I was just missing a desktop file entry so they could show up in my menu; IntelliJ currently distributes their ides as .tar.gz archives, no .deb, no .rpm, so I just packed up two small entries and added them to ~/.local/share/applications:




Replace the PATH_TO_... and enjoy!


EDIT:
As a bonus, by just installing “wmctrl” you can easily setting global shortcuts for moving pycharm & rubymine IDEs on the current screen and putting them in the foreground:

wmctrl -x -R PyCharm
wmctrl -x -R RubyMine
          wmctrl -x -R IDEA

 In GNOME you can assign such commands as a global shortcuts via System -> Preferences -> Keyboard Shortcuts.

#Ollivander #ide #jetbrains

This is a multipart post. See part 2

6. State your mission
What's your project for? Who should use it? Which platform is designed for? 

Answer those questions somewhere in your project page. You won't risk people looking for the wrong solution coming to you and asking strange things, and you'll be able to dismiss requests as "out of scope" if they exceed your vision. Sharing such vision with your users is a necessity, or you'll be called a traitor by somebody, someday. As a user, I want to know what requirements was the project designed to accomodate. If I start building a large corporate webapp with tons of users around a small framework designed for personal, low-traffic blogs, I'm probably doomed to fail.

What's you release policy? How do you manage API changes? 
Explain how releases are done. Do you just keep a stable main branch and pull development and bugfix branches in it as soon as they're ready, so you just sometimes tag the main branch and have a release at that point? Or do you create milestone-oriented branches as soon as you release, e.g. version 1.0.0 is got a release-1.0 branch where you apply bugfixes only and release 1.0.1, 1.0.2 from that? Or have you got a "stable" release and a "development" release, but you sometimes pull stable features from development to stable?

All of that is important from the point of view of the user. I want to understand whether an API can change in an incompatible manner from 1.0.0 to 1.0.1. I want to know if I ever get a 1.0.1 release after 1.0.0 or I'll need to pick 1.1.0 with all its changes.

7. Have an homepage
And make it sure you provide its address in README files and such! You don't need a domain name, even though it makes things simpler. It doesn't need to be wonderful - a working one will suffice. You can use Google Sites, or any which Google Code, bitbucket or github provide out of the box.

This may seem a stupid proposition, but in the modern era many users have access to the code for a project without having ever seen the project homepage - central repositories like cheeseshop for Python and maven for Java make this possible.  Then, when it's time to report an issue or ask a question, the user is puzzled and doesn't know where he should head to.

Make sure the homepage contains all the info a user needs. Where does discussion take place, how should I file a ticket, where are docs, what's the latest release, etcetera - everything should be linked there. Sometimes projects use multiple servers to store their data - the repo is on bitbucket.org, the mailing list on sourceforge.net, and the bugtracker is self-hosted. Of course it's better to disable whatever it's not in use, but the point it is: the homepage should be the authoritative reference for everything regarding your project.

And keep it updated! That's why having a domain name of your own, or a subdomain - e.g. myproject.myowndomain.com - might be a good choice - you can just redirect people if your hosting goes down or you'd like to change something.

Having stale information on your homepage will just delay people trying to help and maybe prevent some of them from telling you something.  I really hate when the homepage says the latest stable version is "3.0.5" but the central repo offers "3.0.6" - WTF? If you don't have the time to keep that in sync, just point to a webpage of the central repo and don't duplicate releases on your homepage - nobody says you must.

Next: part 4


#Ollivander #community #OpenSource