alan

Alan's musings

Sometimes when performing your daily apt-get update, you'll experience messages like

E: The package lists or status file could not be parsed or opened.

or

The package index files are corrupted

Such messages usually don't go away for quite a long time, and may prevent legitimate updates from installing.

This usually happens even more frequently if using a proxy to access apt archives.

While I don't know what's the root problem, I've got a quick workaround that seems to fix the immediate error:

sudo -s -H bash -c 'apt-get clean && rm -f /var/lib/apt/lists/* ; rm -f /var/lib/apt/lists/partial/* && apt-get clean && apt-get update'

#MostlyUnixish #linux #ubuntu #debian #deb #apt

I find it extremely handy to be able to recall my windows by just using the keyboard.

It's pretty easy to do in Linux, just use:

wmctrl -x -R <WM_CLASS>

you can find the WM_CLASS of any window using this command:

xprop | grep WM_CLASS

then clicking on the desired Window. There may be more than one match, just choose one.

Example:

wmctrl -x -R "Gnome-terminal"

When you execute such command, the windows matching such class will be gathered on the current workspace and focused; you can assign global hotkeys in Gnome with System -> Preferences -> Keyboard -> Shortcuts. The Windows key (Super) is usually a great candidate for such global hotkeys.

#MostlyUnixish #xwindows #productivity

This is a spinoff from my Europython 2012 talk (video, slides).

Some days ago I got involved in a brief discussion on the topic, so here's what I think about Python packaging and dependency management: it's in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.

Why? Let's see.

The Packaging Conspiracy

Aim

No, it's not a conspiracy to force you to learn something you don't care about. The whole point of packaging is to make code reuse easy enough, both for the developer's own sake (e.g. shared code between different projects) and for sharing it with the public.

Packaging usually involves dependency management; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don't just release your code in “in the dark”: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.

It's an overkill!

Some developers think all of this is a waste of their time.

D: Hey Alan, are you really saying say I can't just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?

A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you've fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, you're doing by hand the work that a good dependency management tool is supposed to do. If you like that.

D: OK, ok. I got it. So now I've added my dependencies. But what the heck, does the version really matter? I don't care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won't break my code.

A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it's time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.

D: OK, ok, now I've set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?

A: Try doing something meaningful, and stick to your plan. You can pick your own versioning policy. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something “emotional”. If you perform small incremental changes on your codebase – a-la-Chrome – you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.

D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!

A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.

D: YEAH! I've just released version 0.0.2 to the public. Now I'm deleting 0.0.1, right?

A: No. Never. Ideally, unless you must – by must I mean something like “there's a password for my bank account in the source code” or “I mistakenly copypasted code for which I hadn't the copyright” – you should never, ever, ever, remove a released version. You never know who's using it – if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn't care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as “deprecated”, “bugged”, or maybe “donut” if they contain a huge security hole.

D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it's my first daughter and she was born in the very day of such release.

A: I'm happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, “lexicographical” is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means “lower”, so that 2.0.0beta is < 2.0.0, but that's not always the case – a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.'' is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.

D: There're a lot of things to remember about dependency management and packaging. Should I learn everything by heart?

A: What the heck, no. Your packaging tool should guide you in such practices and prevent you from going stray unless it's what you really mean to do! That why I'm complaining in the first place about the way Python does packaging!

In part 2 I dig into Python-specific packaging issues.

#Ollivander #python #packaging

I don't like php, and I've got my reasons. I've met a lot of people who dislike PHP and don't see how possibly anyone, anyhow, could think about ever using PHP in any real-world project out there, and simply dismiss PHP as a crap being created and used from and for incompetent stupid people around the world.

On the other side there're a bunch of PHP supporters which show you how wonderful is PHP, how many people use it and how widespread it is, and whoever says the contrary is just an enterprisey fool which spends nights at getting things done in Java or C#.

I disagree with both groups (of course!) and I think that PHP's got its unique set of killer features that make it still valuable, and those should be acknowledged.

First, it's not PHP by itself – there's the [LAMP stack](http://en.wikipedia.org/wiki/LAMP_(software_bundle%29), a de-facto pseudo-standard; interoperability of PHP applications on many different hostings work quite well. This is very different from most other platforms, where you must either adapt your development environment to what your hosting provides, like in Google App Engine, or you must find a way to tune your hosting environment to match your required dependencies. Not even Java, which heavily relies on standardized APIs, was able to reach a similar consensus on a standard stack.

And this sort of Platform-as-a-Service offering is widespread. This is catching on lately for other languages with things like Google App Engine, Jelastic, Engineyard, Heroku (while this last one is slightly more complex), but on PHP there's simply a incredible amount of really cheap and simple hosting. If you're willing to spend something like 50 EUR an year you can find out really good PHP hostings just everywhere, and you can expect to go down to 15 EUR per year and still have something that's pretty decent.

What's all this hosting good for? Do you create your own custom software in PHP and run on top of that? Of course not! Many experienced PHP developers go with their own infrastracture in order to have full control and tuning; the real bulk of PHP installations (IMHO – I don't have collected actual data on such topic, maybe one day I will) consists of existing free software installations. Existing software written in PHP makes the difference. There're a lot of people that simply don't care about PHP or programming at all, and yet they use PHP every day thanks to existing open source software.

Do you want a blog engine that you can customize and offers a variety of plugins that have nothing to do with a blog? Here's Wordpress. Would you like to manage a community? Drupal is your friend.

Please note that, more often than not, the available PHP software is not the “best of the breed”. Other software exists that is better, or faster, or offers more functionality, but PHP is usually “good enough”, allows “quick and dirty” customization, and it's free.

And doesn't require to be a developer to be installed – it's targeted at the slightly experienced user. I think that installation wizards in PHP are really well constructed (although they often require full manual intervention and it would be hard to automate them via command-line scripts or similar) and I sometimes miss them in other languages (like Ruby, Python, Java), where heavy command-line and config editing is required, many times in non-obvious way. Most PHP applications contain the application AND a web-based configuration tool for such application

  • that's an obvious thumbs up.

The “install your own web software” approach is getting less useful nowadays, with a lot of great services around that let you do almost anything you'd do with a php software, yet it was very common until mid 2000s to setup your own hosting and your own software instance, and it retains a lot of charm on users that want to feel empowered, that want to “own” their website and make modifications to it (and maybe they want to say they're PHP programmers because what they did) – php can be edited and new pages appear live on the server!

Then, don't forget there're many PHP developers around the world. It's not an exotic language – it's quite mainstream, if you need a custom feature you won't go in mancraft-starvation mode. Developer quality in such world may be a problem, but if you can find somebody who's not a PHP-only developer, you should feel safe.

And remember: most software for PHP is content-centered. There's very little business logic in it. PHP works well as long as your application only does only CRUD. Advanced usages, anything that would require a sort of application server or persistent, long-running process, should be avoided.

So, PHP is simply great for its target. If you want to have your own website, with your own domain, where you can do whatever you like, you don't care for quality, uptime, or security, PHP is simply the right tool for you.

You can even use it for learning some programming basics, but PHP is gotcha-intensive, and you aren't a real PHP programmer if you don't know at least 75% of such gotchas :–). Good luck!

#Ollivander #php #programming #web

  • UPDATE: modern ssh-copy-id often support a -p parameter for the port. So this trick is unnecessary and won't probably work.

ssh-copy-id is a cool script which is usually distributed along openssh client nowadays; it performs an automatic install of your local public SSH key to a remote server. Its syntax is simply:

ssh-copy-id [-i [identity_file]] [user@]machine

You may have spotted there's no builtin way of telling it what's the port of the remote server; a workaround exists which leverages the way arguments are passed to the script (it's a simple bash file, take a look at it. You can find various versions around the internet).

Just do this:

ssh-copy-id '-p [port_number] [-i [identity_file]] [user@]machine'

And enjoy your newly-added SSH key!

#MostlyUnixish #ssh

If you develop enough software, most probably you'll have a test/development network at your service.

In such networks, which are usually handled through a virtualization infrastructure, machines come and go very quickly. But ssh clients are usually unhappy about that:

alan@melquiades:/etc/ssh$ ssh root@192.168.0.208
The authenticity of host '192.168.0.208 (192.168.0.208)' can't be established.
RSA key fingerprint is 31:4d:8b:97:c8:57:04:85:6a:1b:72:54:46:ab:04:bc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.208' (RSA) to the list of known hosts.
root@192.168.0.208's password: 

The first time you connect, you're asked if the fingerprint is correct. The second time you'll just be allowed to connect:

alan@melquiades:/etc/ssh$ ssh root@192.168.0.208
root@192.168.0.208's password: 

What then if the machine at such ip address changes, because it's rebuilt or modified and the ssh host key is not retained?

alan@melquiades:/etc/ssh$ ssh root@192.168.0.208
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
7b:42:03:5f:5c:8d:5a:2b:6a:a4:1e:d6:0d:05:de:38.
Please contact your system administrator.
Add correct host key in /Users/alan/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/alan/.ssh/known_hosts:111
  remove with: ssh-keygen -f "/Users/alan/.ssh/known_hosts" -R 192.168.0.208
RSA host key for 192.168.0.208 has changed and you have requested strict checking.
Host key verification failed.

From the point of view of openssh this may be a serious security breach, and you should fear such message if it happens in the wild if you haven't touched a server - a MitM attack is probably going on!

What about your local network? ssh-keygen -R 192.168.0.208 will solve that, but you probably didn't need host key authentication since the beginning.

So, just add this:

Host 192.168.0.*
    CheckHostIP no
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no

at the beginning of your /etc/ssh/ssh_config file, and feel good.

Now, anytime you try connecting inside your network you'll be greeted by this:

alan@melquiades:/etc/ssh$ ssh root@192.168.0.208
Warning: Permanently added '192.168.0.208' (RSA) to the list of known hosts.
root@192.168.0.208's password: 

No confirmations whatsoever.

#MostlyUnixish #linux #ssh

Whenever you use Thunderbird on Linux you may encounter one annoying behaviour.

If you open a message which is got an attachment – you just need to open the message, not to save the attachment! – Thunderbird will place such attachment in our temporary directory, usually /tmp. You find the file there, and you think you used the Save as function to place it there, so you open it and you edit it.

Then you close Thunderbird, and poof! It clears its temporary files and deletes your modified attachment, possibily containing a day's work.

The workaround is simple: just use a launcher that changes Thunderbird TMP directory and use that to start Thunderbird!

#!/bin/bash -e
mkdir -p /tmp/thunderbird
TMP=/tmp/thunderbird /usr/bin/thunderbird "$@"

You may need to adjust the original binary path, then you can put such script in your path before your /usr/bin directory and you're done.

#MostlyUnixish #linux #thunderbird #temporary

Bash builtin disown is a not-so-well known [job control command] http://www.gnu.org/software/bash/manual/html_node/Job-Control-Builtins.html), and yet it's an extremely useful one.

So what? You've started a remote ssh session, launched some command which is taking very, very long to complete, and you just want to go home, or maybe you feel your connectivity may drop, and so on... and you forgot to launch screen (possibly with [byobu] https://launchpad.net/byobu)) or tmux before starting your work!

Now you're desperate and you'd just like to go home, but you fear your longstanding jobs will quit if you close your terminal.

Just background your job, launch disown and all the jobs in background will receive no notification whatsoever of your saying goodbye to them; of course you won't be able to read their output to the terminal.

#MostlyUnixish #linux #bash #ssh

So, here we come to our second part; if you did miss the first, it's there. All of the following discussion is relative to Sun Java 6, but most details – if not every detail – is still valid for Java 7 and 8.

Let's see the way Java handles unicode issues and why it's usually less problematic than Python – or, at least, while it looks less problematic than Python.

String literals in Java, just like in Python 3, are just Unicode objects; so you don't need to think whether it's a “binary string” or a “unicode object”; if it's a binary blob, you'll probably get a byte[], which should result in a 'here be dragons' behaviour about encoding.

Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?

System.out, which is an instance of java.io.PrintStream, internally uses a java.io.OutputStreamWriter which gets initialized using [Charset.defaultCharset()](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset(\)), and such value will be used for conversions between raw bytes and strings:

Encoding.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.nio.charset.Charset;

public class Encoding {
    public static void main(String[] args) throws Exception {
        String myString = "àèìòù";
        System.out.println(System.out.getClass());
        System.out.println(Charset.defaultCharset());
        System.out.println(myString);
        System.out.write(myString.getBytes(Charset.defaultCharset()));
        System.out.println("");
    }
}

result:

class java.io.PrintStream
UTF-8
àèìòù

The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that's discouraged as it can be unpredictable and unreliable. On the contrary, you should always use constructors and methods that accept a charset parameter if you're converting to/from raw bytes, just like String.getBytes(Charset charset) or [OutputStreamWriter(OutputStream outputStream, Charset charset)](http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter\(java.io.OutputStream, java.nio.charset.Charset))

So far, so good. But what happens if the outputstream is unable to handle the chars you'd like it to write?

CantEncode.java

package eu.franzoni.examples.pythonvsjavaencoding;

import java.io.FileOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class CantEncode {
    public static void main(String[] args) throws Exception {
        // change default output stream
        OutputStream outputStream = new FileOutputStream("eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output");
        PrintStream printStream = new PrintStream(outputStream, true, "ASCII");
        System.setOut(printStream);

        String myString = "àèìòù";

        System.out.println(System.out.getClass());
        System.out.println(myString);
     }
}

result:

class java.io.PrintStream
?????

ASCII encoding cannot possibly output the accented chars, so they're replaced with question marks instead of throwing the dreaded UnicodeEncodeError.

Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate 'heads up!'. Otherwise if you're just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.

But there's another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you'll use javac -encoding flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; if you don't pass such option, the platform default converter is used

What's the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.

Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That's what happens, in fact:

echo "utf8 compiled:"
javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding
echo "windows-1252 compiled:"
javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java && java eu.franzoni.examples.pythonvsjavaencoding.Encoding

result:

utf8 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù
windows-1252 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù

As you can see there, it's not that the JVM encoding has any issue at runtime; it's the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There's no way around this!

What I can recommend you is to ALWAYS set the project.build.sourceEncoding property when using Maven to compile, and to ALWAYS set the encoding attribute whenever you're using Ant. This will save you a lot, lot, lot of headaches.

#Ollivander #java #python #unicode #encodings

Whenever I install a production system/server I tend to enable full automatic updating, because it's usually better that something breaks because of a failed update rather than leaving a production machine unpatched; and if you pick a rather stable distribution (Centos or Ubuntu LTS are good examples) chances that you'll end up with a broken system are fairly low.

Btw let's get back to the point: in such cases you may end up with a disk which was filled up with obsolete kernels. The following snippets allows you to get all outdated kernels which are not in use, so you can automagically pipe its output to an apt-get remove -y in a cron job.

That may sound trivial, but what happens if you're not careful (e.g. if you try deleting all kernels but the latest) is that you might try removing the currently running kernel – and it's pretty bad if that happens in an unattended cron job.

#!/usr/bin/env python

# print all installed kernel/headers/etc BUT the current and the latest;
# suitable to be used with apt-get remove.
# requires aptitude to be installed. Works on Ubuntu and probably Debian

from subprocess import check_output
import re

def runcmd(s):
    return check_output(s.split())

# should work for anything until kernel 3.100
kernel_version_pattern = re.compile("[23]\.\d{1,2}\.\d{1,2}-\d{1,3}")
def get_kernel_version(kernelstring):
    return kernel_version_pattern.search(kernelstring).group(0)

def get_providing_packages(pkg):
    return filter(lambda x: x != "",
        map(str.strip,
            runcmd('aptitude search ~i~P%s -F%%p' % pkg).split("\n")
            )
        )

def get_nonmatching_packages(package_list, excluding):
    return [k for k in package_list if not get_kernel_version(k) in
        excluding]

all_kernels = get_providing_packages("linux-image")
all_kernels.sort() # lexicographical

kernel_versions = map(get_kernel_version, all_kernels)
latest_version = kernel_versions[-1]
current_version = get_kernel_version(runcmd("uname -r"))

kernels_to_remove = get_nonmatching_packages(all_kernels, (current_version,
    latest_version))

impl_headers = get_providing_packages("linux-headers")

headers_to_remove = get_nonmatching_packages(impl_headers, (current_version,
    latest_version))

# This seems to be needed in Ubuntu >= 13.04
#base_headers_to_remove = [header.replace("-generic", "") for header in headers_to_remove]
#headers_to_remove += base_headers_to_remove

print " ".join(kernels_to_remove + headers_to_remove)

Suggested use in a cron job, of course, is just:

apt-get remove --purge -y `ubuntu_get_obsolete_kernel_and_headers_pkgs.py`

WARNING: recent changes to the way Ubuntu marks dependencies and provided names for kernel headers require that you uncomment a couple of lines in the script towards the end; just remove the # at the beginning of the lines 46 and 47.

UPDATE: in recent Ubuntu versions, just invoking apt-get autoremove --purge will do the trick. Such command will remove all automatically installed packages that are no longer required in the running system. It can remove other packages beyond outdated kernels, by the way.

#MostlyUnixish #linux #ubuntu #debian #update #maintenance