alan

Alan's musings

I seem very ancientwisdom-oriented these days.

Sometimes you've got a system user on your linux server that's used to run a specific service; it's a good practice to employ those users to achieve privilege separation, since it limits the impact of many security breaches. It's security 101, really.

If you've got a system user, most probably its shell would be /bin/false or /bin/nologin. Again, this is a good practice, since it would be harder for an attacker to gain shell access to your machine through that user.

But I find that many people just change such shell, because sometimes they need to customize the environment for a user, and if you try doing something like

su - systemuser

you'll find yourself just back to your original shell, because su tries to invoke the shell which is set for the target user, and sudo doesn't generally help here, because its --shell option doesn't start a proper login shell for that user (most of the environment is that of the calling user).

Very well, gentlemen: this is a wrongdoing and a potential security hole. You should never enable a shell for your system users. Instead, you should leverage an option of su:

su - systemuser --shell /bin/bash

That lets you choose your favourite shell on a per-launch basis.

#MostlyUnixish #linux #ancientwisdom #su #security

This is another old trick, but I like to tell the world about such amenities.

I often find people writing command lines such as:

ssh arcane-system-user@someextremelylong.andimpossibiletotype.anderrorprone.hostname.com

or, worse

ssh idliketorememberwhichuser@10.20.30.40

or maybe it was

ssh someuserimashamedof@1.2.3.4

yes, sometimes ssh is simply tedious.

But you can do better. In fact, you can do much better.

your ~/.ssh/config or /etc/ssh/ssh_config can contain Host configuration sections; I have already blogged about such directives when I told the world about stopping key verification compliants – they're very flexible, indeed.

Consider adding Host directives in your configuration for your most used services:

Host myniceserver
    Hostname myniceserver.mylove.com
    Port 12345
    User teddybear

And then you can just

ssh myniceserver

You can override most options on a per-server basis and/or select a certain private key for connections.

#MostlyUnixish #linux #ssh #ancientwisdom

This is a very old trick, but I find many people doesn't know about it yet.

If you're using sudo (and you definitely should), one minor complaint is that you need to re-enter your password quite often.

One common pseudo-solution is to completely disable the password for executing sudo with such user; while it may be a good idea sometimes, it does decrease the overall security – if you happen to forget an open shell or screen/tmux lingering somewhere, you give total power to whoever gets in touch with it.

If you happen to let people into your sudo-enabled account with your ssh key, anybody that can get hold of your ssh key will get root access to your servers. It's not the best possible scenario.

There's a very simply workaround: just edit your sudoers file!

sudo visudo

and add a line like that (or modify it if it's already there)

Defaults    timestamp_timeout=120

This means that sudo will ask for a password every two hours instead of the default (15 minutes on most installations). You can use -1 in order to never let it expire (I don't suggest that – 8-10 hours should be enough for everybody.)

The only drawback is that such timeout is per shell, not system-wide – i.e. if you open multiple terminals, you'll have to enter the password again for each new terminal instance.

#MostlyUnixish #linux #sudo

It might be high time to update to the latest Ubuntu 14.04.

I think that the upgrade process that is provided by Canonical is slightly underoptimal: if you're using a local mirror which can provide a very fast download of all packages, they'll usually be disabled during the upgrade process, and an official mirror (possibile one very far away) will be employed.

This can lead to a very slow and painful upgrade process.

The solution is easy.

The following instructions regard moving from 12.04 (precise) to 14.04 (trusty)

Switch your sources

Assuming that your official ubuntu sources are configured from /etc/apt/sources.list just do something like:

sudo sed -i.old -e 's/precise/trusty/g' /etc/apt/sources.list

I'm not considering other sources in /etc/apt/sources.list.d, since they'll be disabled whatsoever; but if you happen to have ubuntu sources there as well, you should repeat the SEDding process for each file.

Update your index

sudo apt-get update

Download updated packages without installing

sudo apt-get dist-upgrade --download-only

Revert to old sources

sudo mv -f /etc/apt/sources.list.old /etc/apt/sources.list

Update index again

sudo apt-get update

Rejoice!

Now you can just follow the upgrade instructions and almost nothing will be downloaded during the upgrade, because everything is already on your hard drive.

#MostlyUnixish #ubuntu #linux #upgrade

A bit of wrap-up after part 1 and part 2

Current state of the things

I seem to have missed something: there's now a very active Python Packaging Authority which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.

Setuptools and distribute got merged and we won't experience a split-experience anymore, we'll just have One Tool To Rule Them All!

So, rejoice! Things are getting better!

In the meantime

I recommend using Pip . Always try using requirements.txt, it's a good way to keep your versions consistent and working.

Use virtualenv for development purposes, and always test your package in a virtualenv with a disabled site-packages! Otherwise you risk you're distributing something that works on your workstation only because of a site-wide package you have installed there!

In order to distribute a package, you'll need setuptools/distribute as well; just do a source distribution and a wheel distribution (you're required to install wheel beside setuptools); forget about eggs, especially for eggs containing binary extensions. Your command line should be something like:

python setup.py clean register sdist bdist_wheel upload

And let's keep our fingers crossed on a better state of things!

#Ollivander #python #packaging #DependencyManagement

UPDATED: 26th January 2015, with Tomcat 8 support.

Today I'm gonna share something that I figured out some times ago: how do I keep my tomcat installation up to date on my servers?

Of course I'm not willing to automatically switch majors, but I'd like to pull bugfix releases in as soon as possible; most of my servers should work unattended, and I prefer to handle it if something crashes rather than being hacked because of some public exploit.

Apache Tomcat makes it incredibly hard to do so for different reasons: first, there's no latest link for the latest release, you must first check the homepage to see what's the latest version; second, the archive you can download mingles the application itself with the configuration, log and webapp dir.

What should we do, then?

Tomcat allows us to configure almost everything. Let's use its power!

For this setup, I use Tomcat7 and I chose to employ /etc/tomcat7/conf as a configuration directory, /opt/tomcat7/latest as tomcat7 own directory, /opt/tomcat7/webapps for our webapps, /opt/tomcat7/logs, /opt/tomcat7/temp, /opt/tomcat7/work as logging, temp and work directory, respectively.

The same applies for tomcat8, it's just the prefix that will change.

To begin with, create a user (maybe a service user, I'll leave the details to you) tomcat with a main group tomcat as well, which we'll use to run the servlet container – you can pick a different user if you like, the class allows you to choose.

Now make sure your Puppet manifest includes the tomcat_versioned class with the proper parameters. In order to do so, you'll need this custom fact that helps the system determine the very latest tomcat7 version, place it in your custom facts directory:

#!/usr/bin/env python
# requires lxml
from lxml.etree import HTML
import re
import sys
import urllib

for version in (7,8):
    pattern = re.compile("^{0}\.0\.\d\d\d?$".format(version))
    root = HTML(urllib.urlopen("http://tomcat.apache.org/download-{0}0.cgi".format(version)).read())
    for e in root.iterdescendants():
        if isinstance(e.text, basestring) and pattern.match(e.text.strip()):
        print "tomcat{1}_latest_version={0}".format(e.text.strip(), version)
        break

and the tomcat_versioned class, of course; just pass the major of the tomcat you'd like to use, and it will find the latest tomcat update for such major, and install it in /opt/tomcatX:

# this will install latest tomcat from apache website, and yet retain your
# config from /etc/tomcat7/conf
# tested with puppet 3.7 and tomcat 7 and 8.

class tomcat_versioned($tomcat_major=8, $tomcat_user="tomcat") {
    # assumption: the user should exist, it won't be created
    # you should define a tomcatX service outside this class
    $tf = getvar("tomcat${tomcat_major}_latest_version")
    $tm = "tomcat${tomcat_major}"

    exec { "/bin/tar xvf apache-${tm}-archive.tar.gz":
    creates =>"/opt/${tm}/apache-${tm}-${tf}",
    cwd => "/opt/${tm}",
    refreshonly => true,
    alias => "${tm}_unpack",
    require => File["/opt/${tm}"]
}

file { "/opt/${tm}":
ensure => "directory",
mode => 0755,
owner => "root",
group => "root"
}

file { "/opt/${tm}/apache-${tm}-archive.tar.gz":
ensure => "present",
source => "/tmp/apache-tomcat-${tf}.tar.gz",
require => Exec["${tm}_download_latest"],
notify => Exec["${tm}_unpack"]
}

exec { "/usr/bin/wget --timestamping http://www.eu.apache.org/dist/tomcat/tomcat-${tomcat_major}/v${tf}/bin/apache-tomcat-${tf}.tar.gz":
alias => "${tm}_download_latest",
cwd => "/tmp"
}

exec { "/bin/ln -sf --no-target-directory apache-tomcat-${tf} latest":
refreshonly => true,
subscribe => Exec["${tm}_unpack"],
cwd => "/opt/${tm}",
alias => "${tm}_symlink"
}

exec { "/bin/rm -rf conf.orig && /bin/mv -f conf conf.orig && /bin/ln -sf --no-target-directory /etc/${tm}/conf conf":
refreshonly => true,
cwd => "/opt/${tm}/latest",
subscribe => Exec["${tm}_symlink"],
notify => Service["${tm}"],
alias => "${tm}_config_move"
}

# first-time only executions. I might like to abstract some logic if I were a bit less lazy than I am.
# in order to stay on the safe side, we never let the normal user to access our files; this may be relaxed,
# depending on your context.

# this contains our config. our servlet container should be able to read it, but never write it.
exec { "/bin/mkdir -p /etc/${tm} && /bin/cp -r /opt/${tm}/latest/conf.orig /etc/${tm}/conf && /bin/chmod 0750 /etc/${tm}/conf && /bin/chown -R root:${tomcat_user} /etc/${tm}/conf && /bin/chmod 0640 /etc/${tm}/conf/*":
creates => "/etc/${tm}/conf",
subscribe => Exec["${tm}_config_move"]
}

# this will contain the actual code of our webapps. Again, the container must be able to read them,
# never write to them.
exec { "/bin/mkdir -p -m 0750 /opt/${tm}/webapps && /bin/chown root:${tomcat_user} /opt/${tm}/webapps":
creates => "/opt/${tm}/webapps",
}

# those are working directories where the container must be able to write.
exec { "/bin/mkdir -p -m 0770 /opt/${tm}/work && /bin/chown root:${tomcat_user} /opt/${tm}/work":
creates => "/opt/${tm}/work",
}

exec { "/bin/mkdir -p -m 0770 /opt/${tm}/temp && /bin/chown root:${tomcat_user} /opt/${tm}/temp":
creates => "/opt/${tm}/temp",
}

exec { "/bin/mkdir -p -m 0770 /opt/${tm}/logs && /bin/chown root:${tomcat_user} /opt/${tm}/logs":
creates => "/opt/${tm}/logs",
}
}

The tomcat7 service is trivial and omitted, fill it in by yourself (or delete it).

Once this is done, you'll find a base configuration in /etc/tomcat7/conf. We must now edit server.xml there and tell Tomcat where we'd like to have our webapps, logs and work directories:

<!-- this should go inside <Server><Service><Engine> -->

<Host name="localhost"  appBase="/opt/tomcat7/webapps"
            unpackWARs="false" autoDeploy="true" workDir="/opt/tomcat7/work">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/opt/tomcat7/logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>

Last but not least, we should tell tomcat what is CATALINA_HOME and CATALINA_TMPDIR. This is an example script that can be used with upstart on Ubuntu 12.04, change it according to your OS:

description "tomcat7"

  start on runlevel [2345]
  stop on runlevel [!2345]
  respawn
  respawn limit 10 5

  # run as non privileged user
  # add user with this command:
  ## adduser --system --ingroup www-data --home /opt/apache-tomcat apache-tomcat
  # Ubuntu 12.04: (use 'exec sudo -u apache-tomcat' when using 10.04)
  setuid tomcat
  setgid tomcat

  # adapt paths:
  env JAVA_HOME=/usr/lib/jvm/java-7-oracle
  env CATALINA_HOME=/opt/tomcat7/latest
  env CATALINA_TMPDIR=/opt/tomcat7/temp
  env HOME=/home/tomcat

  # adapt java options to suit your needs:
  env JAVA_OPTS="-Djava.awt.headless=true"
  env CATALINA_OPTS="-Xmx1536M -server"

  exec $CATALINA_HOME/bin/catalina.sh run

  # cleanup temp directory after stop
  post-stop script
    rm -rf /opt/tomcat7/temp/*
  end script

In this situation I had a “normal home” for tomcat because I needed it, but your requirements may work differently.

I hope that's useful to somebody.

#MostlyUnixish #puppet #ConfigurationManagement #tomcat

Sometimes you would like to use an http URL as the source file for puppet; while it's being discussed since long, no implementation has been created so far.

Of course it would be better to periodically download the file from its http source to the puppet master, and then use puppet file server to send such file to our client; but that's impractical in some contexts, especially if you're using your puppet in headless mode, i.e. without a master. In such situations you may really want to source files from http, without redownloading them if they haven't changed.

I found a good way to do it is to use the excellent [wget] http://www.gnu.org/software/wget/) command line tool.

This is an example of what I use to keep my installation of Jenkins up to date:


class jenkins {
    exec { "/usr/bin/wget --timestamping http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war":
        alias => "jenkinslatest",
        cwd => "/tmp",
    }

    file { "/opt/tomcat7/latest/webapps/jenkins.war":
        ensure => present,
        source => "/tmp/jenkins.war",
        alias => "jenkinswar",
        require => Exec["jenkinslatest"] }

}

The timestamping flag will let wget check the remote server for file modification time; if file hasn't changed, it won't be redownloaded.

Just choose a proper directory for temporary downloads (tmp is volatile and gets sometimes cleaned up at runtime as well), make sure the server properly sets the Last-Modified header and remember to refresh your other resources when files change, and you're done.

#MostlyUnixish #puppet #ConfigurationManagement #http

Sometimes you need to write a quick shell script; many a time, you need to put it on a server, and launch it every now and then, or run it in a cron job. That script may end up doing something important and/or altering the state of the server machine.

If that script starts being using often enough, you'll want logging as well, and that can be a complex beast to tame, for logs require rotation and deletion, otherwise the beast will eat up all your disk space. Setting up logrotate for a single shell script looks like an overkill to me most of the times.

So what? Many a times logging will be neglected, and you'll lose the ability to track what happened and why.

This is my quick recipe for a basic bash logging solution with line-based truncation: the logfile will never exceed RETAIN_NUM_LINES lines (actually, it will never exceed such size when the script is launched, so the number of lines when the script has run is RETAIN_NUM_LINES + number of lines logged by the script).

Whenever you use the log function you'll get output on stdout as well, complete with a timestamp.

#!/bin/bash

# set LOGFILE to the full path of your desired logfile; make sure
# you have write permissions there. set RETAIN_NUM_LINES to the
# maximum number of lines that should be retained at the beginning
# of your program execution.
# execute 'logsetup' once at the beginning of your script, then
# use 'log' how many times you like.

LOGFILE=quicklog.sh.log
RETAIN_NUM_LINES=10

function logsetup {
    TMP=$(tail -n $RETAIN_NUM_LINES $LOGFILE 2>/dev/null) && echo "${TMP}" > $LOGFILE
    exec > >(tee -a $LOGFILE)
    exec 2>&1
}

function log {
    echo "[$(date --rfc-3339=seconds)]: $*"
}

logsetup

log hello this is a log

Launch this a lot of times and you'll end up with this in quicklog.sh.log:

[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:14+02:00]: hello this is a log

Enjoy!

UPDATED on September 26th, 2016 to employ the rfc-3339 date option, which makes the logline quite more readable and greppable.

#MostlyUnixish #bash #scripting #shell

Check out part 1 for my thoughts on what packaging and dependency management should be. I forgot about this post and left it in a draft state for quite a lot of time, I'm sorry!

How Python messed it up

A Diaspora of Tools

There's a plethora of packaging tools for Python! Let's take a look at them.

Distutils

The original distutils is part of the Python standard library, but it's not really feature-packed, and it's considered unsuitable for many production usages. As an example, its requires keyword seems to do absolutely nothing else than documenting requirements. Nothing will happen if you install a package requiring another one; you still need to install dependencies manually.

Distutils is not inherently bad, but since it's a standard library tool it's quite stable – it changes very little and in very predictable ways, so it wasn't updated for a long time and got quite old and tired, and needs an overhaul now (more on that later).

Setuptools

The original setuptools package, which was created by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it's unmantained right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.

It includes easy_install which is probably the first tool that allowed people to install packages straight from pypi (or other URLs, more on that later) on the command line, without a lot of manual intervention; also, if the downloaded package is setuptools-aware and correctly specifies dependencies with install_requires it will install all transitive dependencies.

It includes pkg_resources as well; that last one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of data which is installed in Python libraries, regardless of how it was installed – is it an egg? A flat install? Did the install became part of the standard library? – don't worry: just ask for the resource name and the package name, and pkg_resources will look up the file object for you and open it.

Unluckily the API was not so easy to use – that's something I'm trying to address in pydenji with its resourceloader URL-parsing based system – and many people to handle resource in python just “look for them” from the current directory using introspection by __file__. Too bad.

Distribute

In the end, setuptools had little maintenance applied and was forked; distribute is dropin replacement which shares functions and dysfunctions of its ancestor.

Recently it has been re-merged into setuptools.

This wouldn't be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing a specific-and-old version of setuptools, or you can find projects that know that setuptools is “old” and force installing distribute (which is an issue since setuptools is now an actually updated distribute) :–/

Distutils2

The Next Big Thing in Python packaging, which has never happened so far – should have been available in Python 3.3, but development seems to have stalled.

Pip

Pip is among the best tools you could use and works fine, but it's Yet Another Tool. It works as his author thinks it's well done – which is usually a good thing, since Ian Bicking knows what he's doing.

Dependency Management

Dependency management is hard with distribute; you can't handle diamond dependencies nor do overrides. Pypi repository is mutable (versions can be deleted by package owners) and isn't got a real API, even though, today, there's a way to prevent the packages from being fetched from their homepage and just rely on pypi – see PEP 438

One good way of handling dependency management in such cases is to use Pip's own requirements.txt file, which – of course – is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.

Consequences

It's very easy to find packages from pypi that don't work out of the box, either because of wrong dependencies, or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment is not easy.

And different tools just don't play well together.

What should be done?

  • Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python community.
  • Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can work, and if it doesn't, evolve it, don't just let people invent their in-house solution. It's not important if it's not part of the standard library.
  • Define best practices for packaging and dependencies (package layout, version numbers, metadata, uploaded package formats, etc) and be sure the user doesn't override them unless he knows what and why he does it.
  • Have a good doc linked from the official documentation. The current packaging guide is not complete and it's not linked from anything “official” – it's just another guide.

But?

There's a light at the end of the gallery, and it's not a train! We'll talk about it in part 3. Which I hope should happen soon enough :–)

#Ollivander #python #packaging #DependencyManagement

There's a practice of good programming which is often referred as One Assertion Per Test. In the opinion of some people, this practice would advocate the usage of a single call to xunit assert method for each test.

I think that idea is plainly wrong. Saying “One Test Per Test” would have been better, but implied an high tautology risk.

Let's take an old-and-trite shopping cart example; we have a cart which the user fills, then a checkout happens and we have a completed order, which holds things like a total amount for the order and a list of items.

Sometimes people will complain about this (Python is used throughout the examples, yet the concept applies to any language):


def test_successful_checkout(self):
    item1 = Item(price=10.00)
    item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    completed_order = Checkout.perform(cart)
    self.assertEquals(30.00, completed_order.amount)
    self.assertEquals([item1, item2], completed_order.items)

(I know floats aren't good for currency. I just didn't want to make the example too long.)

and yet would find the following completely acceptable:


def test_successful_checkout(self):
    item1 = Item(price=10.00)
    item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    completed_order = Checkout.perform(cart)
    expected_order = CompletedOrder(amount=30.00, items=[item1, item2])
    self.assertEquals(expected_order, completed_order)

Do you see? One assertion per test.

{% img center /images/download.jpeg %}

Frankly speaking, that's bullshit. One assertion per test does not mean that. One assertion per test means test ONE THING for each of your tests. You should begin thinking about whether your object does too many things and whether your test methods are properly named.

A possible example would be:


def setUp(self):
    item1 = self.item1 = Item(price=10.00)
    item2 = self.item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    self.completed_order = Checkout.perform(cart)

def test_successful_checkout_order_amount_is_the_sum_of_item_prices(self):
    self.assertEquals(30.00, self.completed_order.amount)

def test_successful_checkout_order_items_are_items_from_the_cart(self):
    self.assertEquals([self.item1, self.item2], self.completed_order.items)

Here, since the checkout process is doing multiple things (and it probably should as it is a checkout – this is probably more an example of an acceptance test rather than a unit test), we test them separately. If either fails, we know why and can go straight to the issue – and yet consider rewriting the second test like that:


def test_successful_checkout_order_items_are_items_from_the_cart(self):
    self.assertEquals(self.item1, self.completed_order.items[0])
    self.assertEquals(self.item2, self.completed_order.items[1])

If the example above is good, that's good as well, and you may have reasons for doing things like that – maybe your test framework doesn't show you the differences in a very clear way and you want to be sure to quickly catch the wrong item. This is what Robert C. Martin calls Single Concept per Test in his own Clean Code book – a recommended reading!

So, “You're testing multiple things in a single test” can be a good remark. “You're using multiple assertions per test”, on the contrary, means nothing.

#Ollivander #programming #testing #bestpractices #software