alan

Alan's musings

The Process, today, is king. There're a lot of people focusing just on process, and a lot of them hold it as The True Solution to all the world's problems.

Then, there're those who don't believe in process. They say, we mostly focus on people! Process is a failed attempt at commoditizing individuals; a lot of large organizations survive despite their processes, not because of them!

But then... you start noticing something. When you speak with agile folks, they talk about guidelines. You talk about Google, which is the poster child of the New Enterprise, where people are happy and aren't burdened by strange processes, and you discover that, when designing Site Reliability Engineering practices, checklists are considered an excellent tool.

So?

So, I think that process is useful. Sometimes, it's overly so. Yet, it must be the right kind of process.

Let's see a couple examples for an example process to request some days off.

Example A

In order to request one or more days off, you should send an email stating which days you'd like not to work to both your direct supervisor and the HR department. The request should be sent at least 10 business days before the first day off you'd like to take. Your direct supervisor has got five business days to either confirm or reject your request; if he doesn't answer, your request is automatically approved.

Another process clearly defines who's your supervisor and who takes its place should he be on vacation or ill.
Example B

In order to request one or more days off, you should download the proper *Ask For Some Days Off Form*, fill it in completely, sign it, and send it to your supervisor via email.

Enter the bureaucracy

I'm confident that most corporate environments follow Example B rather than Example A. Example B is the perfect bureaucratic process: while it is short and it doesn't seem overly complicated at a first glance, it is nebulous, and it may enable nothing.

Where should you download the right form? Most probably it won't be linked from the policy page. You'll need to find it somewhere. And, if you finally ask the template to your colleagues, make sure it's the latest, approved version. Then you fill it in. How? Maybe it's a PDF, and you need to print it, fill the required days in a box, sign it, scan it back and send it via email.

And then, what happens? Maybe, nothing. Maybe your supervisor is ill, and won't read emails. Maybe he isn't, but he won't check immediately whether there's somebody who can do your job while you're away, and will forget ever after. There's nothing in such process which guarantees an outcome. There's no explicit responsibility transfer – your supervisor won't risk anything if he doesn't answer your request; you're the only one who's impacted; maybe you won't know anything until the day before you'd like to leave!

A good, enabling process

A good process will make it clear why it exists; we should not create processes for their own sake. A good process establishes a clear guideline to reach a target. And we should not create processes before we need them! A special request which is made once every couple of years by a single employee can be handled manually.

A good process is easy to follow and is not unnecessarily complicated; it won't use forms, software or other specialized tools when simple ones will fit perfectly.

You always know who's in charge at a certain phase of a good process; it's impossible (or rare and highly unlikely) for the process to get stuck without somebody taking care. Inaction is prevented by a default action which allows the process to go on without somebody's consent, if he's unwilling to enter the loop.

A good process either produces the desired outcome, or fails as fast as possible.

An even better process

An even better process is the one which makes it clear which business rules are applied at each approval step; using again the example above, this could be:

For each team, it is required that at least two members to be in the office at all time, in order to handle the basic support levels needed by customers; to achieve such requirement and cope with illness or other emergencies, no days off will be allowed if that would reduce the number of working team members to a number lower than three.

In order to request one or more days off, you should send an email stating which days you'd like not to work to both your direct supervisor and the HR department. The request should be sent at least 10 business days before the first day you'd like off. Your direct supervisor has got five business days to either confirm or reject your request; if he doesn't answer, your request is automatically approved.

Another process clearly defines who's your supervisor and who takes its place should he be on vacation or ill.

This process is probably close to great; people in the team will probably auto-organize with their days off, and won't request things that will inevitably be denied, reducing HR and supervisor's work.

Let's make sure the Template Zombies stay dead, this time!

#Ollivander #process #bureaucracy

Are you unsure whether your mass storage works properly? Do you suspect it's faulty somewhere?

Verify it with a nifty tool, included in most Linux distros and in macOS if you install the ext2fs tools:

sudo badblocks -swv -t random -b 4096 -c 2048 <DEVICE>

It will perform a read-write test of your mass storage device, and tell you what's wrong.

WARNING: everything will be deleted during the test! Make sure you've already backupped your data!

#MostlyUnixish #tips #bash

As much as I like Docker, one thing it doesn't do well: cleanup whatever is left on my hard drive after running it for a while. There're generally two kinds of Docker Waste:

  • Exited containers. Unless you used run --rm when running a new container from an image, docker doesn't automagically remove containers once they're stopped; you need to rm them explicitly. Since it's generally considered (at least, by me) a Docker good practice to never restart a container, they just stay there and take valuable disk space. Of course sometimes it's useful or even necessary not to start a container with --rm, because you may want to inspect the filesystem after exiting, but most of the times it's an oversight.
  • Dangling images. Whenever you build a brand-new Docker image with the same tag as an existing one, you'll be possibly left with an old, dangling image – i.e an image you can only use by tag, not by name. The same could happen when an upstream image changes. More or less just like a git dangling object. Small note: it doesn't necessarily happen. The new image can be a perfect child of the previous image, in that case Docker uses a kind of inheritance mechanism which produces no dangling images. But if the image changes “in the middle” of the Dockerfile, and hence a piece of the old image get invalidated, then you'll be left with some dangling images.

So, here's the script. Just put it in your /etc/cron.daily directory and live quite happily ever after:

#!/bin/bash
docker ps -a -q --filter="status=created" | xargs --no-run-if-empty \
docker rm
docker ps -a -q --filter="status=exited" | xargs --no-run-if-empty \
docker rm
docker images -q -f dangling=true | xargs --no-run-if-empty \
docker rmi

#MostlyUnixish #docker

This seems a simple enough question:

How can I discover the type of any given Python object?

Suppose Python 2.7 (Python 3 is a different beast on this topic), and try implementing such function:

def discover_type_of_object(obj):
    """
    Args:
        obj: Any python object

    Returns:
        the type of input object
    """
    ....

When you're done, scroll the page way down...



































Now, try the function you've wrote with this case:

class Oldstyle:
    pass

class Otheroldstyle:
    pass

print discover_type_of_object(Oldstyle()) == discover_type_of_object(Otheroldstyle())

print discover_type_of_object(Oldstyle)

Does the first print statement write True and/or does the second statement raise an exception? If your output is

False
<type 'classobj'>

Congratulations, you've done everything right!

If not, you got stuck into the NewStyleVsClassicClasses problem, which makes it quite hard to do something as simple as getting the type of an arbitrary object. This is the code I created:

from types import ClassType
_NO_CLASS=object()
def get_object_type(obj):
    obj_type = getattr(obj, "__class__", _NO_CLASS)
    if obj_type is not _NO_CLASS:
        return obj_type
    # AFAIK the only situation where this happens is an old-style class
    obj_type = type(obj)
    if obj_type is not ClassType:
        raise ValueError("Could not determine object '{}' type.".format(obj_type))
    return obj_type

If it weren't that Python 2.7 is in maintenance mode only, I'd probably submit such utility for inclusion in the inspect module.

Originally answered on Stack Overflow: http://stackoverflow.com/questions/2225038/determine-the-type-of-an-object/37876801#37876801

#Ollivander #python #TypeSystem

Some years ago, a git workflow model called git-flow took the software development world by storm.

I'd always thought that the workflow was far too complex and I couldn't figure out what purpose it served. Now, after having used it for some months, I can just confirm my impression: git flow is too complex and has little practical use.

TL;DR: I have a single, precise objection to that: the heavy branching is counterproductive. It won't do any good to your developer workflow, merging multiple branches back to develop will lead to conflicts, and if you really want to do Continuous Integration with it, the setup will be complex.

What are all those branches for? And how do I integrate them in a CI pipeline? And if I don't integrate them, how do I test the changes they carry?

A small explanation: usual CI pipelines offer a set of deployment hosts that continuously integrate the changes from a branch: as an example, let's imagine we've got three components in a project, from three separate repositories:

  • frontend
  • backend
  • data processing

The frontend communicates with the backend, and there's a back-and-forth two-way communication between the data processing and the backend component.

There'll probably be a machine in your staging/development infrastructure called something like integration-master (master is the branch) where those three components are continually deployed and can be tested by anyone, constituting the current state of the art of your deployment (remember: works on my machine is not enough).

It's even possible that those components are kept physically separated; so you'll have integration machines named like integration-frontend-master, integration-backend-master, and so on; for each component and branch, you'll get a new host.

What should I do with git flow? Suppose I create a new backend-only feature, feature/antigravity, departing from the develop branch. Suppose that I can create a proper CI pipeline that builds and tests such feature. Where am I going to deploy such branch? Do i need to create a new hostname in the internal network on the fly? And what frontends and data processing machines should I use? Should I deploy new machines for the frontend and the data processing from the develop branch in order to let the whole system with such feature to be tested?

Yes, I could do that. But that doesn't mean it's a good idea. And surely it's not as simple to implement as a fixed branches model.

Or you could just skip doing the deployment for the feature branches, they'll be tested when they're merged back to develop. Great – now the feature branches are completely useless.

And if I have indeed many feature branches at the same time, won't I experience the same old integration nightmare that killed many a programmer once upon a time? Every merge back to the develop branch would carry a substantial risk of getting messy. Wasn't continuous integration meant to avoid that?

Let's see the alternatives.

git builtin workflows

git help workflows

shows many possible workflows; some include the so-called topic branches, which resemble git-flow feature branches; I don't especially like them, since they have similar issues.

A simpler workflow proposal

For a standard piece of software with linear development – i.e. a software which is got exactly one stable version and for which I don't need to do maintenance of old versions, a simpler model is (use branch names as you prefer):

  • master : holds the current stable release; this is the version of the software which is currently pushed to production.
  • rc : normally, this branch points at the very same commit as master; but if a bugfix to the production software is needed, it's done in this branch, it's tested, and then is quickly merged to master or rollbacked. rc should never diverge from master for long periods of times, it's not meant for that. When doing a new software release (which should happen often), develop is merged to rc, it's briefly tested with the production configuration, and then is merged to master for the final release.
  • develop: where the actual development happens. Everybody commits and pushes to this branch. When the needed feature set for a release is ready, the branch gets merged to rc. Feature Toggles are employed as needed, so that even if a merge of an incomplete or untested feature goes to rc and then to master it's not enabled by default and can do no harm. Any bugfix merged to master must be merged to develop as well.
  • experimental: this is optional and may not be always active; yes, sometimes there's a huge refactoring or there's an experimental feature which really requires a separate branch. But it's a last resort. It's usually created at a certain point from the develop branch, where it gets merged back if the experiment goes well.

This approach is quite simple and easy to understand, and should solve most development issues.

Does it sound familiar? Does it look like Chrome Release Channels? And Chrome Flags look very much like feature toggles?

Here you are.

Update 28 Jul 2016

After some discussions with some people, I'd like to clarify one point: it is entirely possible that you sometimes want some long-lived, named branches. And that, in such context, you'll want to setup separate CI/CD machines for those branches.

What I reject is the a priori attitude towards branching that git-flow tries to pass as a good practice.

#Ollivander #git #workflow #GitFlow

Every now and then, when using Python 2.7 < 2.7.9 and trying to access SSL resources, especially through the requests toolkit, which seems to trigger the issue frequently – but I've seen it on some combinations of pip inside virtualenv as well – you'll get an error or a warning along those lines:

InsecurePlatformWarning: A true SSLContext object is not
available. This prevents urllib3 from configuring SSL appropriately and 
may cause certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest  
/security.html#insecureplatformwarning.

or

SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning

or, even worse:

error: SSLError: hostname 'xxx.franzoni.eu' doesn't match either of www.franzoni.eu, franzoni.eu

This is caused by old libraries in python < 2.7.9 and < 3.4. To fix that, just add to your current virtualenv those three packages:

pip install pyOpenSSL ndg-httpsclient pyasn1

You may need runtime/development packages for python and openssl as well in order for the build to succeed, e.g. python-dev libssl-dev libffi-dev on Ubuntu 14.04.

#MostlyUnixish #python #requests #ssl

Each and every time I think about Parkinson's law, I think that Cyril Northcote Parkinson was a sort of genius, who could gain a great insight into people's heads.

For the few that have never heard about it, it's as simple as:

Work expands so as to fill the time available for its completion

I've not read the original book yet, but I've ordered it and I'll sure get back to the topic once I've read it.

What I'd like to talk about today is:

How does Parkinson's law influence software projects?

I believe that Parkinson's law is a major factor in delays in the software industry; that is, especially in situations where a turnkey approach is expected, but agile/iterative approaches are not exempt, it's just the effects that are smaller.

Whenever we produce an estimate and then give somebody a quote, be it a client or an internal customer, it's usually quite common to keep such estimate as a reference for the work being done.

If we're late, we cry and try to “do things faster”. If, on the contrary, we are able to deliver faster than estimated, something weird happens: it's the “fraudphobia”. If a customer is paying us for doing a certain work and we finish too early, it looks like we're scamming them, so we do unimportant things, maybe we refactor something or improve the product documentation.

This is true on the macro level (whole project estimates) as well as the micro level (single tasks that are individually estimated and whose sum produce the grand total for the project). This behaviour ignores the fact that estimates are usually wild guesses that should guide us during the project, but are not a detailed plan to be followed. It's perfectly normal for tasks to take longer than estimated. But, unless you're estimating a mythical scenario where nothing goes wrong, it's perfecly normal for tasks to take less time than estimated as well.

Let's suppose that we've got ten 100-man-days tasks in a project, and that half of those would be finished twenty days early, while the other half would finish with a twenty days delay. This would result on a perfectly on-time project.

But, most probably, if finishing before the estimated time, we would find other, possibly not-so-important things or improvements to do in order to fully employ the allocated time for each task. Hence, all tasks would take at least the time that was allocated for them, leading to a final 10% delay on the project. Parkinson's law along with estimates as references makes it impossible to finish a project on time.

Takeaway lessons

  • If you never overshoot, you're consistently underestimating. Try harder.
  • Plans are useless, but planning is indispensable (D.D. Eisenhower)
  • It's done when it's done. How much we initially thought it would take us is irrelevant 1

1: Of course comparing our estimates with the outcomes after a task is finished is a good idea, because it helps us at aiming better for our next estimate; but we should never use the estimate as a guidance.

#Ollivander #management #agile #estimates

This is one very, very, very interesting piece of ancient wisdom.

Suppose you need to change permissions for some directories. Maybe the permissions are too strict, e.g. something like 700 for directories and 600 for files, and you need to give access to some other group and not just the owner.

As you know, execute permission on directories is required for traversal. So you might think doing something like:

chmod -R g+rx /var/somedirectory

This would probably work, but would leave you in a sad state: you'll get the executable bit set on files that are not supposed to be executable at all, which may be just annoying but may get risky as well.

A great alternative is:

chmod -R g+rX /var/somedirectory

the capital X means: set the executable bit only if the target a) is a directory b) has already at least one executable bit set for any one of user, group, others.

There's another alternative that may be appealing in some situations, at least when you want to give the group the same permissions as the owner:

chmod -R g=u /var/somedirectory

This will give any file the group permissions identical to the user's permission.

#MostlyUnixish #linux #ancientwisdom #chmod

I was tired of repeating the same old pattern. Send a kill signal. Check with pgrep. Did the process exit? Send a kill -9.

Now all my knowledge has been permanently dumped here:

Kill Me Softly on GitHub

If you want to kill a process by letting it to exit gracefully, and yet kill it without mercy if it's too slow to exit, that's your new favourite shell script.

You can discuss this on Hacker News as well

#MostlyUnixish #linux #processmanagement #kill

A desktop freeze is something that's not so rare nowadays – that's not good, but we shouldn't get desperate about that.

I like Compiz with Unity Desktop on my Ubuntu Trusty very much; I feel it's a step forward compared to traditional desktops. The only major nuisance I had on the previous versions was the inability to disable the global menu (which I like by itself, but I don't like the global menu disappearing when the mouse is not over it), but you can now disable it and have the menu on the window – even though you can't still enable an always-visible behaviour.

The problem is: compiz sometimes gets buggy. Running a different window manager, like metacity and/or xcompmgr just doesn't seem good, you can experience other issues and you're using the stack in a way it wasn't designed for. I don't like that approach.

So, what can you do when your desktop freezes? The symptoms are usually very clear: you can move your mouse, but you can't click nor type anything.

You don't need to reboot nor to restart X. You can fix this without losing your work.

Get a raw tty

Press Ctrl+Alt+F1 and get to a raw linux console, and login with your credentials. Make sure you use the same user you're logged into X with.

Find and kill your stuck process

pgrep compiz

This will print the PID of our stuck window manager.

Let's kill it

pkill compiz

This won't usually be enough because the process is really stuck: if you repeat the pgrep command above, you'll find it still running – i.e. pgrep will return the very same PID as above.

So let's kill it, really this time!

pkill -9 compiz

Then wait some 15-20 seconds.

Now, if you repeat the pgrep , you'll either see a) no result or b) a different PID.

If you see a different PID, it's Ubuntu that noticed the crashed process and relaunched it; press Ctrl+Alt+F7, and you should just get back a working desktop!

Relaunching

If you see nothing with pgrep, you'll need to relaunch Compiz manually.

You cannot simply relaunch compiz on the console, because it must connect to X: the proper way to do it is:

DISPLAY=:0 compiz

Now switch back to your X instance with Ctrl+Alt+F7 and enjoy!

#MostlyUnixish #linux #ancientwisdom #community #ubuntu #compiz