alan

Ollivander

Really. Stop (ab)using just strings and integers!

Yes, they're the building blocks of whatever you do. But if you abuse them, you're not taking full advantage of your Object Oriented programming language.

Let me explain with some concrete example. For the sake of conciseness, I'll use Python here, but the concepts fit most OO languages. Classes are your friends.

Really. Classes and type systems weren't invented to make you feel miserable and add bloating to your code. They were invented to help you to write correct code, letting the machine verify you're doing things properly and consistently.

But if you don't use them, the machine can't help you. And no, dynamically (yet strongly typed) languages aren't a good reason to skip correctness and proper OO design.

Let's start with some examples:

def retrieve_web_resource(http_url):
    #[...]
    requests.get(...)

What is that http_url ? In many contexts, it will probably be a string. But this is not a best practice. Primitive types like strings carry very little semantics and probably do not enforce domain boundaries. What does that mean?

It means that somewhere in your code you have something like

    my\_http\_url = somewhere.create_url_from()

(with my_http_url still being of type str) and then you carry it around as a str. Maybe, at some point, you manipulate it. And, possibly because of some error or because some unexpected data was thrown in, at some point it ceases being a valid http url.

This might mean that, in many parts of your code, whenever you use such value, you may be tempted to check some preconditions:

def retrieve_web_resource(http_url):
    assert http_url.starswith("http"), "not an http url"

But then, you need to check those preconditions everywhere!

And, if you happen to pass your variable around, and at some point you pick a less-than-meaningful variable name, it may be hard to understand, when debugging or if a serious error happened that totally messed up the content, what that variable was supposed to contain.

So, what is a much better solution? Create the right type, always, and use those types in your application. Example:

class HTTPURL(object):
   def __init__(self, scheme, host, path, query=None):
       if scheme != "http":
            raise ValueError, "not an http url"
       # validate host
       # validate path
       # validate query - should be a param-value dict
       self._host = host
       self._path_parts = path_parts
       self._query = query if query is not None else {}

   @classmethod
   def parse_from_str(cls, http_url_as_str):
       # parse the url, then
       return cls(scheme, host, path, query)

   def add_query_param(self, param, value):
       # add param and value to dictionary, return a new
       # HTTPURL object with that additional value. Raise
       # an error if such value is already defined

   def append_path(self, additional_path):
       # properly append additional_path, checking for 
       # missing or duplicate initial slashes, etc, then
       # return a new HTTPURL object.
    
   def __str__(self):
       # create the url from current variables, and return it as
       # a str

(this is simplified – you might want to add auth data, port, etc)

See the point? You're making sure that no invalid http_url instance can exist, and you're providing proper methods to manipulate such instance. This way you're encapsulating your domain knowledge about URLs in a specific class, instead of scattering and duplicating it everywhere and you're making it really hard for users to commit accidental mistakes – they would be caught quickly.

The HTTPURL we've seen here is composite and this approach may make a lot of sense, but in Python you can even use Tiny Types in a very, VERY efficient fashion, when you just need to add some semantics and some validation without a full-fledged class because you just don't need that kind of manipulation:

class Angle(int):
    def __new__(cls, v):
        if isinstance(v, Angle):
            return v
        if v < 0 or v > 360:
            raise ValueError("invalid angle, must be within 0 and 360")
        if not isinstance(v, (int, long)):
            raise TypeError("invalid angle type, must be int or long")
        return super(Angle, cls).__new__(cls, v)

See? The instances you create are of type Angle, but they're still an int! So you can pass them straight to other apis/libraries that just don't support or know nothing about your custom types, and yet you get semantics and validation!

Of course, if you need more operations you'll need to redefine more methods:

class Angle(int):
    def __new__(cls, v):
        if isinstance(v, Angle):
            return v
        if v < 0 or v > 360:
            raise ValueError("invalid angle, must be within 0 and 360")
        if not isinstance(v, (int, long)):
            raise TypeError("invalid angle type, must be int or long")
        return super(Angle, cls).__new__(cls, v)

    def __add__(self, other):
        return Angle(int(self) + int(Angle(other)))

angle = Angle(10)
angle2 = Angle(2)
angle3 = Angle(angle2)

print angle+angle2
print type(angle+angle2)
12
<class '__main__.Angle'>

Do you think all that is too verbose, and you'd just prefer to stick to string and ints? Well... I suggest you try this approach, check how many bugs you prevent and/or how your code becomes more expressive and cohesive, and then think again about verbosity. If your code is more compact at the expense of correctness and automated checking, maybe that compactness isn't worth it.

#Ollivander #programming #TypeSystem

The Process, today, is king. There're a lot of people focusing just on process, and a lot of them hold it as The True Solution to all the world's problems.

Then, there're those who don't believe in process. They say, we mostly focus on people! Process is a failed attempt at commoditizing individuals; a lot of large organizations survive despite their processes, not because of them!

But then... you start noticing something. When you speak with agile folks, they talk about guidelines. You talk about Google, which is the poster child of the New Enterprise, where people are happy and aren't burdened by strange processes, and you discover that, when designing Site Reliability Engineering practices, checklists are considered an excellent tool.

So?

So, I think that process is useful. Sometimes, it's overly so. Yet, it must be the right kind of process.

Let's see a couple examples for an example process to request some days off.

Example A

In order to request one or more days off, you should send an email stating which days you'd like not to work to both your direct supervisor and the HR department. The request should be sent at least 10 business days before the first day off you'd like to take. Your direct supervisor has got five business days to either confirm or reject your request; if he doesn't answer, your request is automatically approved.

Another process clearly defines who's your supervisor and who takes its place should he be on vacation or ill.
Example B

In order to request one or more days off, you should download the proper *Ask For Some Days Off Form*, fill it in completely, sign it, and send it to your supervisor via email.

Enter the bureaucracy

I'm confident that most corporate environments follow Example B rather than Example A. Example B is the perfect bureaucratic process: while it is short and it doesn't seem overly complicated at a first glance, it is nebulous, and it may enable nothing.

Where should you download the right form? Most probably it won't be linked from the policy page. You'll need to find it somewhere. And, if you finally ask the template to your colleagues, make sure it's the latest, approved version. Then you fill it in. How? Maybe it's a PDF, and you need to print it, fill the required days in a box, sign it, scan it back and send it via email.

And then, what happens? Maybe, nothing. Maybe your supervisor is ill, and won't read emails. Maybe he isn't, but he won't check immediately whether there's somebody who can do your job while you're away, and will forget ever after. There's nothing in such process which guarantees an outcome. There's no explicit responsibility transfer – your supervisor won't risk anything if he doesn't answer your request; you're the only one who's impacted; maybe you won't know anything until the day before you'd like to leave!

A good, enabling process

A good process will make it clear why it exists; we should not create processes for their own sake. A good process establishes a clear guideline to reach a target. And we should not create processes before we need them! A special request which is made once every couple of years by a single employee can be handled manually.

A good process is easy to follow and is not unnecessarily complicated; it won't use forms, software or other specialized tools when simple ones will fit perfectly.

You always know who's in charge at a certain phase of a good process; it's impossible (or rare and highly unlikely) for the process to get stuck without somebody taking care. Inaction is prevented by a default action which allows the process to go on without somebody's consent, if he's unwilling to enter the loop.

A good process either produces the desired outcome, or fails as fast as possible.

An even better process

An even better process is the one which makes it clear which business rules are applied at each approval step; using again the example above, this could be:

For each team, it is required that at least two members to be in the office at all time, in order to handle the basic support levels needed by customers; to achieve such requirement and cope with illness or other emergencies, no days off will be allowed if that would reduce the number of working team members to a number lower than three.

In order to request one or more days off, you should send an email stating which days you'd like not to work to both your direct supervisor and the HR department. The request should be sent at least 10 business days before the first day you'd like off. Your direct supervisor has got five business days to either confirm or reject your request; if he doesn't answer, your request is automatically approved.

Another process clearly defines who's your supervisor and who takes its place should he be on vacation or ill.

This process is probably close to great; people in the team will probably auto-organize with their days off, and won't request things that will inevitably be denied, reducing HR and supervisor's work.

Let's make sure the Template Zombies stay dead, this time!

#Ollivander #process #bureaucracy

This seems a simple enough question:

How can I discover the type of any given Python object?

Suppose Python 2.7 (Python 3 is a different beast on this topic), and try implementing such function:

def discover_type_of_object(obj):
    """
    Args:
        obj: Any python object

    Returns:
        the type of input object
    """
    ....

When you're done, scroll the page way down...



































Now, try the function you've wrote with this case:

class Oldstyle:
    pass

class Otheroldstyle:
    pass

print discover_type_of_object(Oldstyle()) == discover_type_of_object(Otheroldstyle())

print discover_type_of_object(Oldstyle)

Does the first print statement write True and/or does the second statement raise an exception? If your output is

False
<type 'classobj'>

Congratulations, you've done everything right!

If not, you got stuck into the NewStyleVsClassicClasses problem, which makes it quite hard to do something as simple as getting the type of an arbitrary object. This is the code I created:

from types import ClassType
_NO_CLASS=object()
def get_object_type(obj):
    obj_type = getattr(obj, "__class__", _NO_CLASS)
    if obj_type is not _NO_CLASS:
        return obj_type
    # AFAIK the only situation where this happens is an old-style class
    obj_type = type(obj)
    if obj_type is not ClassType:
        raise ValueError("Could not determine object '{}' type.".format(obj_type))
    return obj_type

If it weren't that Python 2.7 is in maintenance mode only, I'd probably submit such utility for inclusion in the inspect module.

Originally answered on Stack Overflow: http://stackoverflow.com/questions/2225038/determine-the-type-of-an-object/37876801#37876801

#Ollivander #python #TypeSystem

Some years ago, a git workflow model called git-flow took the software development world by storm.

I'd always thought that the workflow was far too complex and I couldn't figure out what purpose it served. Now, after having used it for some months, I can just confirm my impression: git flow is too complex and has little practical use.

TL;DR: I have a single, precise objection to that: the heavy branching is counterproductive. It won't do any good to your developer workflow, merging multiple branches back to develop will lead to conflicts, and if you really want to do Continuous Integration with it, the setup will be complex.

What are all those branches for? And how do I integrate them in a CI pipeline? And if I don't integrate them, how do I test the changes they carry?

A small explanation: usual CI pipelines offer a set of deployment hosts that continuously integrate the changes from a branch: as an example, let's imagine we've got three components in a project, from three separate repositories:

  • frontend
  • backend
  • data processing

The frontend communicates with the backend, and there's a back-and-forth two-way communication between the data processing and the backend component.

There'll probably be a machine in your staging/development infrastructure called something like integration-master (master is the branch) where those three components are continually deployed and can be tested by anyone, constituting the current state of the art of your deployment (remember: works on my machine is not enough).

It's even possible that those components are kept physically separated; so you'll have integration machines named like integration-frontend-master, integration-backend-master, and so on; for each component and branch, you'll get a new host.

What should I do with git flow? Suppose I create a new backend-only feature, feature/antigravity, departing from the develop branch. Suppose that I can create a proper CI pipeline that builds and tests such feature. Where am I going to deploy such branch? Do i need to create a new hostname in the internal network on the fly? And what frontends and data processing machines should I use? Should I deploy new machines for the frontend and the data processing from the develop branch in order to let the whole system with such feature to be tested?

Yes, I could do that. But that doesn't mean it's a good idea. And surely it's not as simple to implement as a fixed branches model.

Or you could just skip doing the deployment for the feature branches, they'll be tested when they're merged back to develop. Great – now the feature branches are completely useless.

And if I have indeed many feature branches at the same time, won't I experience the same old integration nightmare that killed many a programmer once upon a time? Every merge back to the develop branch would carry a substantial risk of getting messy. Wasn't continuous integration meant to avoid that?

Let's see the alternatives.

git builtin workflows

git help workflows

shows many possible workflows; some include the so-called topic branches, which resemble git-flow feature branches; I don't especially like them, since they have similar issues.

A simpler workflow proposal

For a standard piece of software with linear development – i.e. a software which is got exactly one stable version and for which I don't need to do maintenance of old versions, a simpler model is (use branch names as you prefer):

  • master : holds the current stable release; this is the version of the software which is currently pushed to production.
  • rc : normally, this branch points at the very same commit as master; but if a bugfix to the production software is needed, it's done in this branch, it's tested, and then is quickly merged to master or rollbacked. rc should never diverge from master for long periods of times, it's not meant for that. When doing a new software release (which should happen often), develop is merged to rc, it's briefly tested with the production configuration, and then is merged to master for the final release.
  • develop: where the actual development happens. Everybody commits and pushes to this branch. When the needed feature set for a release is ready, the branch gets merged to rc. Feature Toggles are employed as needed, so that even if a merge of an incomplete or untested feature goes to rc and then to master it's not enabled by default and can do no harm. Any bugfix merged to master must be merged to develop as well.
  • experimental: this is optional and may not be always active; yes, sometimes there's a huge refactoring or there's an experimental feature which really requires a separate branch. But it's a last resort. It's usually created at a certain point from the develop branch, where it gets merged back if the experiment goes well.

This approach is quite simple and easy to understand, and should solve most development issues.

Does it sound familiar? Does it look like Chrome Release Channels? And Chrome Flags look very much like feature toggles?

Here you are.

Update 28 Jul 2016

After some discussions with some people, I'd like to clarify one point: it is entirely possible that you sometimes want some long-lived, named branches. And that, in such context, you'll want to setup separate CI/CD machines for those branches.

What I reject is the a priori attitude towards branching that git-flow tries to pass as a good practice.

#Ollivander #git #workflow #GitFlow

Each and every time I think about Parkinson's law, I think that Cyril Northcote Parkinson was a sort of genius, who could gain a great insight into people's heads.

For the few that have never heard about it, it's as simple as:

Work expands so as to fill the time available for its completion

I've not read the original book yet, but I've ordered it and I'll sure get back to the topic once I've read it.

What I'd like to talk about today is:

How does Parkinson's law influence software projects?

I believe that Parkinson's law is a major factor in delays in the software industry; that is, especially in situations where a turnkey approach is expected, but agile/iterative approaches are not exempt, it's just the effects that are smaller.

Whenever we produce an estimate and then give somebody a quote, be it a client or an internal customer, it's usually quite common to keep such estimate as a reference for the work being done.

If we're late, we cry and try to “do things faster”. If, on the contrary, we are able to deliver faster than estimated, something weird happens: it's the “fraudphobia”. If a customer is paying us for doing a certain work and we finish too early, it looks like we're scamming them, so we do unimportant things, maybe we refactor something or improve the product documentation.

This is true on the macro level (whole project estimates) as well as the micro level (single tasks that are individually estimated and whose sum produce the grand total for the project). This behaviour ignores the fact that estimates are usually wild guesses that should guide us during the project, but are not a detailed plan to be followed. It's perfectly normal for tasks to take longer than estimated. But, unless you're estimating a mythical scenario where nothing goes wrong, it's perfecly normal for tasks to take less time than estimated as well.

Let's suppose that we've got ten 100-man-days tasks in a project, and that half of those would be finished twenty days early, while the other half would finish with a twenty days delay. This would result on a perfectly on-time project.

But, most probably, if finishing before the estimated time, we would find other, possibly not-so-important things or improvements to do in order to fully employ the allocated time for each task. Hence, all tasks would take at least the time that was allocated for them, leading to a final 10% delay on the project. Parkinson's law along with estimates as references makes it impossible to finish a project on time.

Takeaway lessons

  • If you never overshoot, you're consistently underestimating. Try harder.
  • Plans are useless, but planning is indispensable (D.D. Eisenhower)
  • It's done when it's done. How much we initially thought it would take us is irrelevant 1

1: Of course comparing our estimates with the outcomes after a task is finished is a good idea, because it helps us at aiming better for our next estimate; but we should never use the estimate as a guidance.

#Ollivander #management #agile #estimates

A bit of wrap-up after part 1 and part 2

Current state of the things

I seem to have missed something: there's now a very active Python Packaging Authority which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.

Setuptools and distribute got merged and we won't experience a split-experience anymore, we'll just have One Tool To Rule Them All!

So, rejoice! Things are getting better!

In the meantime

I recommend using Pip . Always try using requirements.txt, it's a good way to keep your versions consistent and working.

Use virtualenv for development purposes, and always test your package in a virtualenv with a disabled site-packages! Otherwise you risk you're distributing something that works on your workstation only because of a site-wide package you have installed there!

In order to distribute a package, you'll need setuptools/distribute as well; just do a source distribution and a wheel distribution (you're required to install wheel beside setuptools); forget about eggs, especially for eggs containing binary extensions. Your command line should be something like:

python setup.py clean register sdist bdist_wheel upload

And let's keep our fingers crossed on a better state of things!

#Ollivander #python #packaging #DependencyManagement

Check out part 1 for my thoughts on what packaging and dependency management should be. I forgot about this post and left it in a draft state for quite a lot of time, I'm sorry!

How Python messed it up

A Diaspora of Tools

There's a plethora of packaging tools for Python! Let's take a look at them.

Distutils

The original distutils is part of the Python standard library, but it's not really feature-packed, and it's considered unsuitable for many production usages. As an example, its requires keyword seems to do absolutely nothing else than documenting requirements. Nothing will happen if you install a package requiring another one; you still need to install dependencies manually.

Distutils is not inherently bad, but since it's a standard library tool it's quite stable – it changes very little and in very predictable ways, so it wasn't updated for a long time and got quite old and tired, and needs an overhaul now (more on that later).

Setuptools

The original setuptools package, which was created by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it's unmantained right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.

It includes easy_install which is probably the first tool that allowed people to install packages straight from pypi (or other URLs, more on that later) on the command line, without a lot of manual intervention; also, if the downloaded package is setuptools-aware and correctly specifies dependencies with install_requires it will install all transitive dependencies.

It includes pkg_resources as well; that last one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of data which is installed in Python libraries, regardless of how it was installed – is it an egg? A flat install? Did the install became part of the standard library? – don't worry: just ask for the resource name and the package name, and pkg_resources will look up the file object for you and open it.

Unluckily the API was not so easy to use – that's something I'm trying to address in pydenji with its resourceloader URL-parsing based system – and many people to handle resource in python just “look for them” from the current directory using introspection by __file__. Too bad.

Distribute

In the end, setuptools had little maintenance applied and was forked; distribute is dropin replacement which shares functions and dysfunctions of its ancestor.

Recently it has been re-merged into setuptools.

This wouldn't be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing a specific-and-old version of setuptools, or you can find projects that know that setuptools is “old” and force installing distribute (which is an issue since setuptools is now an actually updated distribute) :–/

Distutils2

The Next Big Thing in Python packaging, which has never happened so far – should have been available in Python 3.3, but development seems to have stalled.

Pip

Pip is among the best tools you could use and works fine, but it's Yet Another Tool. It works as his author thinks it's well done – which is usually a good thing, since Ian Bicking knows what he's doing.

Dependency Management

Dependency management is hard with distribute; you can't handle diamond dependencies nor do overrides. Pypi repository is mutable (versions can be deleted by package owners) and isn't got a real API, even though, today, there's a way to prevent the packages from being fetched from their homepage and just rely on pypi – see PEP 438

One good way of handling dependency management in such cases is to use Pip's own requirements.txt file, which – of course – is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.

Consequences

It's very easy to find packages from pypi that don't work out of the box, either because of wrong dependencies, or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment is not easy.

And different tools just don't play well together.

What should be done?

  • Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python community.
  • Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can work, and if it doesn't, evolve it, don't just let people invent their in-house solution. It's not important if it's not part of the standard library.
  • Define best practices for packaging and dependencies (package layout, version numbers, metadata, uploaded package formats, etc) and be sure the user doesn't override them unless he knows what and why he does it.
  • Have a good doc linked from the official documentation. The current packaging guide is not complete and it's not linked from anything “official” – it's just another guide.

But?

There's a light at the end of the gallery, and it's not a train! We'll talk about it in part 3. Which I hope should happen soon enough :–)

#Ollivander #python #packaging #DependencyManagement

There's a practice of good programming which is often referred as One Assertion Per Test. In the opinion of some people, this practice would advocate the usage of a single call to xunit assert method for each test.

I think that idea is plainly wrong. Saying “One Test Per Test” would have been better, but implied an high tautology risk.

Let's take an old-and-trite shopping cart example; we have a cart which the user fills, then a checkout happens and we have a completed order, which holds things like a total amount for the order and a list of items.

Sometimes people will complain about this (Python is used throughout the examples, yet the concept applies to any language):


def test_successful_checkout(self):
    item1 = Item(price=10.00)
    item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    completed_order = Checkout.perform(cart)
    self.assertEquals(30.00, completed_order.amount)
    self.assertEquals([item1, item2], completed_order.items)

(I know floats aren't good for currency. I just didn't want to make the example too long.)

and yet would find the following completely acceptable:


def test_successful_checkout(self):
    item1 = Item(price=10.00)
    item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    completed_order = Checkout.perform(cart)
    expected_order = CompletedOrder(amount=30.00, items=[item1, item2])
    self.assertEquals(expected_order, completed_order)

Do you see? One assertion per test.

{% img center /images/download.jpeg %}

Frankly speaking, that's bullshit. One assertion per test does not mean that. One assertion per test means test ONE THING for each of your tests. You should begin thinking about whether your object does too many things and whether your test methods are properly named.

A possible example would be:


def setUp(self):
    item1 = self.item1 = Item(price=10.00)
    item2 = self.item2 = Item(price=20.00)
    cart = Cart()
    cart.add(item1)
    cart.add(item2)
    self.completed_order = Checkout.perform(cart)

def test_successful_checkout_order_amount_is_the_sum_of_item_prices(self):
    self.assertEquals(30.00, self.completed_order.amount)

def test_successful_checkout_order_items_are_items_from_the_cart(self):
    self.assertEquals([self.item1, self.item2], self.completed_order.items)

Here, since the checkout process is doing multiple things (and it probably should as it is a checkout – this is probably more an example of an acceptance test rather than a unit test), we test them separately. If either fails, we know why and can go straight to the issue – and yet consider rewriting the second test like that:


def test_successful_checkout_order_items_are_items_from_the_cart(self):
    self.assertEquals(self.item1, self.completed_order.items[0])
    self.assertEquals(self.item2, self.completed_order.items[1])

If the example above is good, that's good as well, and you may have reasons for doing things like that – maybe your test framework doesn't show you the differences in a very clear way and you want to be sure to quickly catch the wrong item. This is what Robert C. Martin calls Single Concept per Test in his own Clean Code book – a recommended reading!

So, “You're testing multiple things in a single test” can be a good remark. “You're using multiple assertions per test”, on the contrary, means nothing.

#Ollivander #programming #testing #bestpractices #software

This is a spinoff from my Europython 2012 talk (video, slides).

Some days ago I got involved in a brief discussion on the topic, so here's what I think about Python packaging and dependency management: it's in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.

Why? Let's see.

The Packaging Conspiracy

Aim

No, it's not a conspiracy to force you to learn something you don't care about. The whole point of packaging is to make code reuse easy enough, both for the developer's own sake (e.g. shared code between different projects) and for sharing it with the public.

Packaging usually involves dependency management; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don't just release your code in “in the dark”: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.

It's an overkill!

Some developers think all of this is a waste of their time.

D: Hey Alan, are you really saying say I can't just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?

A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you've fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, you're doing by hand the work that a good dependency management tool is supposed to do. If you like that.

D: OK, ok. I got it. So now I've added my dependencies. But what the heck, does the version really matter? I don't care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won't break my code.

A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it's time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.

D: OK, ok, now I've set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?

A: Try doing something meaningful, and stick to your plan. You can pick your own versioning policy. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something “emotional”. If you perform small incremental changes on your codebase – a-la-Chrome – you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.

D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!

A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.

D: YEAH! I've just released version 0.0.2 to the public. Now I'm deleting 0.0.1, right?

A: No. Never. Ideally, unless you must – by must I mean something like “there's a password for my bank account in the source code” or “I mistakenly copypasted code for which I hadn't the copyright” – you should never, ever, ever, remove a released version. You never know who's using it – if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn't care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as “deprecated”, “bugged”, or maybe “donut” if they contain a huge security hole.

D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it's my first daughter and she was born in the very day of such release.

A: I'm happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, “lexicographical” is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means “lower”, so that 2.0.0beta is < 2.0.0, but that's not always the case – a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.'' is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.

D: There're a lot of things to remember about dependency management and packaging. Should I learn everything by heart?

A: What the heck, no. Your packaging tool should guide you in such practices and prevent you from going stray unless it's what you really mean to do! That why I'm complaining in the first place about the way Python does packaging!

In part 2 I dig into Python-specific packaging issues.

#Ollivander #python #packaging

I don't like php, and I've got my reasons. I've met a lot of people who dislike PHP and don't see how possibly anyone, anyhow, could think about ever using PHP in any real-world project out there, and simply dismiss PHP as a crap being created and used from and for incompetent stupid people around the world.

On the other side there're a bunch of PHP supporters which show you how wonderful is PHP, how many people use it and how widespread it is, and whoever says the contrary is just an enterprisey fool which spends nights at getting things done in Java or C#.

I disagree with both groups (of course!) and I think that PHP's got its unique set of killer features that make it still valuable, and those should be acknowledged.

First, it's not PHP by itself – there's the [LAMP stack](http://en.wikipedia.org/wiki/LAMP_(software_bundle%29), a de-facto pseudo-standard; interoperability of PHP applications on many different hostings work quite well. This is very different from most other platforms, where you must either adapt your development environment to what your hosting provides, like in Google App Engine, or you must find a way to tune your hosting environment to match your required dependencies. Not even Java, which heavily relies on standardized APIs, was able to reach a similar consensus on a standard stack.

And this sort of Platform-as-a-Service offering is widespread. This is catching on lately for other languages with things like Google App Engine, Jelastic, Engineyard, Heroku (while this last one is slightly more complex), but on PHP there's simply a incredible amount of really cheap and simple hosting. If you're willing to spend something like 50 EUR an year you can find out really good PHP hostings just everywhere, and you can expect to go down to 15 EUR per year and still have something that's pretty decent.

What's all this hosting good for? Do you create your own custom software in PHP and run on top of that? Of course not! Many experienced PHP developers go with their own infrastracture in order to have full control and tuning; the real bulk of PHP installations (IMHO – I don't have collected actual data on such topic, maybe one day I will) consists of existing free software installations. Existing software written in PHP makes the difference. There're a lot of people that simply don't care about PHP or programming at all, and yet they use PHP every day thanks to existing open source software.

Do you want a blog engine that you can customize and offers a variety of plugins that have nothing to do with a blog? Here's Wordpress. Would you like to manage a community? Drupal is your friend.

Please note that, more often than not, the available PHP software is not the “best of the breed”. Other software exists that is better, or faster, or offers more functionality, but PHP is usually “good enough”, allows “quick and dirty” customization, and it's free.

And doesn't require to be a developer to be installed – it's targeted at the slightly experienced user. I think that installation wizards in PHP are really well constructed (although they often require full manual intervention and it would be hard to automate them via command-line scripts or similar) and I sometimes miss them in other languages (like Ruby, Python, Java), where heavy command-line and config editing is required, many times in non-obvious way. Most PHP applications contain the application AND a web-based configuration tool for such application

  • that's an obvious thumbs up.

The “install your own web software” approach is getting less useful nowadays, with a lot of great services around that let you do almost anything you'd do with a php software, yet it was very common until mid 2000s to setup your own hosting and your own software instance, and it retains a lot of charm on users that want to feel empowered, that want to “own” their website and make modifications to it (and maybe they want to say they're PHP programmers because what they did) – php can be edited and new pages appear live on the server!

Then, don't forget there're many PHP developers around the world. It's not an exotic language – it's quite mainstream, if you need a custom feature you won't go in mancraft-starvation mode. Developer quality in such world may be a problem, but if you can find somebody who's not a PHP-only developer, you should feel safe.

And remember: most software for PHP is content-centered. There's very little business logic in it. PHP works well as long as your application only does only CRUD. Advanced usages, anything that would require a sort of application server or persistent, long-running process, should be avoided.

So, PHP is simply great for its target. If you want to have your own website, with your own domain, where you can do whatever you like, you don't care for quality, uptime, or security, PHP is simply the right tool for you.

You can even use it for learning some programming basics, but PHP is gotcha-intensive, and you aren't a real PHP programmer if you don't know at least 75% of such gotchas :–). Good luck!

#Ollivander #php #programming #web