alan

delivery

How badly I wanted something like that?

The problem: Python for scripting

Beside programming and data science, I find Python to be a very useful glue language; I think it's great for shell replacement when bash/zsh scripts get too complex, but there's one caveat: as long as you can work with its standard library, you're in the sweet spot. As soon as you'd like to use an external dependency, that can be a problem, because if you don't want to contaminate your system with external dependencies, you'll either a) hope that your system packages a proper version for such library or b) start needing virtualenv and so on.

Both options are ok for manual development, a bit less ok if you're willing to deliver such scripts to multiple servers for automating some kind of process.

For sure, there're many options to fully package a Python executable – PyInstaller comes to my mind, but other exist. But then you've got a kind of “build process” for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines. Yes, I wouldn't do the same for “real” software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.

The solution: editable python scripts with isolated dependencies

So what? That's what I baked. Not a perfect solution, but a decent one. Just have python and pip on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.

This will install the dependencies in separate location in a temporary directory at first use, then reuse them when necessary.

So: just copy & paste the following snippets, edit the two USER SERVICEABLE sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run requests, so you can just delete the requirements, imports and requests call at the bottom if you don't need that.

#!/usr/bin/python3
import os
import sys
from tempfile import gettempdir, NamedTemporaryFile
import hashlib

# USER SERVICEABLE: paste here your requirements.txt
# the recommendation is to create a development virtualenv,
# install the deps with pip inside it, then do a `pip freeze`
# and paste the output here
REQUIREMENTS = """
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
"""
# USER SERVICEABLE end

def add_custom_site_packages_directory(raise_if_failure=True):
    digest = hashlib.sha256(REQUIREMENTS.encode("utf8")).hexdigest()
    dep_root = os.path.join(gettempdir(), "pyallinone_{}".format(digest))
    os.makedirs(dep_root, exist_ok=True)

    for dirpath, dirnames, filenames in os.walk(dep_root):
        if dirpath.endswith(os.path.sep + "site-packages"):
            # that's our dir!
            sys.path.insert(0, os.path.abspath(dirpath))
            return dep_root

    if raise_if_failure:
        raise ValueError("could not find our site-packages dir")

    return dep_root

dep_root = add_custom_site_packages_directory(False)

deps_installed = False

while True:
    try:
        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!
        import requests
        # USER SERVICEABLE end

        break
    except ImportError:
        if deps_installed:
            raise ValueError("Something was broken, could not install dependencies")
        try:
            from pip import main as pipmain
        except ImportError:
            from pip._internal import main as pipmain

        with NamedTemporaryFile() as req:
            req.write(REQUIREMENTS.encode("utf-8"))
            req.flush()
            pipmain(["install", "--prefix", dep_root, "--upgrade", "--no-cache-dir", "--no-deps", "-r", req.name])

        add_custom_site_packages_directory()
        deps_installed = True

# HERE you can start writing the actual code of your script

r = requests.get("https://www.google.com")
print(r.status_code)

How does this work?

  • It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using pip. Such subdir name is autogenerated depending on your requirements – so, if your requirements change, a new directory is employed.
  • Then, it adds such directory to your sys.path, allowing Python to find modules and packages there.
  • When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.

CAVEATS:

  • Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.
  • If your requirements change, nothing deletes files in your temp directory. But that's usually sweeped at system boot or by cronjobs, so it's not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).
  • If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There's no silver bullet for that in this recipe.

Alternative approaches:

Photo by Milan Popovic on Unsplash

#MostlyUnixish #python #packaging #delivery

This story has been boiling in my head since long; today I chose to (finally) publish it.

Long story short: in order to use a certain application, I should not need to understand how to use the language or its packaging ecosystem. Delivery and distribution is a relevant part of your app.

This does not apply to libraries, frameworks, or tools that are highly contextual for a certain language/environment ecosystem, and would be used only by a developer, in any case.

What do I mean? I'll start with an example. It's chronological, it's just the latest thing I came into; here's b2, the command line tool to get access to Backblaze B2 backup repositories. And those are its installation instructions:

GitHub\_-\_Backblaze\_B2\_Command\_Line\_Tool\_\_The\_command-line\_tool\_that\_gives\_easy\_access\_to\_all\_of\_the\_capabilities\_of\_B2\_Cloud\_Storage

  • What is pip?
  • Is such command safe to use?
  • Will it work in all situations? answers:
  • Python's package manager. It makes the b2 library available, and such library has the so-called b2 script, which exposes the b2 CLI executable.
  • possibly safe, but will install b2 within your global/per-user python dependencies, and may alter or install additional packages as dependencies, and such deps may be picked by other, unrelated software in the system;
  • it may require root to be run correctly.

Should every user of Backblaze B2 CLI know how to code in Python and understand how the Python environment works? I think that should not be necessary.

b2 is just an example, other famous tools – awscli and fpm come to my mind – just follow suit.

Most probably, if the tool you're downloading is written in C or C++, you'll probably expect to find a compiled binary, at least for some OSes and architectures; The same applies for Go, which makes binary creation for multiple platforms exceptionally easy. You should strive to provide some kind of equivalent piece of software for your users. .

But, even for tools written in Python or JavaScript (they look to be the ones that suffer most from the problem I'm describing), in most situations, the “right thing” to do is to provide a standalone binary, regardless of how it's written. You can provide standalone binaries for some archs (this is what borg does – it's a great backup utility written in Python), or you can rely on native or add-on packaging (additional repositories for Linux, choco for Windows, homebrew for Mac), like httpie. And, when you create such binary or package, you should make sure it's totally standalone (i.e. it doesn't alter global system state).

Why? Because leveraging packaging tools that were originally meant for developers of a language is brittle and risky, and can be hard to understand. You may end up compromising the system integrity if the wrong dependency slips into globally installed packages – this is especially true for Python, which is often used by many, many system tools. And, if all third-party apps did it that way, we could have continuous breakages. You can't assume yours is the only application installed in a certain system!

And, if you're writing an app, you can have full control of your execution environment and dependencies. You can choose a single Python or NodeJS version, and all of its dependencies. You don't need to support tons of variations!

In some situations, a CLI evolved out of a library. This is the case for pygments, a very nice syntax highlighting library. It's meant to be used as a library from inside python projects, but it also exposes a widely used pygmentize binary that is employed by many other tools. I'd love to have that CLI tool available as an independent, standalone package!

Final considerations:

  • As a software author, think whether you're building a library, a framework, or an application. If you're writing an application, think how you can deliver it to your final users, without the need for them to understand how you've written it.
  • If you're a maintainer for a package, think about what you're maintaining. If something is both a library and a command line tool, consider creating multiple (possibly independent) packages.

In the meantime: If you find a Python CLI tool that you'd like to use, and it uses the pip way and you don't to mess with system dependencies, I recommend pipsi. I don't know if something similar exists for Node or other environments.

FAQ: Q: But I'm a solo open source developer! Packaging this way would steal too much of my time! A: That's totally fine. Just make sure you don't suggest dangerous things to people. If you're using a developer-only approach, just state that as your target; but the tools I've written about above (b2 and awscli) certainly are of a different class.

Q: Any suggestion for easy packaging? A: I find homebrew very nice for Mac packaging and distribution. For Linux, I use fpm with docker – I've actually created a specific integration project fpm-within-docker. For Windows... no idea! I know about chocolatey and that's about it.

Photo by Malcolm Lightbody on Unsplash

#Ollivander #packaging #software #delivery