<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>packaging &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:packaging</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:32:54 +0000</pubDate>
    <item>
      <title>Standalone, single-file, editable Python scripts WITH DEPENDENCIES</title>
      <link>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</link>
      <description>&lt;![CDATA[&#xA;&#xA;How badly I wanted something like that?&#xA;&#xA;The problem: Python for scripting&#xA;&#xA;Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing virtualenv and so on.&#xA;&#xA;Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.&#xA;&#xA;For sure, there&#39;re many options to fully package a Python executable - PyInstaller comes to my mind, but other exist. But then you&#39;ve got a kind of &#34;build process&#34; for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines. Yes, I wouldn&#39;t do the same for &#34;real&#34; software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.&#xA;&#xA;The solution: editable python scripts with isolated dependencies&#xA;&#xA;So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have python and pip on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.&#xA;&#xA;This will install the dependencies in separate location in a temporary directory at first use, then reuse them when necessary.&#xA;&#xA;So: just copy &amp; paste the following snippets, edit the two USER SERVICEABLE sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run requests, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.&#xA;&#xA;!/usr/bin/python3&#xA;import os&#xA;import sys&#xA;from tempfile import gettempdir, NamedTemporaryFile&#xA;import hashlib&#xA;&#xA;USER SERVICEABLE: paste here your requirements.txt&#xA;the recommendation is to create a development virtualenv,&#xA;install the deps with pip inside it, then do a pip freeze&#xA;and paste the output here&#xA;REQUIREMENTS = &#34;&#34;&#34;&#xA;certifi==2018.11.29&#xA;chardet==3.0.4&#xA;idna==2.8&#xA;requests==2.21.0&#xA;urllib3==1.24.1&#xA;&#34;&#34;&#34;&#xA;USER SERVICEABLE end&#xA;&#xA;def addcustomsitepackagesdirectory(raiseiffailure=True):&#xA;    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()&#xA;    deproot = os.path.join(gettempdir(), &#34;pyallinone{}&#34;.format(digest))&#xA;    os.makedirs(deproot, existok=True)&#xA;&#xA;    for dirpath, dirnames, filenames in os.walk(deproot):&#xA;        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):&#xA;            # that&#39;s our dir!&#xA;            sys.path.insert(0, os.path.abspath(dirpath))&#xA;            return deproot&#xA;&#xA;    if raiseiffailure:&#xA;        raise ValueError(&#34;could not find our site-packages dir&#34;)&#xA;&#xA;    return deproot&#xA;&#xA;deproot = addcustomsitepackagesdirectory(False)&#xA;&#xA;depsinstalled = False&#xA;&#xA;while True:&#xA;    try:&#xA;        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!&#xA;        import requests&#xA;        # USER SERVICEABLE end&#xA;&#xA;        break&#xA;    except ImportError:&#xA;        if depsinstalled:&#xA;            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)&#xA;        try:&#xA;            from pip import main as pipmain&#xA;        except ImportError:&#xA;            from pip.internal import main as pipmain&#xA;&#xA;        with NamedTemporaryFile() as req:&#xA;            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))&#xA;            req.flush()&#xA;            pipmain([&#34;install&#34;, &#34;--prefix&#34;, deproot, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])&#xA;&#xA;        addcustomsitepackagesdirectory()&#xA;        depsinstalled = True&#xA;&#xA;HERE you can start writing the actual code of your script&#xA;&#xA;r = requests.get(&#34;https://www.google.com&#34;)&#xA;print(r.statuscode)&#xA;&#xA;How does this work?&#xA;&#xA;It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using pip. Such subdir name is autogenerated depending on your requirements - so, if your requirements change, a new directory is employed.&#xA;Then, it adds such directory to your sys.path, allowing Python to find modules and packages there.&#xA;When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.&#xA;&#xA;CAVEATS:&#xA;&#xA;Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.&#xA;If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).&#xA;If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.&#xA;&#xA;Alternative approaches:&#xA;&#xA;If you already use uv and you assume it&#39;s already installed on your systems, there&#39;s a great article here on how to use uv for python scripts with dependencies&#xA;&#xA;Photo by Milan Popovic on Unsplash&#xA;&#xA;#MostlyUnixish #python #packaging #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/02/milan-popovic-674483-unsplash-1-.jpg" alt=""></p>

<p>How badly I wanted something like that?</p>

<h3 id="the-problem-python-for-scripting">The problem: Python for scripting</h3>

<p>Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing <a href="https://virtualenv.pypa.io/en/latest/" rel="nofollow">virtualenv</a> and so on.</p>

<p>Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.</p>

<p>For sure, there&#39;re many options to fully package a Python executable – <a href="https://www.pyinstaller.org/" rel="nofollow">PyInstaller</a> comes to my mind, but other exist. But then you&#39;ve got a kind of “build process” for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: <strong>I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines.</strong> Yes, I wouldn&#39;t do the same for “real” software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.</p>

<h3 id="the-solution-editable-python-scripts-with-isolated-dependencies">The solution: editable python scripts with isolated dependencies</h3>

<p>So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have <em>python</em> and <em>pip</em> on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.</p>

<p>This will install the dependencies in <strong>separate</strong> location in a temporary directory at first use, then reuse them when necessary.</p>

<p>So: just copy &amp; paste the following snippets, edit the two <code>USER SERVICEABLE</code> sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run <code>requests</code>, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.</p>

<pre><code class="language-python">#!/usr/bin/python3
import os
import sys
from tempfile import gettempdir, NamedTemporaryFile
import hashlib

# USER SERVICEABLE: paste here your requirements.txt
# the recommendation is to create a development virtualenv,
# install the deps with pip inside it, then do a `pip freeze`
# and paste the output here
REQUIREMENTS = &#34;&#34;&#34;
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
&#34;&#34;&#34;
# USER SERVICEABLE end

def add_custom_site_packages_directory(raise_if_failure=True):
    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()
    dep_root = os.path.join(gettempdir(), &#34;pyallinone_{}&#34;.format(digest))
    os.makedirs(dep_root, exist_ok=True)

    for dirpath, dirnames, filenames in os.walk(dep_root):
        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):
            # that&#39;s our dir!
            sys.path.insert(0, os.path.abspath(dirpath))
            return dep_root

    if raise_if_failure:
        raise ValueError(&#34;could not find our site-packages dir&#34;)

    return dep_root

dep_root = add_custom_site_packages_directory(False)

deps_installed = False

while True:
    try:
        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!
        import requests
        # USER SERVICEABLE end

        break
    except ImportError:
        if deps_installed:
            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)
        try:
            from pip import main as pipmain
        except ImportError:
            from pip._internal import main as pipmain

        with NamedTemporaryFile() as req:
            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))
            req.flush()
            pipmain([&#34;install&#34;, &#34;--prefix&#34;, dep_root, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])

        add_custom_site_packages_directory()
        deps_installed = True

# HERE you can start writing the actual code of your script

r = requests.get(&#34;https://www.google.com&#34;)
print(r.status_code)
</code></pre>

<h3 id="how-does-this-work">How does this work?</h3>
<ul><li>It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using <code>pip</code>. Such subdir name is autogenerated depending on your requirements – so, if your requirements change, a new directory is employed.</li>
<li>Then, it adds such directory to your <code>sys.path</code>, allowing Python to find modules and packages there.</li>
<li>When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.</li></ul>

<h3 id="caveats">CAVEATS:</h3>
<ul><li>Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.</li>
<li>If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).</li>
<li>If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.</li></ul>

<h3 id="alternative-approaches">Alternative approaches:</h3>
<ul><li>If you already use <code>uv</code> and you assume it&#39;s already installed on your systems, there&#39;s a <a href="https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/" rel="nofollow">great article here on how to use uv for python scripts with dependencies</a></li></ul>

<p>Photo by <a href="https://unsplash.com/@itsmiki5?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge" rel="nofollow">Milan Popovic</a> on Unsplash</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</guid>
      <pubDate>Tue, 19 Feb 2019 20:58:00 +0000</pubDate>
    </item>
    <item>
      <title>Application authors: please don&#39;t force users into your language or packaging details</title>
      <link>https://www.franzoni.eu/alan/software-authors-please-dont-force-me-into-your-packaging-woes</link>
      <description>&lt;![CDATA[&#xA;&#xA;This story has been boiling in my head since long; today I chose to (finally) publish it.&#xA;&#xA;Long story short: in order to use a certain application, I should not need to understand how to use the language or its packaging ecosystem. Delivery and distribution is a relevant part of your app.&#xA;&#xA;This does not apply to libraries, frameworks, or tools that are highly contextual for a certain language/environment ecosystem, and would be used only by a developer, in any case.&#xA;&#xA;What do I mean? I&#39;ll start with an example. It&#39;s chronological, it&#39;s just the latest thing I came into; here&#39;s b2, the command line tool to get access to Backblaze B2 backup repositories. And those are its installation instructions:&#xA;&#xA;GitHub\-\Backblaze\B2\Command\Line\Tool\\The\command-line\tool\that\gives\easy\access\to\all\of\the\capabilities\of\B2\Cloud\Storage&#xA;&#xA;What is pip?&#xA;Is such command safe to use?&#xA;Will it work in all situations?&#xA;  answers:&#xA;Python&#39;s package manager. It makes the b2 library available, and such library has the so-called b2 script, which exposes the b2 CLI executable.&#xA;possibly safe, but will install b2 within your global/per-user python dependencies, and may alter or install additional packages as dependencies, and such deps may be picked by other, unrelated software in the system;&#xA;it may require root to be run correctly.&#xA;&#xA;Should every user of Backblaze B2 CLI know how to code in Python and understand how the Python environment works? I think that should not be necessary.&#xA;&#xA;b2 is just an example, other famous tools - awscli and fpm come to my mind - just follow suit.&#xA;&#xA;Most probably, if the tool you&#39;re downloading is written in C or C++, you&#39;ll probably expect to find a compiled binary, at least for some OSes and architectures; The same applies for Go, which makes binary creation for multiple platforms exceptionally easy. You should strive to provide some kind of equivalent piece of software for your users. .&#xA;&#xA;But, even for tools written in Python or JavaScript (they look to be the ones that suffer most from the problem I&#39;m describing), in most situations, the &#34;right thing&#34; to do is to provide a standalone binary, regardless of how it&#39;s written. You can provide standalone binaries for some archs (this is what borg does - it&#39;s a great backup utility written in Python), or you can rely on native or add-on packaging (additional repositories for Linux, choco for Windows, homebrew for Mac), like httpie. And, when you create such binary or package, you should make sure it&#39;s totally standalone (i.e. it doesn&#39;t alter global system state).&#xA;&#xA;Why? Because leveraging packaging tools that were originally meant for developers of a language is brittle and risky, and can be hard to understand. You may end up compromising the system integrity if the wrong dependency slips into globally installed packages - this is especially true for Python, which is often used by many, many system tools. And, if all third-party apps did it that way, we could have continuous breakages. You can&#39;t assume yours is the only application installed in a certain system!&#xA;&#xA;And, if you&#39;re writing an app, you can have full control of your execution environment and dependencies. You can choose a single Python or NodeJS version, and all of its dependencies. You don&#39;t need to support tons of variations!&#xA;&#xA;In some situations, a CLI evolved out of a library. This is the case for pygments, a very nice syntax highlighting library. It&#39;s meant to be used as a library from inside python projects, but it also exposes a widely used pygmentize binary that is employed by many other tools. I&#39;d love to have that CLI tool available as an independent, standalone package!&#xA;&#xA;Final considerations:&#xA;&#xA;As a software author, think whether you&#39;re building a library, a framework, or an application. If you&#39;re writing an application, think how you can deliver it to your final users, without the need for them to understand how you&#39;ve written it.&#xA;If you&#39;re a maintainer for a package, think about what you&#39;re maintaining. If something is both a library and a command line tool, consider creating multiple (possibly independent) packages.&#xA;&#xA;In the meantime:&#xA;If you find a Python CLI tool that you&#39;d like to use, and it uses the pip way and you don&#39;t to mess with system dependencies, I recommend pipsi. I don&#39;t know if something similar exists for Node or other environments.&#xA;&#xA;FAQ:&#xA;Q: But I&#39;m a solo open source developer! Packaging this way would steal too much of my time!&#xA;A: That&#39;s totally fine. Just make sure you don&#39;t suggest dangerous things to people. If you&#39;re using a developer-only approach, just state that as your target; but the tools I&#39;ve written about above (b2 and awscli) certainly are of a different class.&#xA;&#xA;Q: Any suggestion for easy packaging?&#xA;A: I find homebrew very nice for Mac packaging and distribution. For Linux, I use fpm with docker - I&#39;ve actually created a specific integration project fpm-within-docker. For Windows... no idea! I know about chocolatey and that&#39;s about it.&#xA;&#xA;Photo by Malcolm Lightbody on Unsplash&#xA;&#xA;#Ollivander #packaging #software #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/01/malcolm-lightbody-1081282-unsplash.jpg" alt=""></p>

<p>This story has been boiling in my head since long; today I chose to (finally) publish it.</p>

<p>Long story short: <strong>in order to use a certain application, I should not need to understand how to use the language or its packaging ecosystem. Delivery and distribution is a relevant part of your app.</strong></p>

<p>This does <strong>not</strong> apply to libraries, frameworks, or tools that are highly contextual for a certain language/environment ecosystem, and would be used <strong>only</strong> by a developer, in any case.</p>

<p>What do I mean? I&#39;ll start with an example. It&#39;s chronological, it&#39;s just the latest thing I came into; here&#39;s <a href="https://github.com/Backblaze/B2_Command_Line_Tool" rel="nofollow">b2</a>, the <strong>command line tool</strong> to get access to Backblaze B2 backup repositories. And those are its installation instructions:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2019/01/GitHub_-_Backblaze_B2_Command_Line_Tool__The_command-line_tool_that_gives_easy_access_to_all_of_the_capabilities_of_B2_Cloud_Storage.png" alt="GitHub\_-\_Backblaze\_B2\_Command\_Line\_Tool\_\_The\_command-line\_tool\_that\_gives\_easy\_access\_to\_all\_of\_the\_capabilities\_of\_B2\_Cloud\_Storage"></p>
<ul><li>What is pip?</li>
<li>Is such command safe to use?</li>
<li>Will it work in all situations?
<em>answers:</em></li>
<li>Python&#39;s package manager. It makes the b2 library available, and such library has the so-called b2 script, which exposes the b2 CLI executable.</li>
<li>possibly safe, but will install b2 within your global/per-user python dependencies, and may alter or install additional packages as dependencies, and such deps may be picked by other, unrelated software in the system;</li>
<li>it may require root to be run correctly.</li></ul>

<p>Should every user of Backblaze B2 CLI know how to code in Python and understand how the Python environment works? I think that should <strong>not</strong> be necessary.</p>

<p>b2 is just an example, other famous tools – <a href="https://aws.amazon.com/cli/" rel="nofollow">awscli</a> and <a href="https://fpm.readthedocs.io/en/latest/installing.html" rel="nofollow">fpm</a> come to my mind – just follow suit.</p>

<p>Most probably, if the tool you&#39;re downloading is written in C or C++, you&#39;ll probably expect to find a compiled binary, at least for some OSes and architectures; The same applies for Go, which makes binary creation for multiple platforms exceptionally easy. <strong>You should strive to provide some kind of equivalent piece of software for your users.</strong> .</p>

<p>But, even for tools written in Python or JavaScript (they look to be the ones that suffer most from the problem I&#39;m describing), in most situations, the “right thing” to do is to provide a standalone binary, <strong>regardless of how it&#39;s written.</strong> You can provide standalone binaries for some archs (this is what <a href="https://github.com/borgbackup/borg" rel="nofollow">borg</a> does – it&#39;s a great backup utility written in Python), or you can rely on native or add-on packaging (additional repositories for Linux, choco for Windows, homebrew for Mac), like <a href="https://httpie.org/" rel="nofollow">httpie</a>. And, when you create such binary or package, you should make sure it&#39;s totally standalone (i.e. it doesn&#39;t alter global system state).</p>

<p>Why? Because leveraging packaging tools that were originally meant for developers of a language is <strong>brittle and risky</strong>, and can be hard to understand. You may end up compromising the system integrity if the wrong dependency slips into globally installed packages – this is especially true for Python, which is often used by many, many system tools. And, if all third-party apps did it that way, we could have continuous breakages. <strong>You can&#39;t assume yours is the only application installed in a certain system!</strong></p>

<p>And, if you&#39;re writing an app, you can have <strong>full control</strong> of your execution environment and dependencies. You can choose a single Python or NodeJS version, and all of its dependencies. You don&#39;t need to support tons of variations!</p>

<p>In some situations, a CLI evolved out of a library. This is the case for <a href="http://pygments.org/" rel="nofollow">pygments</a>, a very nice syntax highlighting library. It&#39;s meant to be used as a library from inside python projects, but it also exposes a widely used <a href="http://pygments.org/docs/cmdline/" rel="nofollow">pygmentize</a> binary that is employed by many other tools. I&#39;d love to have that CLI tool available as an independent, standalone package!</p>

<p>Final considerations:</p>
<ul><li>As a software author, think whether you&#39;re building a library, a framework, or an application. If you&#39;re writing an application, think how you can deliver it to your final users, without the need for them to understand how you&#39;ve written it.</li>
<li>If you&#39;re a maintainer for a package, think about what you&#39;re maintaining. If something is both a library and a command line tool, consider creating multiple (possibly independent) packages.</li></ul>

<p>In the meantime:
If you find a Python CLI tool that you&#39;d like to use, and it uses the <em>pip way</em> and you don&#39;t to mess with system dependencies, I recommend <a href="https://github.com/mitsuhiko/pipsi" rel="nofollow">pipsi</a>. I don&#39;t know if something similar exists for Node or other environments.</p>

<p>FAQ:
Q: But I&#39;m a solo open source developer! Packaging this way would steal too much of my time!
A: That&#39;s totally fine. Just make sure you don&#39;t suggest dangerous things to people. If you&#39;re using a developer-only approach, just state that as your target; but the tools I&#39;ve written about above (b2 and awscli) certainly are of a different class.</p>

<p>Q: Any suggestion for easy packaging?
A: I find <a href="https://brew.sh/" rel="nofollow">homebrew</a> very nice for Mac packaging and distribution. For Linux, I use <a href="https://github.com/jordansissel/fpm" rel="nofollow">fpm</a> with <a href="https://www.docker.com/" rel="nofollow">docker</a> – I&#39;ve actually created a specific integration project <a href="https://github.com/alanfranz/fpm-within-docker" rel="nofollow">fpm-within-docker</a>. For Windows... no idea! I know about <a href="https://chocolatey.org/" rel="nofollow">chocolatey</a> and that&#39;s about it.</p>

<p>Photo by Malcolm Lightbody on <a href="https://unsplash.com/photos/401OD83Ke6o" rel="nofollow">Unsplash</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:software" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">software</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/software-authors-please-dont-force-me-into-your-packaging-woes</guid>
      <pubDate>Thu, 10 Jan 2019 17:10:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging Woes: part 3</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-3</link>
      <description>&lt;![CDATA[A bit of wrap-up after part 1 and part 2&#xA;&#xA;Current state of the things&#xA;&#xA;I seem to have missed something: there&#39;s now a very active Python Packaging Authority which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.&#xA;&#xA;Setuptools and distribute got merged and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!&#xA;&#xA;So, rejoice! Things are getting better!&#xA;&#xA;In the meantime&#xA;&#xA;I recommend using Pip . Always try using requirements.txt, it&#39;s a good way to keep your versions&#xA;consistent and working.&#xA;&#xA;Use virtualenv for development purposes, and always test your package in a virtualenv with a disabled site-packages! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!&#xA;&#xA;In order to distribute a package, you&#39;ll need setuptools/distribute as well;&#xA;just do a source distribution and a wheel distribution (you&#39;re required to&#xA;install wheel beside setuptools); forget&#xA;about eggs, especially for eggs containing binary extensions. Your command line&#xA;should be something like:&#xA;&#xA;python setup.py clean register sdist bdist_wheel upload&#xA;&#xA;And let&#39;s keep our fingers crossed on a better state of things!&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>A bit of wrap-up after <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> and <a href="/alan/python-packaging-woes-part-2" rel="nofollow">part 2</a></p>

<h3 id="current-state-of-the-things">Current state of the things</h3>

<p>I seem to have missed something: there&#39;s now a very active <a href="https://github.com/pypa" rel="nofollow">Python</a> <a href="https://bitbucket.org/pypa" rel="nofollow">Packaging</a> <a href="https://groups.google.com/forum/#!forum/pypa-dev" rel="nofollow">Authority</a> which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.</p>

<p>Setuptools and distribute got <a href="https://mail.python.org/pipermail/distutils-sig/2013-March/020126.html" rel="nofollow">merged</a> and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!</p>

<p>So, rejoice! Things are getting better!</p>

<h3 id="in-the-meantime">In the meantime</h3>

<p>I recommend using <a href="http://www.pip-installer.org" rel="nofollow">Pip</a> . Always try using requirements.txt, it&#39;s a good way to keep your versions
consistent and working.</p>

<p>Use <a href="http://www.virtualenv.org" rel="nofollow">virtualenv</a> for development purposes, and <strong>always test your package in a virtualenv with a disabled site-packages</strong>! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!</p>

<p>In order to distribute a package, you&#39;ll need setuptools/distribute as well;
just do a source distribution and a wheel distribution (you&#39;re required to
install <a href="https://pypi.python.org/pypi/wheel" rel="nofollow">wheel</a> beside setuptools); forget
about eggs, especially for eggs containing binary extensions. Your command line
should be something like:</p>

<pre><code>python setup.py clean register sdist bdist_wheel upload
</code></pre>

<p>And let&#39;s keep our fingers crossed on a better state of things!</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-3</guid>
      <pubDate>Sun, 14 Sep 2014 20:13:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging Woes - part 2</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-2</link>
      <description>&lt;![CDATA[Check out part 1 for my thoughts on what packaging and dependency&#xA;management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!&#xA;&#xA;How Python messed it up&#xA;&#xA;A Diaspora of Tools&#xA;&#xA;There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.&#xA;&#xA;Distutils&#xA;&#xA;The original distutils is part of the Python standard library, but&#xA;it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,&#xA;its&#xA;requires&#xA;keyword seems to do&#xA;absolutely nothing else&#xA;than documenting requirements. Nothing will happen if you install a package requiring another one;&#xA;you still need to install dependencies manually.&#xA;&#xA;Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable - it changes&#xA;very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and&#xA;tired, and needs an overhaul now (more on that later).&#xA;&#xA;Setuptools&#xA;&#xA;The original setuptools package, which was created&#xA;by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained&#xA;right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.&#xA;&#xA;It includes easy\install which is probably the&#xA;first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)&#xA;on the command line, without a lot of manual intervention; also, if the downloaded package is&#xA;setuptools-aware and correctly specifies dependencies with&#xA;install\requires it will&#xA;install all transitive dependencies.&#xA;&#xA;It includes pkg\resources as well; that last&#xA;one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of&#xA;data which is installed in Python libraries, regardless of how it was installed - is it an egg? A&#xA;flat install? Did the install became part of the standard library? - don&#39;t worry: just ask for the&#xA;resource name and the package name, and pkg\resources will look up the file object for you and open&#xA;it.&#xA;&#xA;Unluckily the API was not so easy to use - that&#39;s something I&#39;m trying to address in&#xA;pydenji with its resourceloader URL-parsing based system - and many&#xA;people to handle resource in python just &#34;look for them&#34; from the current directory using&#xA;introspection by file. Too bad.&#xA;&#xA;Distribute&#xA;&#xA;In the end, setuptools had little maintenance applied and was forked; distribute is dropin&#xA;replacement which shares functions and dysfunctions of its ancestor.&#xA;&#xA;Recently it has been re-merged into setuptools.&#xA;&#xA;This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing&#xA;a specific-and-old version of setuptools, or you can find projects that know that setuptools is &#34;old&#34; and force&#xA;installing distribute (which is an issue since setuptools is now an actually updated distribute) :-/&#xA;&#xA;Distutils2&#xA;&#xA;The Next Big Thing in Python packaging, which has never happened so far - should have been available in Python 3.3, but&#xA;development seems to have stalled.&#xA;&#xA;Pip&#xA;&#xA;Pip is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works&#xA;as his author thinks it&#39;s well done - which is usually a good thing, since Ian Bicking knows what he&#39;s doing.&#xA;&#xA;Dependency Management&#xA;&#xA;Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.&#xA;Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s&#xA;a way to prevent the packages from being fetched from their homepage and just rely on pypi - see PEP 438&#xA;&#xA;One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which - of course - is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.&#xA;&#xA;Consequences&#xA;&#xA;It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,&#xA;or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment&#xA;is not easy.&#xA;&#xA;And different tools just don&#39;t play well together.&#xA;&#xA;What should be done?&#xA;&#xA;Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python&#xA;  community.&#xA;Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can&#xA;  work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not&#xA;  important if it&#39;s not part of the standard library.&#xA;Define best practices for packaging and dependencies (package layout, version numbers, metadata,&#xA;  uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why&#xA;  he does it.&#xA;Have a good doc linked from the official documentation. The current [packaging&#xA;  guide](http://guide.python-distribute.org/) is not complete and it&#39;s not linked from anything&#xA;  &#34;official&#34; - it&#39;s just another guide.&#xA;&#xA;But?&#xA;&#xA;There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :-)&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Check out <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> for my thoughts on what packaging and dependency
management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!</p>

<h2 id="how-python-messed-it-up">How Python messed it up</h2>

<h3 id="a-diaspora-of-tools">A Diaspora of Tools</h3>

<p>There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.</p>

<h4 id="distutils">Distutils</h4>

<p>The original <a href="http://docs.python.org/2/distutils/" rel="nofollow">distutils</a> is part of the Python standard library, but
it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,
its
<a href="http://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages" rel="nofollow">requires</a>
keyword seems to do
<a href="http://blog.doughellmann.com/2007/11/requiring-packages-with-distutils.html" rel="nofollow">absolutely nothing else</a>
than documenting requirements. Nothing will happen if you install a package requiring another one;
you still need to install dependencies manually.</p>

<p>Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable – it changes
very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and
tired, and needs an overhaul now (more on that later).</p>

<h4 id="setuptools">Setuptools</h4>

<p>The original <a href="http://peak.telecommunity.com/DevCenter/setuptools" rel="nofollow">setuptools</a> package, which was created
by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained
right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/EasyInstall" rel="nofollow">easy_install</a> which is probably the
first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)
on the command line, without a lot of manual intervention; also, if the downloaded package is
setuptools-aware and correctly specifies dependencies with
<a href="http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies" rel="nofollow">install_requires</a> it will
install all transitive dependencies.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/PkgResources" rel="nofollow">pkg_resources</a> as well; that last
one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of
data which is installed in Python libraries, regardless of how it was installed – is it an egg? A
flat install? Did the install became part of the standard library? – don&#39;t worry: just ask for the
resource name and the package name, and pkg_resources will look up the file object for you and open
it.</p>

<p>Unluckily the API was not so easy to use – that&#39;s something I&#39;m trying to address in
<a href="http://pydenji.franzoni.eu" rel="nofollow">pydenji</a> with its resourceloader URL-parsing based system – and many
people to handle resource in python just “look for them” from the current directory using
introspection by <code>__file__</code>. Too bad.</p>

<h4 id="distribute">Distribute</h4>

<p>In the end, setuptools had little maintenance applied and was forked; distribute is dropin
replacement which shares functions and dysfunctions of its ancestor.</p>

<p>Recently it has been re-merged into setuptools.</p>

<p>This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing
a specific-and-old version of setuptools, or you can find projects that know that setuptools is “old” and force
installing distribute (which is an issue since setuptools is now an actually updated distribute) :–/</p>

<h4 id="distutils2">Distutils2</h4>

<p>The Next Big Thing in Python packaging, which has never happened so far – should have been available in Python 3.3, but
development seems to have <a href="http://hg.python.org/distutils2" rel="nofollow">stalled</a>.</p>

<h4 id="pip">Pip</h4>

<p><a href="http://www.pip-installer.org/" rel="nofollow">Pip</a> is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works
as his author thinks it&#39;s well done – which is usually a good thing, since Ian Bicking knows what he&#39;s doing.</p>

<h3 id="dependency-management">Dependency Management</h3>

<p>Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.
Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s
a way to prevent the packages from being fetched from their homepage and just rely on pypi – see <a href="http://www.python.org/dev/peps/pep-0438/" rel="nofollow">PEP 438</a></p>

<p>One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which – of course – is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.</p>

<h3 id="consequences">Consequences</h3>

<p>It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,
or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment
is not easy.</p>

<p>And different tools just don&#39;t play well together.</p>

<h2 id="what-should-be-done">What should be done?</h2>
<ul><li>Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python
community.</li>
<li>Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can
work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not
important if it&#39;s not part of the standard library.</li>
<li>Define best practices for packaging and dependencies (package layout, version numbers, metadata,
uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why
he does it.</li>
<li>Have a good doc linked from the official documentation. The current <a href="http://guide.python-distribute.org/" rel="nofollow">packaging
guide</a> is not complete and it&#39;s not linked from anything
“official” – it&#39;s just another guide.</li></ul>

<h2 id="but">But?</h2>

<p>There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :–)</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-2</guid>
      <pubDate>Fri, 06 Sep 2013 20:09:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging woes - part 1</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-1</link>
      <description>&lt;![CDATA[This is a spinoff from my Europython 2012 talk (video, slides).&#xA;&#xA;Some days ago I got involved in a brief discussion on the topic, so here&#39;s what I think about Python packaging and dependency management: it&#39;s in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.&#xA;&#xA;Why? Let&#39;s see.&#xA;&#xA;The Packaging Conspiracy&#xA;&#xA;Aim&#xA;&#xA;No, it&#39;s not a conspiracy to force you to learn something you don&#39;t care about. The whole point of packaging is to make code reuse easy enough, both for the developer&#39;s own sake (e.g. shared code between different projects) and for sharing it with the public.&#xA;&#xA;Packaging usually involves dependency management; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don&#39;t just release your code in &#34;in the dark&#34;: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.&#xA;&#xA;It&#39;s an overkill!&#xA;&#xA;Some developers think all of this is a waste of their time.&#xA;&#xA;D: Hey Alan, are you really saying say I can&#39;t just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?&#xA;&#xA;A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you&#39;ve fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, you&#39;re doing by hand the work that a good dependency management tool is supposed to do. If you like that.&#xA;&#xA;D: OK, ok. I got it. So now I&#39;ve added my dependencies. But what the heck, does the version really matter? I don&#39;t care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won&#39;t break my code.&#xA;&#xA;A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it&#39;s time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.&#xA;&#xA;D: OK, ok, now I&#39;ve set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?&#xA;&#xA;A: Try doing something meaningful, and stick to your plan. You can pick your own versioning policy. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something &#34;emotional&#34;. If you perform small incremental changes on your codebase - a-la-Chrome - you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.&#xA;&#xA;D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!&#xA;&#xA;A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.&#xA;&#xA;D: YEAH! I&#39;ve just released version 0.0.2 to the public. Now I&#39;m deleting 0.0.1, right?&#xA;&#xA;A: No. Never. Ideally, unless you must - by must I mean something like &#34;there&#39;s a password for my bank account in the source code&#34; or &#34;I mistakenly copypasted code for which I hadn&#39;t the copyright&#34; - you should never, ever, ever, remove a released version. You never know who&#39;s using it - if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn&#39;t care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as &#34;deprecated&#34;, &#34;bugged&#34;, or maybe &#34;donut&#34; if they contain a huge security hole.&#xA;&#xA;D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it&#39;s my first daughter and she was born in the very day of such release.&#xA;&#xA;A: I&#39;m happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, &#34;lexicographical&#34; is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means &#34;lower&#34;, so that 2.0.0beta is \&lt; 2.0.0, but that&#39;s not always the case - a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.&#39;&#39; is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.&#xA;&#xA;D: There&#39;re a lot of things to remember about dependency management and packaging. Should I learn everything by heart?&#xA;&#xA;A: What the heck, no. Your packaging tool should guide you in such practices and prevent you from going stray unless it&#39;s what you really mean to do! That why I&#39;m complaining in the first place about the way Python does packaging!&#xA;&#xA;In part 2 I dig into Python-specific packaging issues.&#xA;&#xA;#Ollivander #python #packaging&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>This is a spinoff from my <a href="https://ep2012.europython.eu/conference/talks/language-alone-wont-pay-your-bills" rel="nofollow">Europython 2012 talk</a> (<a href="http://www.youtube.com/watch?v=JlNYO4zUDAQ" rel="nofollow">video</a>, <a href="https://ep2013.europython.eu/media/conference/slides/language-alone-wont-pay-your-bills.pdf" rel="nofollow">slides</a>).</p>

<p>Some days ago I got involved in a brief <a href="https://twitter.com/hpk42/status/292599238095933440" rel="nofollow">discussion</a> on the topic, <strong>so here&#39;s what I think about Python packaging and dependency management: it&#39;s in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.</strong></p>

<p>Why? Let&#39;s see.</p>

<h2 id="the-packaging-conspiracy">The Packaging Conspiracy</h2>

<h3 id="aim">Aim</h3>

<p>No, it&#39;s not a conspiracy to force you to learn something you don&#39;t care about. The whole point of packaging is to make code reuse easy enough, both for the developer&#39;s own sake (e.g. shared code between different projects) and for sharing it with the public.</p>

<p><strong>Packaging usually involves dependency management</strong>; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don&#39;t just release your code in “in the dark”: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.</p>

<h3 id="it-s-an-overkill">It&#39;s an overkill!</h3>

<p>Some developers think all of this is a waste of their time.</p>

<p><em>D: Hey Alan, are you really saying say I can&#39;t just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?</em></p>

<p>A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you&#39;ve fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, <strong>you&#39;re doing by hand the work that a good dependency management tool is supposed to do</strong>. If you like that.</p>

<p><em>D: OK, ok. I got it. So now I&#39;ve added my dependencies. But what the heck, does the version really matter? I don&#39;t care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won&#39;t break my code.</em></p>

<p>A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it&#39;s time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.</p>

<p><em>D: OK, ok, now I&#39;ve set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?</em></p>

<p>A: Try doing something meaningful, and stick to your plan. <strong>You can pick your own versioning policy</strong>. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something “emotional”. If you perform small incremental changes on your codebase – <a href="http://blog.chromium.org/2010/07/release-early-release-often.html" rel="nofollow">a-la-Chrome</a> – you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.</p>

<p><em>D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!</em></p>

<p>A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.</p>

<p><em>D: YEAH! I&#39;ve just released version 0.0.2 to the public. Now I&#39;m deleting 0.0.1, right?</em></p>

<p>A: No. Never. Ideally, unless you must – by must I mean something like “there&#39;s a password for my bank account in the source code” or “I mistakenly copypasted code for which I hadn&#39;t the copyright” – <strong>you should never, ever, ever, remove a released version</strong>. You never know who&#39;s using it – if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn&#39;t care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as “deprecated”, “bugged”, or maybe “donut” if they contain a huge security hole.</p>

<p><em>D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it&#39;s my first daughter and she was born in the very day of such release.</em></p>

<p>A: I&#39;m happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, “lexicographical” is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means “lower”, so that 2.0.0beta is &lt; 2.0.0, but that&#39;s not always the case – a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.&#39;&#39; is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.</p>

<p><em>D: There&#39;re a lot of things to remember about dependency management and packaging. Should I learn everything by heart?</em></p>

<p>A: What the heck, no. <strong>Your packaging tool should guide you in such practices</strong> and prevent you from going stray unless it&#39;s what you really mean to do! That why I&#39;m complaining in the first place about the way Python does packaging!</p>

<p>In <a href="/alan/python-packaging-woes-part-2" rel="nofollow">part 2</a> I dig into Python-specific packaging issues.</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-1</guid>
      <pubDate>Mon, 21 Jan 2013 20:33:00 +0000</pubDate>
    </item>
  </channel>
</rss>