<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>dependencymanagement &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:dependencymanagement</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:52:51 +0000</pubDate>
    <item>
      <title>Python Packaging Woes: part 3</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-3</link>
      <description>&lt;![CDATA[A bit of wrap-up after part 1 and part 2&#xA;&#xA;Current state of the things&#xA;&#xA;I seem to have missed something: there&#39;s now a very active Python Packaging Authority which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.&#xA;&#xA;Setuptools and distribute got merged and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!&#xA;&#xA;So, rejoice! Things are getting better!&#xA;&#xA;In the meantime&#xA;&#xA;I recommend using Pip . Always try using requirements.txt, it&#39;s a good way to keep your versions&#xA;consistent and working.&#xA;&#xA;Use virtualenv for development purposes, and always test your package in a virtualenv with a disabled site-packages! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!&#xA;&#xA;In order to distribute a package, you&#39;ll need setuptools/distribute as well;&#xA;just do a source distribution and a wheel distribution (you&#39;re required to&#xA;install wheel beside setuptools); forget&#xA;about eggs, especially for eggs containing binary extensions. Your command line&#xA;should be something like:&#xA;&#xA;python setup.py clean register sdist bdist_wheel upload&#xA;&#xA;And let&#39;s keep our fingers crossed on a better state of things!&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>A bit of wrap-up after <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> and <a href="/alan/python-packaging-woes-part-2" rel="nofollow">part 2</a></p>

<h3 id="current-state-of-the-things">Current state of the things</h3>

<p>I seem to have missed something: there&#39;s now a very active <a href="https://github.com/pypa" rel="nofollow">Python</a> <a href="https://bitbucket.org/pypa" rel="nofollow">Packaging</a> <a href="https://groups.google.com/forum/#!forum/pypa-dev" rel="nofollow">Authority</a> which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.</p>

<p>Setuptools and distribute got <a href="https://mail.python.org/pipermail/distutils-sig/2013-March/020126.html" rel="nofollow">merged</a> and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!</p>

<p>So, rejoice! Things are getting better!</p>

<h3 id="in-the-meantime">In the meantime</h3>

<p>I recommend using <a href="http://www.pip-installer.org" rel="nofollow">Pip</a> . Always try using requirements.txt, it&#39;s a good way to keep your versions
consistent and working.</p>

<p>Use <a href="http://www.virtualenv.org" rel="nofollow">virtualenv</a> for development purposes, and <strong>always test your package in a virtualenv with a disabled site-packages</strong>! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!</p>

<p>In order to distribute a package, you&#39;ll need setuptools/distribute as well;
just do a source distribution and a wheel distribution (you&#39;re required to
install <a href="https://pypi.python.org/pypi/wheel" rel="nofollow">wheel</a> beside setuptools); forget
about eggs, especially for eggs containing binary extensions. Your command line
should be something like:</p>

<pre><code>python setup.py clean register sdist bdist_wheel upload
</code></pre>

<p>And let&#39;s keep our fingers crossed on a better state of things!</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-3</guid>
      <pubDate>Sun, 14 Sep 2014 20:13:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging Woes - part 2</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-2</link>
      <description>&lt;![CDATA[Check out part 1 for my thoughts on what packaging and dependency&#xA;management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!&#xA;&#xA;How Python messed it up&#xA;&#xA;A Diaspora of Tools&#xA;&#xA;There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.&#xA;&#xA;Distutils&#xA;&#xA;The original distutils is part of the Python standard library, but&#xA;it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,&#xA;its&#xA;requires&#xA;keyword seems to do&#xA;absolutely nothing else&#xA;than documenting requirements. Nothing will happen if you install a package requiring another one;&#xA;you still need to install dependencies manually.&#xA;&#xA;Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable - it changes&#xA;very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and&#xA;tired, and needs an overhaul now (more on that later).&#xA;&#xA;Setuptools&#xA;&#xA;The original setuptools package, which was created&#xA;by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained&#xA;right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.&#xA;&#xA;It includes easy\install which is probably the&#xA;first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)&#xA;on the command line, without a lot of manual intervention; also, if the downloaded package is&#xA;setuptools-aware and correctly specifies dependencies with&#xA;install\requires it will&#xA;install all transitive dependencies.&#xA;&#xA;It includes pkg\resources as well; that last&#xA;one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of&#xA;data which is installed in Python libraries, regardless of how it was installed - is it an egg? A&#xA;flat install? Did the install became part of the standard library? - don&#39;t worry: just ask for the&#xA;resource name and the package name, and pkg\resources will look up the file object for you and open&#xA;it.&#xA;&#xA;Unluckily the API was not so easy to use - that&#39;s something I&#39;m trying to address in&#xA;pydenji with its resourceloader URL-parsing based system - and many&#xA;people to handle resource in python just &#34;look for them&#34; from the current directory using&#xA;introspection by file. Too bad.&#xA;&#xA;Distribute&#xA;&#xA;In the end, setuptools had little maintenance applied and was forked; distribute is dropin&#xA;replacement which shares functions and dysfunctions of its ancestor.&#xA;&#xA;Recently it has been re-merged into setuptools.&#xA;&#xA;This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing&#xA;a specific-and-old version of setuptools, or you can find projects that know that setuptools is &#34;old&#34; and force&#xA;installing distribute (which is an issue since setuptools is now an actually updated distribute) :-/&#xA;&#xA;Distutils2&#xA;&#xA;The Next Big Thing in Python packaging, which has never happened so far - should have been available in Python 3.3, but&#xA;development seems to have stalled.&#xA;&#xA;Pip&#xA;&#xA;Pip is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works&#xA;as his author thinks it&#39;s well done - which is usually a good thing, since Ian Bicking knows what he&#39;s doing.&#xA;&#xA;Dependency Management&#xA;&#xA;Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.&#xA;Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s&#xA;a way to prevent the packages from being fetched from their homepage and just rely on pypi - see PEP 438&#xA;&#xA;One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which - of course - is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.&#xA;&#xA;Consequences&#xA;&#xA;It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,&#xA;or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment&#xA;is not easy.&#xA;&#xA;And different tools just don&#39;t play well together.&#xA;&#xA;What should be done?&#xA;&#xA;Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python&#xA;  community.&#xA;Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can&#xA;  work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not&#xA;  important if it&#39;s not part of the standard library.&#xA;Define best practices for packaging and dependencies (package layout, version numbers, metadata,&#xA;  uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why&#xA;  he does it.&#xA;Have a good doc linked from the official documentation. The current [packaging&#xA;  guide](http://guide.python-distribute.org/) is not complete and it&#39;s not linked from anything&#xA;  &#34;official&#34; - it&#39;s just another guide.&#xA;&#xA;But?&#xA;&#xA;There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :-)&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Check out <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> for my thoughts on what packaging and dependency
management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!</p>

<h2 id="how-python-messed-it-up">How Python messed it up</h2>

<h3 id="a-diaspora-of-tools">A Diaspora of Tools</h3>

<p>There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.</p>

<h4 id="distutils">Distutils</h4>

<p>The original <a href="http://docs.python.org/2/distutils/" rel="nofollow">distutils</a> is part of the Python standard library, but
it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,
its
<a href="http://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages" rel="nofollow">requires</a>
keyword seems to do
<a href="http://blog.doughellmann.com/2007/11/requiring-packages-with-distutils.html" rel="nofollow">absolutely nothing else</a>
than documenting requirements. Nothing will happen if you install a package requiring another one;
you still need to install dependencies manually.</p>

<p>Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable – it changes
very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and
tired, and needs an overhaul now (more on that later).</p>

<h4 id="setuptools">Setuptools</h4>

<p>The original <a href="http://peak.telecommunity.com/DevCenter/setuptools" rel="nofollow">setuptools</a> package, which was created
by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained
right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/EasyInstall" rel="nofollow">easy_install</a> which is probably the
first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)
on the command line, without a lot of manual intervention; also, if the downloaded package is
setuptools-aware and correctly specifies dependencies with
<a href="http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies" rel="nofollow">install_requires</a> it will
install all transitive dependencies.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/PkgResources" rel="nofollow">pkg_resources</a> as well; that last
one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of
data which is installed in Python libraries, regardless of how it was installed – is it an egg? A
flat install? Did the install became part of the standard library? – don&#39;t worry: just ask for the
resource name and the package name, and pkg_resources will look up the file object for you and open
it.</p>

<p>Unluckily the API was not so easy to use – that&#39;s something I&#39;m trying to address in
<a href="http://pydenji.franzoni.eu" rel="nofollow">pydenji</a> with its resourceloader URL-parsing based system – and many
people to handle resource in python just “look for them” from the current directory using
introspection by <code>__file__</code>. Too bad.</p>

<h4 id="distribute">Distribute</h4>

<p>In the end, setuptools had little maintenance applied and was forked; distribute is dropin
replacement which shares functions and dysfunctions of its ancestor.</p>

<p>Recently it has been re-merged into setuptools.</p>

<p>This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing
a specific-and-old version of setuptools, or you can find projects that know that setuptools is “old” and force
installing distribute (which is an issue since setuptools is now an actually updated distribute) :–/</p>

<h4 id="distutils2">Distutils2</h4>

<p>The Next Big Thing in Python packaging, which has never happened so far – should have been available in Python 3.3, but
development seems to have <a href="http://hg.python.org/distutils2" rel="nofollow">stalled</a>.</p>

<h4 id="pip">Pip</h4>

<p><a href="http://www.pip-installer.org/" rel="nofollow">Pip</a> is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works
as his author thinks it&#39;s well done – which is usually a good thing, since Ian Bicking knows what he&#39;s doing.</p>

<h3 id="dependency-management">Dependency Management</h3>

<p>Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.
Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s
a way to prevent the packages from being fetched from their homepage and just rely on pypi – see <a href="http://www.python.org/dev/peps/pep-0438/" rel="nofollow">PEP 438</a></p>

<p>One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which – of course – is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.</p>

<h3 id="consequences">Consequences</h3>

<p>It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,
or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment
is not easy.</p>

<p>And different tools just don&#39;t play well together.</p>

<h2 id="what-should-be-done">What should be done?</h2>
<ul><li>Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python
community.</li>
<li>Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can
work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not
important if it&#39;s not part of the standard library.</li>
<li>Define best practices for packaging and dependencies (package layout, version numbers, metadata,
uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why
he does it.</li>
<li>Have a good doc linked from the official documentation. The current <a href="http://guide.python-distribute.org/" rel="nofollow">packaging
guide</a> is not complete and it&#39;s not linked from anything
“official” – it&#39;s just another guide.</li></ul>

<h2 id="but">But?</h2>

<p>There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :–)</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-2</guid>
      <pubDate>Fri, 06 Sep 2013 20:09:00 +0000</pubDate>
    </item>
  </channel>
</rss>