<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>python &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:python</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:32:51 +0000</pubDate>
    <item>
      <title>Standalone, single-file, editable Python scripts WITH DEPENDENCIES</title>
      <link>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</link>
      <description>&lt;![CDATA[&#xA;&#xA;How badly I wanted something like that?&#xA;&#xA;The problem: Python for scripting&#xA;&#xA;Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing virtualenv and so on.&#xA;&#xA;Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.&#xA;&#xA;For sure, there&#39;re many options to fully package a Python executable - PyInstaller comes to my mind, but other exist. But then you&#39;ve got a kind of &#34;build process&#34; for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines. Yes, I wouldn&#39;t do the same for &#34;real&#34; software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.&#xA;&#xA;The solution: editable python scripts with isolated dependencies&#xA;&#xA;So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have python and pip on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.&#xA;&#xA;This will install the dependencies in separate location in a temporary directory at first use, then reuse them when necessary.&#xA;&#xA;So: just copy &amp; paste the following snippets, edit the two USER SERVICEABLE sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run requests, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.&#xA;&#xA;!/usr/bin/python3&#xA;import os&#xA;import sys&#xA;from tempfile import gettempdir, NamedTemporaryFile&#xA;import hashlib&#xA;&#xA;USER SERVICEABLE: paste here your requirements.txt&#xA;the recommendation is to create a development virtualenv,&#xA;install the deps with pip inside it, then do a pip freeze&#xA;and paste the output here&#xA;REQUIREMENTS = &#34;&#34;&#34;&#xA;certifi==2018.11.29&#xA;chardet==3.0.4&#xA;idna==2.8&#xA;requests==2.21.0&#xA;urllib3==1.24.1&#xA;&#34;&#34;&#34;&#xA;USER SERVICEABLE end&#xA;&#xA;def addcustomsitepackagesdirectory(raiseiffailure=True):&#xA;    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()&#xA;    deproot = os.path.join(gettempdir(), &#34;pyallinone{}&#34;.format(digest))&#xA;    os.makedirs(deproot, existok=True)&#xA;&#xA;    for dirpath, dirnames, filenames in os.walk(deproot):&#xA;        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):&#xA;            # that&#39;s our dir!&#xA;            sys.path.insert(0, os.path.abspath(dirpath))&#xA;            return deproot&#xA;&#xA;    if raiseiffailure:&#xA;        raise ValueError(&#34;could not find our site-packages dir&#34;)&#xA;&#xA;    return deproot&#xA;&#xA;deproot = addcustomsitepackagesdirectory(False)&#xA;&#xA;depsinstalled = False&#xA;&#xA;while True:&#xA;    try:&#xA;        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!&#xA;        import requests&#xA;        # USER SERVICEABLE end&#xA;&#xA;        break&#xA;    except ImportError:&#xA;        if depsinstalled:&#xA;            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)&#xA;        try:&#xA;            from pip import main as pipmain&#xA;        except ImportError:&#xA;            from pip.internal import main as pipmain&#xA;&#xA;        with NamedTemporaryFile() as req:&#xA;            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))&#xA;            req.flush()&#xA;            pipmain([&#34;install&#34;, &#34;--prefix&#34;, deproot, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])&#xA;&#xA;        addcustomsitepackagesdirectory()&#xA;        depsinstalled = True&#xA;&#xA;HERE you can start writing the actual code of your script&#xA;&#xA;r = requests.get(&#34;https://www.google.com&#34;)&#xA;print(r.statuscode)&#xA;&#xA;How does this work?&#xA;&#xA;It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using pip. Such subdir name is autogenerated depending on your requirements - so, if your requirements change, a new directory is employed.&#xA;Then, it adds such directory to your sys.path, allowing Python to find modules and packages there.&#xA;When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.&#xA;&#xA;CAVEATS:&#xA;&#xA;Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.&#xA;If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).&#xA;If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.&#xA;&#xA;Alternative approaches:&#xA;&#xA;If you already use uv and you assume it&#39;s already installed on your systems, there&#39;s a great article here on how to use uv for python scripts with dependencies&#xA;&#xA;Photo by Milan Popovic on Unsplash&#xA;&#xA;#MostlyUnixish #python #packaging #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/02/milan-popovic-674483-unsplash-1-.jpg" alt=""></p>

<p>How badly I wanted something like that?</p>

<h3 id="the-problem-python-for-scripting">The problem: Python for scripting</h3>

<p>Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing <a href="https://virtualenv.pypa.io/en/latest/" rel="nofollow">virtualenv</a> and so on.</p>

<p>Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.</p>

<p>For sure, there&#39;re many options to fully package a Python executable – <a href="https://www.pyinstaller.org/" rel="nofollow">PyInstaller</a> comes to my mind, but other exist. But then you&#39;ve got a kind of “build process” for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: <strong>I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines.</strong> Yes, I wouldn&#39;t do the same for “real” software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.</p>

<h3 id="the-solution-editable-python-scripts-with-isolated-dependencies">The solution: editable python scripts with isolated dependencies</h3>

<p>So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have <em>python</em> and <em>pip</em> on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.</p>

<p>This will install the dependencies in <strong>separate</strong> location in a temporary directory at first use, then reuse them when necessary.</p>

<p>So: just copy &amp; paste the following snippets, edit the two <code>USER SERVICEABLE</code> sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run <code>requests</code>, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.</p>

<pre><code class="language-python">#!/usr/bin/python3
import os
import sys
from tempfile import gettempdir, NamedTemporaryFile
import hashlib

# USER SERVICEABLE: paste here your requirements.txt
# the recommendation is to create a development virtualenv,
# install the deps with pip inside it, then do a `pip freeze`
# and paste the output here
REQUIREMENTS = &#34;&#34;&#34;
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
&#34;&#34;&#34;
# USER SERVICEABLE end

def add_custom_site_packages_directory(raise_if_failure=True):
    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()
    dep_root = os.path.join(gettempdir(), &#34;pyallinone_{}&#34;.format(digest))
    os.makedirs(dep_root, exist_ok=True)

    for dirpath, dirnames, filenames in os.walk(dep_root):
        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):
            # that&#39;s our dir!
            sys.path.insert(0, os.path.abspath(dirpath))
            return dep_root

    if raise_if_failure:
        raise ValueError(&#34;could not find our site-packages dir&#34;)

    return dep_root

dep_root = add_custom_site_packages_directory(False)

deps_installed = False

while True:
    try:
        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!
        import requests
        # USER SERVICEABLE end

        break
    except ImportError:
        if deps_installed:
            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)
        try:
            from pip import main as pipmain
        except ImportError:
            from pip._internal import main as pipmain

        with NamedTemporaryFile() as req:
            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))
            req.flush()
            pipmain([&#34;install&#34;, &#34;--prefix&#34;, dep_root, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])

        add_custom_site_packages_directory()
        deps_installed = True

# HERE you can start writing the actual code of your script

r = requests.get(&#34;https://www.google.com&#34;)
print(r.status_code)
</code></pre>

<h3 id="how-does-this-work">How does this work?</h3>
<ul><li>It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using <code>pip</code>. Such subdir name is autogenerated depending on your requirements – so, if your requirements change, a new directory is employed.</li>
<li>Then, it adds such directory to your <code>sys.path</code>, allowing Python to find modules and packages there.</li>
<li>When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.</li></ul>

<h3 id="caveats">CAVEATS:</h3>
<ul><li>Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.</li>
<li>If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).</li>
<li>If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.</li></ul>

<h3 id="alternative-approaches">Alternative approaches:</h3>
<ul><li>If you already use <code>uv</code> and you assume it&#39;s already installed on your systems, there&#39;s a <a href="https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/" rel="nofollow">great article here on how to use uv for python scripts with dependencies</a></li></ul>

<p>Photo by <a href="https://unsplash.com/@itsmiki5?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge" rel="nofollow">Milan Popovic</a> on Unsplash</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</guid>
      <pubDate>Tue, 19 Feb 2019 20:58:00 +0000</pubDate>
    </item>
    <item>
      <title>Command line data crunching with Python</title>
      <link>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</link>
      <description>&lt;![CDATA[&#xA;&#xA;Every time I&#39;m doing some data crunching on the command line, I find myself juggling between sed, awk, sort, uniq, etc. While I like the UNIX way of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.&#xA;&#xA;I know that Perl and Ruby support implicit loops / prints - see this and that. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.&#xA;&#xA;On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy - the -c switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:&#xA;&#xA;$ python -c &#39;import sys;for x in sys.stdin:print x&#39;&#xA;&#xA;won&#39;t &#34;just work&#34;:&#xA;&#xA;  File &#34;string&#34;, line 1&#xA;    import sys;for x in [1,5]:    print x;print x&#xA;                 ^&#xA;SyntaxError: invalid syntax&#xA;&#xA;But: there&#39;s a bash feature to interpret escape sequences in single-quoted strings, so this will work fine:&#xA;&#xA;$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;hello&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.&#xA;&#xA;AN IMPORTANT NOTE: if you&#39;re treating non-ascii data, I suggest you set the PYTHONIOENCODING variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:&#xA;&#xA;echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;ààà&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend this book enough.&#xA;&#xA;Photo by Daniel Cheung on Unsplash&#xA;&#xA;#Ollivander #python #datacrunching #MostlyUnixish&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2018/02/daniel-cheung-129839-1-.jpg" alt=""></p>

<p>Every time I&#39;m doing some data crunching on the command line, I find myself juggling between <em>sed, awk, sort, uniq, etc</em>. While I like <em>the UNIX way</em> of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.</p>

<p>I know that Perl and Ruby support implicit loops / prints – see <a href="http://www.wellho.net/resources/ex.php4?item=p210/spot" rel="nofollow">this</a> and <a href="http://www.wellho.net/resources/ex.php4?item=r110/ruby_awk.rb" rel="nofollow">that</a>. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.</p>

<p>On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy – the <code>-c</code> switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:</p>

<pre><code>$ python -c &#39;import sys;for x in sys.stdin:print x&#39;
</code></pre>

<p>won&#39;t “just work”:</p>

<pre><code>  File &#34;&lt;string&gt;&#34;, line 1
    import sys;for x in [1,5]:    print x;print x
                 ^
SyntaxError: invalid syntax
</code></pre>

<p>But: there&#39;s a <a href="https://stackoverflow.com/questions/11966312/how-does-the-leading-dollar-sign-affect-single-quotes-in-bash" rel="nofollow">bash feature to interpret escape sequences in single-quoted strings</a>, so this will work fine:</p>

<pre><code>$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
hello
world
this
is
me
</code></pre>

<p>I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.</p>

<p><strong>AN IMPORTANT NOTE:</strong> if you&#39;re treating non-ascii data, I suggest you set the <em>PYTHONIOENCODING</em> variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:</p>

<pre><code>echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
ààà
world
this
is
me
</code></pre>

<p>Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend <a href="https://www.datascienceatthecommandline.com/" rel="nofollow">this book</a> enough.</p>

<p><em>Photo by Daniel Cheung on <a href="https://unsplash.com/photos/cPF2nlWcMY4" rel="nofollow">Unsplash</a></em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:datacrunching" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">datacrunching</span></a> <a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</guid>
      <pubDate>Wed, 14 Feb 2018 14:44:51 +0000</pubDate>
    </item>
    <item>
      <title>Python: discover the type of an object</title>
      <link>https://www.franzoni.eu/alan/python-discover-the-type-of-an-object</link>
      <description>&lt;![CDATA[This seems a simple enough question:&#xA;&#xA;How can I discover the type of any given Python object?&#xA;&#xA;Suppose Python 2.7 (Python 3 is a different beast on this topic), and try implementing such function:&#xA;&#xA;def discovertypeofobject(obj):&#xA;    &#34;&#34;&#34;&#xA;    Args:&#xA;        obj: Any python object&#xA;&#xA;    Returns:&#xA;        the type of input object&#xA;    &#34;&#34;&#34;&#xA;    ....&#xA;&#xA;When you&#39;re done, scroll the page way down...&#xA;&#xA;br&#xA;br&#xA;br&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;brbr&#xA;br&#xA;&#xA;Now, try the function you&#39;ve wrote with this case:&#xA;&#xA;class Oldstyle:&#xA;    pass&#xA;&#xA;class Otheroldstyle:&#xA;    pass&#xA;&#xA;print discovertypeofobject(Oldstyle()) == discovertypeofobject(Otheroldstyle())&#xA;&#xA;print discovertypeofobject(Oldstyle)&#xA;&#xA;Does the first print statement write True and/or does the second statement raise an exception? If your output is&#xA;&#xA;False&#xA;type &#39;classobj&#39;&#xA;&#xA;Congratulations, you&#39;ve done everything right!&#xA;&#xA;If not, you got stuck into the NewStyleVsClassicClasses problem, which makes it quite hard to do something as simple as getting the type of an arbitrary object. This is the code I created:&#xA;&#xA;from types import ClassType&#xA;NOCLASS=object()&#xA;def getobjecttype(obj):&#xA;    objtype = getattr(obj, &#34;class&#34;, NOCLASS)&#xA;    if objtype is not NOCLASS:&#xA;        return objtype&#xA;    # AFAIK the only situation where this happens is an old-style class&#xA;    objtype = type(obj)&#xA;    if objtype is not ClassType:&#xA;        raise ValueError(&#34;Could not determine object &#39;{}&#39; type.&#34;.format(objtype))&#xA;    return obj_type&#xA;&#xA;If it weren&#39;t that Python 2.7 is in maintenance mode only, I&#39;d probably submit such utility for inclusion in the inspect module.&#xA;&#xA;Originally answered on Stack Overflow:&#xA;http://stackoverflow.com/questions/2225038/determine-the-type-of-an-object/37876801\#37876801&#xA;&#xA;#Ollivander #python #TypeSystem&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>This seems a simple enough question:</p>

<p>How can I discover the type of any given Python object?</p>

<p>Suppose Python 2.7 (Python 3 is a different beast on this topic), and try implementing such function:</p>

<pre><code class="language-python">def discover_type_of_object(obj):
    &#34;&#34;&#34;
    Args:
        obj: Any python object

    Returns:
        the type of input object
    &#34;&#34;&#34;
    ....
</code></pre>

<p>When you&#39;re done, scroll the page way down...</p>

<p><br>
<br>
<br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br><br>
<br></p>

<p>Now, try the function you&#39;ve wrote with this case:</p>

<pre><code class="language-python">class Oldstyle:
    pass

class Otheroldstyle:
    pass

print discover_type_of_object(Oldstyle()) == discover_type_of_object(Otheroldstyle())

print discover_type_of_object(Oldstyle)
</code></pre>

<p>Does the first print statement write <code>True</code> and/or does the second statement raise an exception? If your output is</p>

<pre><code>False
&lt;type &#39;classobj&#39;&gt;
</code></pre>

<p>Congratulations, you&#39;ve done everything right!</p>

<p>If not, you got stuck into the <a href="https://wiki.python.org/moin/NewClassVsClassicClass" rel="nofollow">NewStyleVsClassicClasses</a> problem, which makes it quite hard to do something as simple as getting the type of an arbitrary object. This is the code I created:</p>

<pre><code class="language-python">from types import ClassType
_NO_CLASS=object()
def get_object_type(obj):
    obj_type = getattr(obj, &#34;__class__&#34;, _NO_CLASS)
    if obj_type is not _NO_CLASS:
        return obj_type
    # AFAIK the only situation where this happens is an old-style class
    obj_type = type(obj)
    if obj_type is not ClassType:
        raise ValueError(&#34;Could not determine object &#39;{}&#39; type.&#34;.format(obj_type))
    return obj_type
</code></pre>

<p>If it weren&#39;t that Python 2.7 is in maintenance mode only, I&#39;d probably submit such utility for inclusion in the <code>inspect</code> module.</p>

<p>Originally answered on Stack Overflow:
<a href="http://stackoverflow.com/questions/2225038/determine-the-type-of-an-object/37876801#37876801" rel="nofollow">http://stackoverflow.com/questions/2225038/determine-the-type-of-an-object/37876801#37876801</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:TypeSystem" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">TypeSystem</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-discover-the-type-of-an-object</guid>
      <pubDate>Fri, 22 Jul 2016 21:41:00 +0000</pubDate>
    </item>
    <item>
      <title>Python requests SSL and InsecurePlatformWarning</title>
      <link>https://www.franzoni.eu/alan/python-requests-ssl-and-insecureplatformwarning</link>
      <description>&lt;![CDATA[Every now and then, when using Python 2.7 \&lt; 2.7.9 and trying to access SSL resources, especially through the requests toolkit, which seems to trigger the issue frequently - but I&#39;ve seen it on some combinations of pip inside virtualenv as well - you&#39;ll get an error or a warning along those lines:&#xA;&#xA;InsecurePlatformWarning: A true SSLContext object is not&#xA;available. This prevents urllib3 from configuring SSL appropriately and &#xA;may cause certain SSL connections to fail. For more information, see &#xA;https://urllib3.readthedocs.org/en/latest  &#xA;/security.html#insecureplatformwarning.&#xA;&#xA;or&#xA;&#xA;SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.&#xA;  SNIMissingWarning&#xA;&#xA;or, even worse:&#xA;&#xA;error: SSLError: hostname &#39;xxx.franzoni.eu&#39; doesn&#39;t match either of www.franzoni.eu, franzoni.eu&#xA;&#xA;This is caused by old libraries in python \&lt; 2.7.9 and \&lt; 3.4. To fix that, just add to your current virtualenv those three packages:&#xA;&#xA;pip install pyOpenSSL ndg-httpsclient pyasn1&#xA;&#xA;You may need runtime/development packages for python and openssl as well in order for the build to succeed, e.g. python-dev libssl-dev libffi-dev on Ubuntu 14.04.&#xA;&#xA;#MostlyUnixish #python #requests #ssl&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Every now and then, when using Python 2.7 &lt; 2.7.9 and trying to access SSL resources, especially through the <a href="http://docs.python-requests.org/en/master/" rel="nofollow">requests</a> toolkit, which seems to trigger the issue frequently – but I&#39;ve seen it on some combinations of <strong>pip</strong> inside <strong>virtualenv</strong> as well – you&#39;ll get an error or a warning along those lines:</p>

<pre><code>InsecurePlatformWarning: A true SSLContext object is not
available. This prevents urllib3 from configuring SSL appropriately and 
may cause certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest  
/security.html#insecureplatformwarning.
</code></pre>

<p>or</p>

<pre><code>SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
</code></pre>

<p>or, even worse:</p>

<pre><code>error: SSLError: hostname &#39;xxx.franzoni.eu&#39; doesn&#39;t match either of www.franzoni.eu, franzoni.eu
</code></pre>

<p>This is caused by old libraries in python &lt; 2.7.9 and &lt; 3.4. To fix that, just add to your current virtualenv those three packages:</p>

<pre><code>pip install pyOpenSSL ndg-httpsclient pyasn1
</code></pre>

<p>You may need runtime/development packages for python and openssl as well in order for the build to succeed, e.g. <code>python-dev libssl-dev libffi-dev</code> on Ubuntu 14.04.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:requests" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">requests</span></a> <a href="/alan/tag:ssl" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">ssl</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-requests-ssl-and-insecureplatformwarning</guid>
      <pubDate>Thu, 14 Jul 2016 21:41:49 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging Woes: part 3</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-3</link>
      <description>&lt;![CDATA[A bit of wrap-up after part 1 and part 2&#xA;&#xA;Current state of the things&#xA;&#xA;I seem to have missed something: there&#39;s now a very active Python Packaging Authority which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.&#xA;&#xA;Setuptools and distribute got merged and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!&#xA;&#xA;So, rejoice! Things are getting better!&#xA;&#xA;In the meantime&#xA;&#xA;I recommend using Pip . Always try using requirements.txt, it&#39;s a good way to keep your versions&#xA;consistent and working.&#xA;&#xA;Use virtualenv for development purposes, and always test your package in a virtualenv with a disabled site-packages! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!&#xA;&#xA;In order to distribute a package, you&#39;ll need setuptools/distribute as well;&#xA;just do a source distribution and a wheel distribution (you&#39;re required to&#xA;install wheel beside setuptools); forget&#xA;about eggs, especially for eggs containing binary extensions. Your command line&#xA;should be something like:&#xA;&#xA;python setup.py clean register sdist bdist_wheel upload&#xA;&#xA;And let&#39;s keep our fingers crossed on a better state of things!&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>A bit of wrap-up after <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> and <a href="/alan/python-packaging-woes-part-2" rel="nofollow">part 2</a></p>

<h3 id="current-state-of-the-things">Current state of the things</h3>

<p>I seem to have missed something: there&#39;s now a very active <a href="https://github.com/pypa" rel="nofollow">Python</a> <a href="https://bitbucket.org/pypa" rel="nofollow">Packaging</a> <a href="https://groups.google.com/forum/#!forum/pypa-dev" rel="nofollow">Authority</a> which is trying very hard to push Python packaging in the right direction, so that we can have a good set of tools that work properly together. Checkout their repositories and mailing lists.</p>

<p>Setuptools and distribute got <a href="https://mail.python.org/pipermail/distutils-sig/2013-March/020126.html" rel="nofollow">merged</a> and we won&#39;t experience a split-experience anymore, we&#39;ll just have One Tool To Rule Them All!</p>

<p>So, rejoice! Things are getting better!</p>

<h3 id="in-the-meantime">In the meantime</h3>

<p>I recommend using <a href="http://www.pip-installer.org" rel="nofollow">Pip</a> . Always try using requirements.txt, it&#39;s a good way to keep your versions
consistent and working.</p>

<p>Use <a href="http://www.virtualenv.org" rel="nofollow">virtualenv</a> for development purposes, and <strong>always test your package in a virtualenv with a disabled site-packages</strong>! Otherwise you risk you&#39;re distributing something that works on your workstation only because of a site-wide package you have installed there!</p>

<p>In order to distribute a package, you&#39;ll need setuptools/distribute as well;
just do a source distribution and a wheel distribution (you&#39;re required to
install <a href="https://pypi.python.org/pypi/wheel" rel="nofollow">wheel</a> beside setuptools); forget
about eggs, especially for eggs containing binary extensions. Your command line
should be something like:</p>

<pre><code>python setup.py clean register sdist bdist_wheel upload
</code></pre>

<p>And let&#39;s keep our fingers crossed on a better state of things!</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-3</guid>
      <pubDate>Sun, 14 Sep 2014 20:13:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging Woes - part 2</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-2</link>
      <description>&lt;![CDATA[Check out part 1 for my thoughts on what packaging and dependency&#xA;management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!&#xA;&#xA;How Python messed it up&#xA;&#xA;A Diaspora of Tools&#xA;&#xA;There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.&#xA;&#xA;Distutils&#xA;&#xA;The original distutils is part of the Python standard library, but&#xA;it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,&#xA;its&#xA;requires&#xA;keyword seems to do&#xA;absolutely nothing else&#xA;than documenting requirements. Nothing will happen if you install a package requiring another one;&#xA;you still need to install dependencies manually.&#xA;&#xA;Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable - it changes&#xA;very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and&#xA;tired, and needs an overhaul now (more on that later).&#xA;&#xA;Setuptools&#xA;&#xA;The original setuptools package, which was created&#xA;by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained&#xA;right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.&#xA;&#xA;It includes easy\install which is probably the&#xA;first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)&#xA;on the command line, without a lot of manual intervention; also, if the downloaded package is&#xA;setuptools-aware and correctly specifies dependencies with&#xA;install\requires it will&#xA;install all transitive dependencies.&#xA;&#xA;It includes pkg\resources as well; that last&#xA;one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of&#xA;data which is installed in Python libraries, regardless of how it was installed - is it an egg? A&#xA;flat install? Did the install became part of the standard library? - don&#39;t worry: just ask for the&#xA;resource name and the package name, and pkg\resources will look up the file object for you and open&#xA;it.&#xA;&#xA;Unluckily the API was not so easy to use - that&#39;s something I&#39;m trying to address in&#xA;pydenji with its resourceloader URL-parsing based system - and many&#xA;people to handle resource in python just &#34;look for them&#34; from the current directory using&#xA;introspection by file. Too bad.&#xA;&#xA;Distribute&#xA;&#xA;In the end, setuptools had little maintenance applied and was forked; distribute is dropin&#xA;replacement which shares functions and dysfunctions of its ancestor.&#xA;&#xA;Recently it has been re-merged into setuptools.&#xA;&#xA;This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing&#xA;a specific-and-old version of setuptools, or you can find projects that know that setuptools is &#34;old&#34; and force&#xA;installing distribute (which is an issue since setuptools is now an actually updated distribute) :-/&#xA;&#xA;Distutils2&#xA;&#xA;The Next Big Thing in Python packaging, which has never happened so far - should have been available in Python 3.3, but&#xA;development seems to have stalled.&#xA;&#xA;Pip&#xA;&#xA;Pip is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works&#xA;as his author thinks it&#39;s well done - which is usually a good thing, since Ian Bicking knows what he&#39;s doing.&#xA;&#xA;Dependency Management&#xA;&#xA;Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.&#xA;Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s&#xA;a way to prevent the packages from being fetched from their homepage and just rely on pypi - see PEP 438&#xA;&#xA;One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which - of course - is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.&#xA;&#xA;Consequences&#xA;&#xA;It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,&#xA;or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment&#xA;is not easy.&#xA;&#xA;And different tools just don&#39;t play well together.&#xA;&#xA;What should be done?&#xA;&#xA;Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python&#xA;  community.&#xA;Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can&#xA;  work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not&#xA;  important if it&#39;s not part of the standard library.&#xA;Define best practices for packaging and dependencies (package layout, version numbers, metadata,&#xA;  uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why&#xA;  he does it.&#xA;Have a good doc linked from the official documentation. The current [packaging&#xA;  guide](http://guide.python-distribute.org/) is not complete and it&#39;s not linked from anything&#xA;  &#34;official&#34; - it&#39;s just another guide.&#xA;&#xA;But?&#xA;&#xA;There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :-)&#xA;&#xA;#Ollivander #python #packaging #DependencyManagement&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Check out <a href="/alan/python-packaging-woes-part-1" rel="nofollow">part 1</a> for my thoughts on what packaging and dependency
management should be. I forgot about this post and left it in a draft state for quite a lot of time, I&#39;m sorry!</p>

<h2 id="how-python-messed-it-up">How Python messed it up</h2>

<h3 id="a-diaspora-of-tools">A Diaspora of Tools</h3>

<p>There&#39;s a plethora of packaging tools for Python! Let&#39;s take a look at them.</p>

<h4 id="distutils">Distutils</h4>

<p>The original <a href="http://docs.python.org/2/distutils/" rel="nofollow">distutils</a> is part of the Python standard library, but
it&#39;s not really feature-packed, and it&#39;s considered unsuitable for many production usages. As an example,
its
<a href="http://docs.python.org/2/distutils/setupscript.html#relationships-between-distributions-and-packages" rel="nofollow">requires</a>
keyword seems to do
<a href="http://blog.doughellmann.com/2007/11/requiring-packages-with-distutils.html" rel="nofollow">absolutely nothing else</a>
than documenting requirements. Nothing will happen if you install a package requiring another one;
you still need to install dependencies manually.</p>

<p>Distutils is not inherently bad, but since it&#39;s a standard library tool it&#39;s quite stable – it changes
very little and in very predictable ways, so it wasn&#39;t updated for a long time and got quite old and
tired, and needs an overhaul now (more on that later).</p>

<h4 id="setuptools">Setuptools</h4>

<p>The original <a href="http://peak.telecommunity.com/DevCenter/setuptools" rel="nofollow">setuptools</a> package, which was created
by a clever guy called Phillip J. Eby, is what people used for a lot of time, and that it&#39;s unmantained
right now. It builds on distutils and extends/monkeypatches it, adding a lot of functionalities.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/EasyInstall" rel="nofollow">easy_install</a> which is probably the
first tool that allowed people to install packages straight from pypi (or other URLs, more on that later)
on the command line, without a lot of manual intervention; also, if the downloaded package is
setuptools-aware and correctly specifies dependencies with
<a href="http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies" rel="nofollow">install_requires</a> it will
install all transitive dependencies.</p>

<p>It includes <a href="http://peak.telecommunity.com/DevCenter/PkgResources" rel="nofollow">pkg_resources</a> as well; that last
one is one of the great Python underdogs, IMHO; it allows Python programmers to access any kind of
data which is installed in Python libraries, regardless of how it was installed – is it an egg? A
flat install? Did the install became part of the standard library? – don&#39;t worry: just ask for the
resource name and the package name, and pkg_resources will look up the file object for you and open
it.</p>

<p>Unluckily the API was not so easy to use – that&#39;s something I&#39;m trying to address in
<a href="http://pydenji.franzoni.eu" rel="nofollow">pydenji</a> with its resourceloader URL-parsing based system – and many
people to handle resource in python just “look for them” from the current directory using
introspection by <code>__file__</code>. Too bad.</p>

<h4 id="distribute">Distribute</h4>

<p>In the end, setuptools had little maintenance applied and was forked; distribute is dropin
replacement which shares functions and dysfunctions of its ancestor.</p>

<p>Recently it has been re-merged into setuptools.</p>

<p>This wouldn&#39;t be inherently a problem, but sometimes you can find projects that are distribute-unaware and try forcing
a specific-and-old version of setuptools, or you can find projects that know that setuptools is “old” and force
installing distribute (which is an issue since setuptools is now an actually updated distribute) :–/</p>

<h4 id="distutils2">Distutils2</h4>

<p>The Next Big Thing in Python packaging, which has never happened so far – should have been available in Python 3.3, but
development seems to have <a href="http://hg.python.org/distutils2" rel="nofollow">stalled</a>.</p>

<h4 id="pip">Pip</h4>

<p><a href="http://www.pip-installer.org/" rel="nofollow">Pip</a> is among the best tools you could use and works fine, but it&#39;s Yet Another Tool. It works
as his author thinks it&#39;s well done – which is usually a good thing, since Ian Bicking knows what he&#39;s doing.</p>

<h3 id="dependency-management">Dependency Management</h3>

<p>Dependency management is hard with distribute; you can&#39;t handle diamond dependencies nor do overrides.
Pypi repository is mutable (versions can be deleted by package owners) and isn&#39;t got a real API, even though, today, there&#39;s
a way to prevent the packages from being fetched from their homepage and just rely on pypi – see <a href="http://www.python.org/dev/peps/pep-0438/" rel="nofollow">PEP 438</a></p>

<p>One good way of handling dependency management in such cases is to use Pip&#39;s own requirements.txt file, which – of course – is not a standard, and may not work in all environments, or simply the project you love may not provide it at all.</p>

<h3 id="consequences">Consequences</h3>

<p>It&#39;s very easy to find packages from pypi that don&#39;t work out of the box, either because of wrong dependencies,
or because somehow pypi changed hosted versions since the time the package was built. Replicating a deployment
is not easy.</p>

<p>And different tools just don&#39;t play well together.</p>

<h2 id="what-should-be-done">What should be done?</h2>
<ul><li>Focusing more on the problem. Packaging and dependency management seem to be underdogs in the Python
community.</li>
<li>Choose a tool or an API (Maven repos became de-facto standard for non-Maven users as well) that can
work, and if it doesn&#39;t, evolve it, don&#39;t just let people invent their in-house solution. It&#39;s not
important if it&#39;s not part of the standard library.</li>
<li>Define best practices for packaging and dependencies (package layout, version numbers, metadata,
uploaded package formats, etc) and be sure the user doesn&#39;t override them unless he knows what and why
he does it.</li>
<li>Have a good doc linked from the official documentation. The current <a href="http://guide.python-distribute.org/" rel="nofollow">packaging
guide</a> is not complete and it&#39;s not linked from anything
“official” – it&#39;s just another guide.</li></ul>

<h2 id="but">But?</h2>

<p>There&#39;s a light at the end of the gallery, and it&#39;s not a train! We&#39;ll talk about it in part 3. Which I hope should happen soon enough :–)</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:DependencyManagement" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">DependencyManagement</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-2</guid>
      <pubDate>Fri, 06 Sep 2013 20:09:00 +0000</pubDate>
    </item>
    <item>
      <title>Python Packaging woes - part 1</title>
      <link>https://www.franzoni.eu/alan/python-packaging-woes-part-1</link>
      <description>&lt;![CDATA[This is a spinoff from my Europython 2012 talk (video, slides).&#xA;&#xA;Some days ago I got involved in a brief discussion on the topic, so here&#39;s what I think about Python packaging and dependency management: it&#39;s in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.&#xA;&#xA;Why? Let&#39;s see.&#xA;&#xA;The Packaging Conspiracy&#xA;&#xA;Aim&#xA;&#xA;No, it&#39;s not a conspiracy to force you to learn something you don&#39;t care about. The whole point of packaging is to make code reuse easy enough, both for the developer&#39;s own sake (e.g. shared code between different projects) and for sharing it with the public.&#xA;&#xA;Packaging usually involves dependency management; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don&#39;t just release your code in &#34;in the dark&#34;: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.&#xA;&#xA;It&#39;s an overkill!&#xA;&#xA;Some developers think all of this is a waste of their time.&#xA;&#xA;D: Hey Alan, are you really saying say I can&#39;t just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?&#xA;&#xA;A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you&#39;ve fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, you&#39;re doing by hand the work that a good dependency management tool is supposed to do. If you like that.&#xA;&#xA;D: OK, ok. I got it. So now I&#39;ve added my dependencies. But what the heck, does the version really matter? I don&#39;t care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won&#39;t break my code.&#xA;&#xA;A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it&#39;s time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.&#xA;&#xA;D: OK, ok, now I&#39;ve set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?&#xA;&#xA;A: Try doing something meaningful, and stick to your plan. You can pick your own versioning policy. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something &#34;emotional&#34;. If you perform small incremental changes on your codebase - a-la-Chrome - you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.&#xA;&#xA;D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!&#xA;&#xA;A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.&#xA;&#xA;D: YEAH! I&#39;ve just released version 0.0.2 to the public. Now I&#39;m deleting 0.0.1, right?&#xA;&#xA;A: No. Never. Ideally, unless you must - by must I mean something like &#34;there&#39;s a password for my bank account in the source code&#34; or &#34;I mistakenly copypasted code for which I hadn&#39;t the copyright&#34; - you should never, ever, ever, remove a released version. You never know who&#39;s using it - if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn&#39;t care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as &#34;deprecated&#34;, &#34;bugged&#34;, or maybe &#34;donut&#34; if they contain a huge security hole.&#xA;&#xA;D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it&#39;s my first daughter and she was born in the very day of such release.&#xA;&#xA;A: I&#39;m happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, &#34;lexicographical&#34; is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means &#34;lower&#34;, so that 2.0.0beta is \&lt; 2.0.0, but that&#39;s not always the case - a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.&#39;&#39; is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.&#xA;&#xA;D: There&#39;re a lot of things to remember about dependency management and packaging. Should I learn everything by heart?&#xA;&#xA;A: What the heck, no. Your packaging tool should guide you in such practices and prevent you from going stray unless it&#39;s what you really mean to do! That why I&#39;m complaining in the first place about the way Python does packaging!&#xA;&#xA;In part 2 I dig into Python-specific packaging issues.&#xA;&#xA;#Ollivander #python #packaging&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>This is a spinoff from my <a href="https://ep2012.europython.eu/conference/talks/language-alone-wont-pay-your-bills" rel="nofollow">Europython 2012 talk</a> (<a href="http://www.youtube.com/watch?v=JlNYO4zUDAQ" rel="nofollow">video</a>, <a href="https://ep2013.europython.eu/media/conference/slides/language-alone-wont-pay-your-bills.pdf" rel="nofollow">slides</a>).</p>

<p>Some days ago I got involved in a brief <a href="https://twitter.com/hpk42/status/292599238095933440" rel="nofollow">discussion</a> on the topic, <strong>so here&#39;s what I think about Python packaging and dependency management: it&#39;s in a very bad state, requires a lot of manual intervention to work properly and heavily limits code reuse.</strong></p>

<p>Why? Let&#39;s see.</p>

<h2 id="the-packaging-conspiracy">The Packaging Conspiracy</h2>

<h3 id="aim">Aim</h3>

<p>No, it&#39;s not a conspiracy to force you to learn something you don&#39;t care about. The whole point of packaging is to make code reuse easy enough, both for the developer&#39;s own sake (e.g. shared code between different projects) and for sharing it with the public.</p>

<p><strong>Packaging usually involves dependency management</strong>; this happens in most Linux distributions as well as other packaging systems (e.g. Apache Maven, Rubygems), because you don&#39;t just release your code in “in the dark”: you are releasing version 1.0.0 of your code on January, 1st 2013, and that usually matches (or should match) an exact revision on your source control system, and a precise set of dependencies, because your own software or library may have its own requirements.</p>

<h3 id="it-s-an-overkill">It&#39;s an overkill!</h3>

<p>Some developers think all of this is a waste of their time.</p>

<p><em>D: Hey Alan, are you really saying say I can&#39;t just copy and paste some source code in some directory and just go on? How much time should I spend at this tedious task?!?</em></p>

<p>A: Sure, just copy and paste. And remember copying and pasting all transitive dependencies as well. And since this is Python, start hoping all your transitive dependencies are pure Python, otherwise good luck at copypasting C source code into your repo and remembering to compile it every time. And whenever you need a bugfix in your upstream dependency, good luck at remembering where you&#39;ve fetched that code from, and at which revision. Oh, did you write the URL and the revision somewhere in the VCS commit message? Fine, <strong>you&#39;re doing by hand the work that a good dependency management tool is supposed to do</strong>. If you like that.</p>

<p><em>D: OK, ok. I got it. So now I&#39;ve added my dependencies. But what the heck, does the version really matter? I don&#39;t care! If a new version of something I rely on is released, it will contain bugfixes and maybe new features, but won&#39;t break my code.</em></p>

<p>A: Dear developer, I think your great trust in the Developer From The Next Door is something that feeds your heart but hinders your brain. You should depend on Only One Version for your production code; when it&#39;s time to upgrade to a later one, you need to do a manual test run and verify everything is fine. If you forget about doing that, your imagination will be the only limit to whatever mess can happen.</p>

<p><em>D: OK, ok, now I&#39;ve set versions for all my deps. What version should I use for my own software, though? Can I just pick any version I like? First version 1.0, then 1.1, until I get to 1.99 and i move to 2.0?</em></p>

<p>A: Try doing something meaningful, and stick to your plan. <strong>You can pick your own versioning policy</strong>. Usually your version numbers should distinguish at least between major releases, where you can break backwards compatibility (if you must, not just because you enjoy) and add a lot of new things, and bugfix releases, where you try to minimize the changes to what really needs to be fixed, in order not to mess up client code. Many softwares used to use a two-dotted version number, something like MAJOR.MINOR.REVISION, with REVISION being used for bugfixes only, but the distinction between MAJOR and MINOR is blurred and often just something “emotional”. If you perform small incremental changes on your codebase – <a href="http://blog.chromium.org/2010/07/release-early-release-often.html" rel="nofollow">a-la-Chrome</a> – you can stick with just MAJOR.REVISION. Every feature-adding release increases the MAJOR, every bugfix increases the REVISION. Of course, remember that your MAJOR should be strictly increasing, while REVISION is usually reset at 0 for every MAJOR and then strictly increasing.</p>

<p><em>D: So, what should I do now? Where do I release my code? Should I copypaste my code somewhere? I know of a great pastebin!</em></p>

<p>A: You should usually aggregate your build into an artifact (= product of the build) and you should upload it to a shared repository, where you and other people agree uploading your software to and fetching other people software from.</p>

<p><em>D: YEAH! I&#39;ve just released version 0.0.2 to the public. Now I&#39;m deleting 0.0.1, right?</em></p>

<p>A: No. Never. Ideally, unless you must – by must I mean something like “there&#39;s a password for my bank account in the source code” or “I mistakenly copypasted code for which I hadn&#39;t the copyright” – <strong>you should never, ever, ever, remove a released version</strong>. You never know who&#39;s using it – if my software works fine with 0.0.1, why should I be forced to upgrade? You can shout at me if I ever ask anything about it to you, according to your version policy, but you shouldn&#39;t care about old versions otherwise. If tagging is supported on the repository you might want to tag old versions as “deprecated”, “bugged”, or maybe “donut” if they contain a huge security hole.</p>

<p><em>D: I like fancy names. Can I name my next release 2.0.5sarah? You know, it&#39;s my first daughter and she was born in the very day of such release.</em></p>

<p>A: I&#39;m happy of hearing that, but please stick with numbers only. Is 2.0.5sarah higher or lower than 2.0.6? No, “lexicographical” is not what you and I want to hear in such context. You may want to use a suffix/prefix/tag to mark non-final releases, e.g. alpha/beta/development releases, preferably a SINGLE suffix, which is widely recognized, for all of those, and decide once-for-all if it means lower or higher. Usually it means “lower”, so that 2.0.0beta is &lt; 2.0.0, but that&#39;s not always the case – a very good idea is to deny that the same version of a package can exist both for a development and production version. I.e. if 2.1.&#39;&#39; is beta, 2.0.5 can be a production release, but not 2.1.0 or 2.1.1.</p>

<p><em>D: There&#39;re a lot of things to remember about dependency management and packaging. Should I learn everything by heart?</em></p>

<p>A: What the heck, no. <strong>Your packaging tool should guide you in such practices</strong> and prevent you from going stray unless it&#39;s what you really mean to do! That why I&#39;m complaining in the first place about the way Python does packaging!</p>

<p>In <a href="/alan/python-packaging-woes-part-2" rel="nofollow">part 2</a> I dig into Python-specific packaging issues.</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-packaging-woes-part-1</guid>
      <pubDate>Mon, 21 Jan 2013 20:33:00 +0000</pubDate>
    </item>
    <item>
      <title>Unicode and Encoding: Python vs Java Shootout, part 2</title>
      <link>https://www.franzoni.eu/alan/unicode-and-encoding-python-vs-java-shootout-part-2</link>
      <description>&lt;![CDATA[So, here we come to our second part; if you did miss the first, it&#39;s there. All of the following discussion is relative to Sun Java 6, but most details - if not every detail - is still valid for Java 7 and 8.&#xA;&#xA;Let&#39;s see the way Java handles unicode issues and why it&#39;s usually less problematic than Python - or, at least, while it looks less problematic than Python.&#xA;&#xA;String literals in Java, just like in Python 3, are just Unicode objects; so you don&#39;t need to think whether it&#39;s a &#34;binary string&#34; or a &#34;unicode object&#34;; if it&#39;s a binary blob, you&#39;ll probably get a byte\[\], which should result in a &#39;here be dragons&#39; behaviour about encoding.&#xA;&#xA;Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?&#xA;&#xA;System.out, which is an instance of java.io.PrintStream, internally uses a java.io.OutputStreamWriter which gets initialized using \Charset.defaultCharset()\](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset%28%5C%29)), and such value will be used for conversions between raw bytes and strings:&#xA;&#xA;Encoding.java&#xA;&#xA;package eu.franzoni.examples.pythonvsjavaencoding;&#xA;&#xA;import java.nio.charset.Charset;&#xA;&#xA;public class Encoding {&#xA;    public static void main(String[] args) throws Exception {&#xA;        String myString = &#34;àèìòù&#34;;&#xA;        System.out.println(System.out.getClass());&#xA;        System.out.println(Charset.defaultCharset());&#xA;        System.out.println(myString);&#xA;        System.out.write(myString.getBytes(Charset.defaultCharset()));&#xA;        System.out.println(&#34;&#34;);&#xA;    }&#xA;}&#xA;&#xA;result:&#xA;&#xA;class java.io.PrintStream&#xA;UTF-8&#xA;àèìòù&#xA;&#xA;The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that&#39;s discouraged as it can be unpredictable and unreliable. On the contrary, you should always use constructors and methods that accept a charset parameter if you&#39;re converting to/from raw bytes, just like String.getBytes(Charset charset) or \OutputStreamWriter(OutputStream outputStream, Charset charset)\(http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter%5C%28java.io.OutputStream), java.nio.charset.Charset))&#xA;&#xA;So far, so good. But what happens if the outputstream is unable to handle the chars you&#39;d like it to write?&#xA;&#xA;CantEncode.java&#xA;&#xA;package eu.franzoni.examples.pythonvsjavaencoding;&#xA;&#xA;import java.io.FileOutputStream;&#xA;import java.io.OutputStream;&#xA;import java.io.PrintStream;&#xA;&#xA;public class CantEncode {&#xA;    public static void main(String[] args) throws Exception {&#xA;        // change default output stream&#xA;        OutputStream outputStream = new FileOutputStream(&#34;eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output&#34;);&#xA;        PrintStream printStream = new PrintStream(outputStream, true, &#34;ASCII&#34;);&#xA;        System.setOut(printStream);&#xA;&#xA;        String myString = &#34;àèìòù&#34;;&#xA;&#xA;        System.out.println(System.out.getClass());&#xA;        System.out.println(myString);&#xA;     }&#xA;}&#xA;&#xA;result:&#xA;&#xA;class java.io.PrintStream&#xA;?????&#xA;&#xA;ASCII encoding cannot possibly output the accented chars, so they&#39;re replaced with question marks instead of throwing the dreaded UnicodeEncodeError.&#xA;&#xA;Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate &#39;heads up!&#39;. Otherwise if you&#39;re just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.&#xA;&#xA;But there&#39;s another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you&#39;ll use javac -encoding flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; if you don&#39;t pass such option, the platform default converter is used&#xA;&#xA;What&#39;s the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.&#xA;&#xA;Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That&#39;s what happens, in fact:&#xA;&#xA;echo &#34;utf8 compiled:&#34;&#xA;javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java &amp;&amp; java eu.franzoni.examples.pythonvsjavaencoding.Encoding&#xA;echo &#34;windows-1252 compiled:&#34;&#xA;javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java &amp;&amp; java eu.franzoni.examples.pythonvsjavaencoding.Encoding&#xA;&#xA;result:&#xA;&#xA;utf8 compiled:&#xA;class java.io.PrintStream&#xA;UTF-8&#xA;àèìòù&#xA;àèìòù&#xA;windows-1252 compiled:&#xA;class java.io.PrintStream&#xA;UTF-8&#xA;Ã Ã¨Ã¬Ã²Ã¹&#xA;Ã Ã¨Ã¬Ã²Ã¹&#xA;&#xA;As you can see there, it&#39;s not that the JVM encoding has any issue at runtime; it&#39;s the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There&#39;s no way around this!&#xA;&#xA;What I can recommend you is to ALWAYS set the project.build.sourceEncoding property when using Maven to compile, and to ALWAYS set the encoding attribute whenever you&#39;re using Ant. This will save you a lot, lot, lot of headaches.&#xA;&#xA;#Ollivander #java #python #unicode #encodings&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>So, here we come to our second part; if you did miss the first, it&#39;s <a href="/alan/unicode-and-encoding-python-vs-java-shootout-part-1" rel="nofollow">there</a>. All of the following discussion is relative to Sun Java 6, but most details – if not every detail – is still valid for Java 7 and 8.</p>

<p>Let&#39;s see the way Java handles unicode issues and why it&#39;s usually less problematic than Python – or, at least, while it <strong>looks</strong> less problematic than Python.</p>

<p>String literals in Java, just like in Python 3, are just Unicode objects; so you don&#39;t need to think whether it&#39;s a “binary string” or a “unicode object”; if it&#39;s a binary blob, you&#39;ll probably get a byte[], which should result in a &#39;here be dragons&#39; behaviour about encoding.</p>

<p>Nevertheless, whenever you want to print out a string you actually need binary data, raw bytes! How does Java find out what to do?</p>

<p><a href="http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#out" rel="nofollow">System.out</a>, which is an instance of <a href="http://docs.oracle.com/javase/6/docs/api/java/io/PrintStream.html" rel="nofollow">java.io.PrintStream</a>, internally uses a <a href="http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html" rel="nofollow">java.io.OutputStreamWriter</a> which gets initialized using [Charset.defaultCharset()](<a href="http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset%28%5C%29" rel="nofollow">http://docs.oracle.com/javase/6/docs/api/java/nio/charset/Charset.html#defaultCharset(\)</a>), and such value will be used for conversions between raw bytes and strings:</p>

<p><em>Encoding.java</em></p>

<pre><code class="language-java">package eu.franzoni.examples.pythonvsjavaencoding;

import java.nio.charset.Charset;

public class Encoding {
    public static void main(String[] args) throws Exception {
        String myString = &#34;àèìòù&#34;;
        System.out.println(System.out.getClass());
        System.out.println(Charset.defaultCharset());
        System.out.println(myString);
        System.out.write(myString.getBytes(Charset.defaultCharset()));
        System.out.println(&#34;&#34;);
    }
}
</code></pre>

<p>result:</p>

<pre><code>class java.io.PrintStream
UTF-8
àèìòù
</code></pre>

<p>The value of the default charset is detected by the JVM at startup, and it can vary on different OSes and because of different language environment settings. On Sun Java you can even set at JVM launch time through the file.encoding property, but that&#39;s discouraged as it can be unpredictable and unreliable. On the contrary, <strong>you should always use constructors and methods that accept a charset parameter</strong> if you&#39;re converting to/from raw bytes, just like <a href="http://docs.oracle.com/javase/6/docs/api/java/lang/String.html#getBytes%28java.nio.charset.Charset%29" rel="nofollow">String.getBytes(Charset charset)</a> or [OutputStreamWriter(OutputStream outputStream, Charset charset)](<a href="http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter%5C%28java.io.OutputStream" rel="nofollow">http://docs.oracle.com/javase/6/docs/api/java/io/OutputStreamWriter.html#OutputStreamWriter\(java.io.OutputStream</a>, java.nio.charset.Charset))</p>

<p>So far, so good. But what happens if the outputstream is unable to handle the chars you&#39;d like it to write?</p>

<p><em>CantEncode.java</em></p>

<pre><code>package eu.franzoni.examples.pythonvsjavaencoding;

import java.io.FileOutputStream;
import java.io.OutputStream;
import java.io.PrintStream;

public class CantEncode {
    public static void main(String[] args) throws Exception {
        // change default output stream
        OutputStream outputStream = new FileOutputStream(&#34;eu.franzoni.examples/src/eu/franzoni/examples/pythonvsjavaencoding/CantEncode.output&#34;);
        PrintStream printStream = new PrintStream(outputStream, true, &#34;ASCII&#34;);
        System.setOut(printStream);

        String myString = &#34;àèìòù&#34;;

        System.out.println(System.out.getClass());
        System.out.println(myString);
     }
}
</code></pre>

<p>result:</p>

<pre><code>class java.io.PrintStream
?????
</code></pre>

<p>ASCII encoding cannot possibly output the accented chars, so they&#39;re replaced with question marks instead of throwing the dreaded UnicodeEncodeError.</p>

<p>Which approach is the best? That depends on the context. If you expect somebody to actually read your text, the Java approach might just let an error slip unnoticed for a long time, while the Python approach would trigger an immediate &#39;heads up!&#39;. Otherwise if you&#39;re just logging something to the console (or to any other stream) an exception might disrupt your otherwise well-working program and force an unnecessary quit. The Java approach just looks easier for the non-charset-aware because it throws fewer errors around.</p>

<p>But there&#39;s another situation where Java falls short of Python: source file encoding. While Python allows setting it on a per-file basis and defaults it to iso-8859-1 if the coding directive is not there, on Java you&#39;ll use <a href="http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/javac.html" rel="nofollow">javac</a> <strong>-encoding</strong> flag to set it globally, per compile, in order to tell the compiler which kind of encoding the source file uses; <strong>if you don&#39;t pass such option, the platform default converter is used</strong></p>

<p>What&#39;s the problem with this approach? Consider a very common situation: you work on a project on Linux, your platform defaults to UTF-8 and everything just works.</p>

<p>Then you checkout your project on Windows; you compile it, the default charset is Windows-1252 and all your files print garbage. That&#39;s what happens, in fact:</p>

<pre><code>echo &#34;utf8 compiled:&#34;
javac -encoding utf8 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java &amp;&amp; java eu.franzoni.examples.pythonvsjavaencoding.Encoding
echo &#34;windows-1252 compiled:&#34;
javac -encoding windows-1252 eu/franzoni/examples/pythonvsjavaencoding/Encoding.java &amp;&amp; java eu.franzoni.examples.pythonvsjavaencoding.Encoding
</code></pre>

<p>result:</p>

<pre><code>utf8 compiled:
class java.io.PrintStream
UTF-8
àèìòù
àèìòù
windows-1252 compiled:
class java.io.PrintStream
UTF-8
Ã Ã¨Ã¬Ã²Ã¹
Ã Ã¨Ã¬Ã²Ã¹
</code></pre>

<p>As you can see there, it&#39;s not that the JVM encoding has any issue at runtime; it&#39;s the parser, at compile time, which is told a wrong encoding about the string literals in your source code. There&#39;s no way around this!</p>

<p>What I can recommend you is to ALWAYS set the <a href="http://maven.apache.org/general.html#encoding-warning" rel="nofollow">project.build.sourceEncoding</a> property when using Maven to compile, and to ALWAYS set the <a href="http://ant.apache.org/manual/Tasks/javac.html" rel="nofollow">encoding attribute</a> whenever you&#39;re using Ant. This will save you a lot, lot, lot of headaches.</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:java" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">java</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:unicode" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">unicode</span></a> <a href="/alan/tag:encodings" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">encodings</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/unicode-and-encoding-python-vs-java-shootout-part-2</guid>
      <pubDate>Mon, 23 Jul 2012 19:25:00 +0000</pubDate>
    </item>
    <item>
      <title>Unicode and encoding:  Python vs Java shootout, part 1</title>
      <link>https://www.franzoni.eu/alan/unicode-and-encoding-python-vs-java-shootout-part-1</link>
      <description>&lt;![CDATA[Before going on with this post, be sure you&#39;ve read a href=&#34;http://www.joelonsoftware.com/articles/Unicode.html&#34; target=&#34;blank&#34;The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)/a - otherwise, I won&#39;t be able to solve your issues.br /br /Sometimes Java fans just say that Python Unicode support sucks. While there&#39;s a bit of truth in that assertion - for Python 2.x at least, since 3.x solved the problem at its root - the real problem lies in the fact that many programmers don&#39;t know what&#39;s going on &#34;under the hood&#34;, and that python default behaviour is a bit unforgiving.br /br /First things first: Python 2.x has got ba href=&#34;http://docs.python.org/library/stdtypes.html#string-methods&#34;two distinct string types/a: /bthe so-called &#34;byte strings&#34; (str type) are one, and unicode objects (unicode type) are another one. Unicode objects are much like Java strings; they&#39;re an internal abstraction of Python, and need to be interpreted to/from byte strings whenever printing, reading/writing from/to a file, etc.br /br /Sometimes such interpreting &#34;just works&#34; (most probably if you&#39;re dealing with ascii-only text), but if such conversion fails, errors occur: Python infamous and dreaded UnicodeDecodeError can happen in many places and might sometime just puzzle the programmer:br /br /pre class=&#34;brush: py&#34;\# -- coding: utf-8 --br /import sysbr /br /print &#34;current default encoding: &#34; + sys.getdefaultencoding()br /br /&#34;àèìòù&#34; + u&#34;asd&#34;br //prebr /pre class=&#34;brush: plain&#34;javapythonunicode$ python unicode\concat.pybr /current default encoding: asciibr /Traceback (most recent call last):br / File &#34;unicode\concat.py&#34;, line 6, in \module\br / &#34;àèìòù&#34; + u&#34;asd&#34;br /UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xc3 in position 0: ordinal not in range(128)br //prebr / This happens when trying to concatenate an unicode string to string object. What really happens here, behind the scenes, is explained in unicode\concat\explained.py:br /br /pre class=&#34;brush: py&#34;\# -- coding: utf-8 --br /import sysbr /br /print &#34;current default encoding: &#34; + sys.getdefaultencoding()br /br /&#34;àèìòù&#34;.decode(sys.getdefaultencoding()) + u&#34;asd&#34;br //prebr /When using a byte string (not a unicode object) python tries to convert that string to unicode before concatenating to a unicode object. Since an encoding wasn&#39;t specified, python just uses the default.br /br /So, if you&#39;re planning to mix unicode and bytestring objects (which is not a good idea), balways remember to explicitly convert string objects to unicode by their decode() method./b Or, if you&#39;re sure the encoding is always the same, you could add a a href=&#34;http://docs.python.org/library/site.html&#34;sitecustomize.py/a to your interpreter or your project PYTHONPATH, and set such encoding. Beware that setting it system wide may lead to unexpected results when running your project on another machine.br /br /Also, just remember that the a href=&#34;http://www.python.org/dev/peps/pep-0263/&#34;coding directive/a at the begininning of the file does not change unicode encoding/decoding behaviour: that&#39;s a directive to the parser, which is used to instruct it about the current encoding of the file. That information is not retained nor used at runtime.br /br /br /Let&#39;s go for another much dreaded error now: UnicodeEncodeError!br /br /pre class=&#34;brush: py&#34;\# -- coding: utf-8 --br /import sysbr /print &#34;current stdout encoding: &#34; + str(sys.stdout.encoding)br /print &#34;current default encoding: &#34; + sys.getdefaultencoding()br /br /print u&#34;àèìòù&#34;br //prebr /results in this output: br /br /pre class=&#34;brush: plain&#34;javapythonunicode$ python unicode\print.pybr /current stdout encoding: UTF-8br /àèìòùbr //prebr /Everything seems to work (as long as your terminal supports a charset, like UTF-8, or iso-8859-1, which can display accented chars) ! But now let&#39;s go for some &#34;black magic&#34;:br /br /script class=&#34;brush: plain&#34; type=&#34;syntaxhighlighter&#34;\\![CDATA\[ javapythonunicode$ python unicode\print.py \| tee Traceback (most recent call last): File &#34;unicode\print.py&#34;, line 6, in &lt;module print u&#34;àèìòù&#34; UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode characters in position 0-4: ordinal not in range(128) current stdout encoding: None current default encoding: ascii \]\] /scriptbr /That may be puzzling, but the sys.stdout printout should give you an hint.br /br /Whenever output is to a terminal, Python does perform autodetection of the encoding of your terminal, and subsequently sets sys.stdout accordingly. When sys.stdout encoding is set, any unicode object that gets printed is encoded accordingly. When output is NOT to a terminal, python does not autodetect sys.stdout encoding and, when sys.stdout has no encoding set any unicode object that is printed is a href=&#34;http://docs.python.org/library/stdtypes.html#file-objects&#34; target=&#34;blank&#34;converted according to sys.getdefaultencoding()/abr /br /So, what really happens here is:br /br /pre class=&#34;brush: py&#34;\# -- coding: utf-8 --br /import sysbr /print &#34;current stdout encoding: &#34; + str(sys.stdout.encoding)br /print &#34;current default encoding: &#34; + sys.getdefaultencoding()br /br /sys.stdout.write(br / u&#34;àèìòù&#34;.encode(sys.stdout.encoding or sys.getdefaultencoding())br / )br //prebr /Since accented characters can be encoded to UTF-8 but not to ASCII, such UnicodeEncodeErrors arise.br /br /Also beware of a href=&#34;http://bugs.python.org/issue4947&#34; target=&#34;blank&#34;bug 4947/a - it may hit you if you&#39;re using Python 2.6 or or older.br /br /&#xA;You can find the second part of this article a href=&#34;/alan/unicode-and-encoding-python-vs-java-shootout-part-2&#34;here/a&#xA;br /&#xA;&#xA;#Ollivander #python #java #unicode #encodings&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Before going on with this post, be sure you&#39;ve read <a href="http://www.joelonsoftware.com/articles/Unicode.html" target="_blank" rel="nofollow noopener">The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)</a> - otherwise, I won&#39;t be able to solve your issues.<br/><br/>Sometimes Java fans just say that Python Unicode support sucks. While there&#39;s a bit of truth in that assertion – for Python 2.x at least, since 3.x solved the problem at its root – the real problem lies in the fact that many programmers don&#39;t know what&#39;s going on “under the hood”, and that python default behaviour is a bit unforgiving.<br/><br/>First things first: Python 2.x has got <b><a href="http://docs.python.org/library/stdtypes.html#string-methods" rel="nofollow">two distinct string types</a>: </b>the so-called “byte strings” (str type) are one, and unicode objects (unicode type) are another one. Unicode objects are much like Java strings; they&#39;re an internal abstraction of Python, and need to be interpreted to/from byte strings whenever printing, reading/writing from/to a file, etc.<br/><br/>Sometimes such interpreting “just works” (most probably if you&#39;re dealing with ascii-only text), but if such conversion fails, errors occur: Python infamous and dreaded UnicodeDecodeError can happen in many places and might sometime just puzzle the programmer:<br/><br/><pre class="brush: py"># -<em>– coding: utf-8 -</em>-<br/>import sys<br/><br/>print “current default encoding: “ + sys.getdefaultencoding()<br/><br/>“àèìòù” + u”asd”<br/></pre><br/><pre class="brush: plain">javapythonunicode$ python unicode_concat.py<br/>current default encoding: ascii<br/>Traceback (most recent call last):<br/> File “unicode_concat.py”, line 6, in &lt;module&gt;<br/> “àèìòù” + u”asd”<br/>UnicodeDecodeError: &#39;ascii&#39; codec can&#39;t decode byte 0xc3 in position 0: ordinal not in range(128)<br/></pre><br/> This happens when trying to concatenate an unicode string to string object. What really happens here, behind the scenes, is explained in unicode_concat_explained.py:<br/><br/><pre class="brush: py"># -<em>– coding: utf-8 -</em>-<br/>import sys<br/><br/>print “current default encoding: “ + sys.getdefaultencoding()<br/><br/>“àèìòù”.decode(sys.getdefaultencoding()) + u”asd”<br/></pre><br/>When using a byte string (not a unicode object) python tries to convert that string to unicode before concatenating to a unicode object. Since an encoding wasn&#39;t specified, python just uses the default.<br/><br/>So, if you&#39;re planning to mix unicode and bytestring objects (which is not a good idea), <b>always remember to explicitly convert string objects to unicode by their decode() method.</b> Or, if you&#39;re sure the encoding is always the same, you could add a <a href="http://docs.python.org/library/site.html" rel="nofollow">sitecustomize.py</a> to your interpreter or your project PYTHONPATH, and set such encoding. Beware that setting it system wide may lead to unexpected results when running your project on another machine.<br/><br/>Also, just remember that the <a href="http://www.python.org/dev/peps/pep-0263/" rel="nofollow">coding directive</a> at the begininning of the file does not change unicode encoding/decoding behaviour: that&#39;s a directive to the parser, which is used to instruct it about the current encoding of the file. That information is not retained nor used at runtime.<br/><br/><br/>Let&#39;s go for another much dreaded error now: UnicodeEncodeError!<br/><br/><pre class="brush: py"># -<em>– coding: utf-8 -</em>-<br/>import sys<br/>print “current stdout encoding: “ + str(sys.stdout.encoding)<br/>print “current default encoding: “ + sys.getdefaultencoding()<br/><br/>print u”àèìòù”<br/></pre><br/>results in this output: <br/><br/><pre class="brush: plain">javapythonunicode$ python unicode_print.py<br/>current stdout encoding: UTF-8<br/>àèìòù<br/></pre><br/>Everything seems to work (as long as your terminal supports a charset, like UTF-8, or iso-8859-1, which can display accented chars) ! But now let&#39;s go for some “black magic”:<br/><br/><br/>That may be puzzling, but the sys.stdout printout should give you an hint.<br/><br/>Whenever output is to a terminal, Python does perform autodetection of the encoding of your terminal, and subsequently sets sys.stdout accordingly. When sys.stdout encoding is set, any unicode object that gets printed is encoded accordingly. When output is NOT to a terminal, python does not autodetect sys.stdout encoding and, when sys.stdout has no encoding set any unicode object that is printed is <a href="http://docs.python.org/library/stdtypes.html#file-objects" target="_blank" rel="nofollow noopener">converted according to sys.getdefaultencoding()</a><br/><br/>So, what really happens here is:<br/><br/><pre class="brush: py"># -<em>– coding: utf-8 -</em>-<br/>import sys<br/>print “current stdout encoding: “ + str(sys.stdout.encoding)<br/>print “current default encoding: “ + sys.getdefaultencoding()<br/><br/>sys.stdout.write(<br/> u”àèìòù”.encode(sys.stdout.encoding or sys.getdefaultencoding())<br/> )<br/></pre><br/>Since accented characters can be encoded to UTF-8 but not to ASCII, such UnicodeEncodeErrors arise.<br/><br/>Also beware of <a href="http://bugs.python.org/issue4947" target="_blank" rel="nofollow noopener">bug 4947</a> - it may hit you if you&#39;re using Python 2.6 or or older.<br/><br/>
You can find the second part of this article <a href="/alan/unicode-and-encoding-python-vs-java-shootout-part-2" rel="nofollow">here</a>
<br/></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:java" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">java</span></a> <a href="/alan/tag:unicode" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">unicode</span></a> <a href="/alan/tag:encodings" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">encodings</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/unicode-and-encoding-python-vs-java-shootout-part-1</guid>
      <pubDate>Wed, 22 Feb 2012 20:23:00 +0000</pubDate>
    </item>
    <item>
      <title>Unit testing with Twisted: testing protocols</title>
      <link>https://www.franzoni.eu/alan/unit-testing-with-twisted-testing-protocols</link>
      <description>&lt;![CDATA[I had some hard time when testing a href=&#34;http://twistedmatrix.com/trac/&#34;Twisted/a protocols. Although they should be decoupled from factories, most examples I could find, including those from the a href=&#34;http://twistedmatrix.com/trac/browser/trunk/doc/core/howto/trial.xhtml?format=raw&#34;official doc/a, were just too noisy and suggested using a factory.br /br /I don&#39;t like that approach - I should not be forced to use a factory - that creates an unnecessary coupling - let&#39;s just pretend a factory existed and creates a connections, that&#39;s all our protocol should need.br /br /Of course sometimes a protocol is bound to a very specific factory since it needs to invoke methods on it for non protocol-specific data (remember, there&#39;s just one factory, while there&#39;s one protocol per connection). I don&#39;t like that behaviour, since it creates an unnecessary coupling and makes factories huge and not cohesive, and I think protocols should just be injected whatever peers they need during the a href=&#34;http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Factory.html#buildProtocol&#34;buildProtocol()/a invocation - but I&#39;ll talk about that in another post. In the meantime, I reccommend just creating a mock factory and setting it manually on the protocol.br /br /So, here it is the solution I baked:br /br /script src=&#34;https://gist.github.com/942093.js&#34; /scriptbr /br /What do you think about this solution?&#xA;&#xA;#Ollivander #python #twisted #testing&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>I had some hard time when testing <a href="http://twistedmatrix.com/trac/" rel="nofollow">Twisted</a> protocols. Although they should be decoupled from factories, most examples I could find, including those from the <a href="http://twistedmatrix.com/trac/browser/trunk/doc/core/howto/trial.xhtml?format=raw" rel="nofollow">official doc</a>, were just too noisy and suggested using a factory.<br/><br/>I don&#39;t like that approach – I should not be forced to use a factory – that creates an unnecessary coupling – let&#39;s just pretend a factory existed and creates a connections, that&#39;s all our protocol should need.<br/><br/>Of course sometimes a protocol is bound to a very specific factory since it needs to invoke methods on it for non protocol-specific data (remember, there&#39;s just one factory, while there&#39;s one protocol per connection). I don&#39;t like that behaviour, since it creates an unnecessary coupling and makes factories huge and not cohesive, and I think protocols should just be injected whatever peers they need during the <a href="http://twistedmatrix.com/documents/current/api/twisted.internet.protocol.Factory.html#buildProtocol" rel="nofollow">buildProtocol()</a> invocation – but I&#39;ll talk about that in another post. In the meantime, I reccommend just creating a mock factory and setting it manually on the protocol.<br/><br/>So, here it is the solution I baked:<br/><br/><br/><br/>What do you think about this solution?</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:twisted" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">twisted</span></a> <a href="/alan/tag:testing" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">testing</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/unit-testing-with-twisted-testing-protocols</guid>
      <pubDate>Sun, 24 Apr 2011 19:03:00 +0000</pubDate>
    </item>
  </channel>
</rss>