<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>mostlyunixish &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:mostlyunixish</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:40:57 +0000</pubDate>
    <item>
      <title>How to do a bad comparison of cloud vs colocation</title>
      <link>https://www.franzoni.eu/alan/how-to-do-a</link>
      <description>&lt;![CDATA[&#xA;&#xA;Apples and oranges&#xA;&#xA;There&#39;s this post by Ahrefs , a SEO company from Singapore, which is getting quite a lot of shares recently.&#xA;&#xA;While nobody ever said that the cloud was cheap, it&#39;s a terrible comparison. Basically, the article enumerates what they have bought (a number of Dell servers + network equipment and colocation costs) and roughly translates those to AWS cloud services.&#xA;&#xA;Some problems:&#xA;&#xA;Apparently, all of Ahrefs servers lie in a single datacenter. Good luck with disaster recovery and availability.&#xA;The cloud is nice because, you know, it&#39;s a cloud. Scalability. Flexibility. On-demand resources. Nobody forces any customer to always rent 850 EC2 servers. Ahrefs could have used EC2 machines and cloud databases with autoscaling to pay less without getting performance problems. They could have used completely managed solutions (e.g. Lambda) to totally forget about the need to manage a server, and to get a dynamic pricing.&#xA;No operational cost is included. It&#39;s true, cloud engineers aren&#39;t cheap; but a few of them can automate and manage quite a large fleet of cloud resources. On the contrary, deployment and maintenance of such a large hardware fleet is probably more complex. What was the setup cost?&#xA;&#xA;Don&#39;t do those comparisons. Choose to compare a scenario, instead, and put some numbers to estimate the risk of problems, the cost in case of downtime (which is quite easy to prevent on the cloud), and the availability of relevant resources (are you keeping your datacenter technicians on payroll even when they don&#39;t have any work to do?).&#xA;&#xA;The cloud will almost inevitably come out more expensive than raw server/colocation costs, but quite often it&#39;s going to be cheaper than a lot of &#34;enterprise&#34; managed servers that I&#39;ve seen in my life (like, some agency managing servers in a DC for a customer).&#xA;&#xA;Towards the end of the article, Ahref acknowledges that there are tradeoffs for cloud vs on premise, so I think the first part of the article is even more insincere. They know what they&#39;re doing: it&#39;s an hybrid approch where the predictable workload is kept in a private DC, but probably there&#39;s an underlying cloud infrastructure that, if everything goes wrong (e.g. the DC burns) would be able to take 100% of the required workload. Quite a different story.&#xA;&#xA;#MostlyUnixish #cloud&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2023/03/DALL-E-2023-03-13-22.40.25---server-farm-with-one-apple-and-one-orange-close-by.jpg" alt=""></p>

<p><em>Apples and oranges</em></p>

<p>There&#39;s <a href="https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8" rel="nofollow">this post by Ahrefs</a> , a SEO company from Singapore, which is getting quite a lot of shares recently.</p>

<p>While nobody ever said that the cloud was cheap, it&#39;s a terrible comparison. Basically, the article enumerates what they have bought (a number of Dell servers + network equipment and colocation costs) and roughly translates those to AWS cloud services.</p>

<p>Some problems:</p>
<ul><li>Apparently, all of Ahrefs servers lie in a single datacenter. Good luck with disaster recovery and availability.</li>
<li>The cloud is nice because, you know, it&#39;s a cloud. Scalability. Flexibility. On-demand resources. Nobody forces any customer to always rent 850 EC2 servers. Ahrefs could have used EC2 machines and cloud databases with autoscaling to pay less without getting performance problems. They could have used completely managed solutions (e.g. Lambda) to totally forget about the need to manage a server, <em>and</em> to get a dynamic pricing.</li>
<li>No operational cost is included. It&#39;s true, cloud engineers aren&#39;t cheap; but a few of them can automate and manage quite a large fleet of cloud resources. On the contrary, deployment and maintenance of such a large hardware fleet is probably more complex. What was the setup cost?</li></ul>

<p>Don&#39;t do those comparisons. Choose to compare a scenario, instead, and put some numbers to estimate the risk of problems, the cost in case of downtime (which is quite easy to prevent on the cloud), and the availability of relevant resources (are you keeping your datacenter technicians on payroll even when they don&#39;t have any work to do?).</p>

<p>The cloud will almost inevitably come out more expensive than raw server/colocation costs, but quite often it&#39;s going to be cheaper than a lot of “enterprise” managed servers that I&#39;ve seen in my life (like, some agency managing servers in a DC for a customer).</p>

<p>Towards the end of the article, Ahref acknowledges that there are tradeoffs for cloud vs on premise, so I think the first part of the article is even more insincere. They know what they&#39;re doing: it&#39;s an hybrid approch where the predictable workload is kept in a private DC, but probably there&#39;s an underlying cloud infrastructure that, if everything goes wrong (e.g. the DC burns) would be able to take 100% of the required workload. Quite a different story.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:cloud" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">cloud</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/how-to-do-a</guid>
      <pubDate>Mon, 13 Mar 2023 21:49:05 +0000</pubDate>
    </item>
    <item>
      <title>Standalone, single-file, editable Python scripts WITH DEPENDENCIES</title>
      <link>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</link>
      <description>&lt;![CDATA[&#xA;&#xA;How badly I wanted something like that?&#xA;&#xA;The problem: Python for scripting&#xA;&#xA;Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing virtualenv and so on.&#xA;&#xA;Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.&#xA;&#xA;For sure, there&#39;re many options to fully package a Python executable - PyInstaller comes to my mind, but other exist. But then you&#39;ve got a kind of &#34;build process&#34; for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines. Yes, I wouldn&#39;t do the same for &#34;real&#34; software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.&#xA;&#xA;The solution: editable python scripts with isolated dependencies&#xA;&#xA;So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have python and pip on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.&#xA;&#xA;This will install the dependencies in separate location in a temporary directory at first use, then reuse them when necessary.&#xA;&#xA;So: just copy &amp; paste the following snippets, edit the two USER SERVICEABLE sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run requests, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.&#xA;&#xA;!/usr/bin/python3&#xA;import os&#xA;import sys&#xA;from tempfile import gettempdir, NamedTemporaryFile&#xA;import hashlib&#xA;&#xA;USER SERVICEABLE: paste here your requirements.txt&#xA;the recommendation is to create a development virtualenv,&#xA;install the deps with pip inside it, then do a pip freeze&#xA;and paste the output here&#xA;REQUIREMENTS = &#34;&#34;&#34;&#xA;certifi==2018.11.29&#xA;chardet==3.0.4&#xA;idna==2.8&#xA;requests==2.21.0&#xA;urllib3==1.24.1&#xA;&#34;&#34;&#34;&#xA;USER SERVICEABLE end&#xA;&#xA;def addcustomsitepackagesdirectory(raiseiffailure=True):&#xA;    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()&#xA;    deproot = os.path.join(gettempdir(), &#34;pyallinone{}&#34;.format(digest))&#xA;    os.makedirs(deproot, existok=True)&#xA;&#xA;    for dirpath, dirnames, filenames in os.walk(deproot):&#xA;        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):&#xA;            # that&#39;s our dir!&#xA;            sys.path.insert(0, os.path.abspath(dirpath))&#xA;            return deproot&#xA;&#xA;    if raiseiffailure:&#xA;        raise ValueError(&#34;could not find our site-packages dir&#34;)&#xA;&#xA;    return deproot&#xA;&#xA;deproot = addcustomsitepackagesdirectory(False)&#xA;&#xA;depsinstalled = False&#xA;&#xA;while True:&#xA;    try:&#xA;        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!&#xA;        import requests&#xA;        # USER SERVICEABLE end&#xA;&#xA;        break&#xA;    except ImportError:&#xA;        if depsinstalled:&#xA;            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)&#xA;        try:&#xA;            from pip import main as pipmain&#xA;        except ImportError:&#xA;            from pip.internal import main as pipmain&#xA;&#xA;        with NamedTemporaryFile() as req:&#xA;            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))&#xA;            req.flush()&#xA;            pipmain([&#34;install&#34;, &#34;--prefix&#34;, deproot, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])&#xA;&#xA;        addcustomsitepackagesdirectory()&#xA;        depsinstalled = True&#xA;&#xA;HERE you can start writing the actual code of your script&#xA;&#xA;r = requests.get(&#34;https://www.google.com&#34;)&#xA;print(r.statuscode)&#xA;&#xA;How does this work?&#xA;&#xA;It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using pip. Such subdir name is autogenerated depending on your requirements - so, if your requirements change, a new directory is employed.&#xA;Then, it adds such directory to your sys.path, allowing Python to find modules and packages there.&#xA;When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.&#xA;&#xA;CAVEATS:&#xA;&#xA;Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.&#xA;If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).&#xA;If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.&#xA;&#xA;Alternative approaches:&#xA;&#xA;If you already use uv and you assume it&#39;s already installed on your systems, there&#39;s a great article here on how to use uv for python scripts with dependencies&#xA;&#xA;Photo by Milan Popovic on Unsplash&#xA;&#xA;#MostlyUnixish #python #packaging #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/02/milan-popovic-674483-unsplash-1-.jpg" alt=""></p>

<p>How badly I wanted something like that?</p>

<h3 id="the-problem-python-for-scripting">The problem: Python for scripting</h3>

<p>Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing <a href="https://virtualenv.pypa.io/en/latest/" rel="nofollow">virtualenv</a> and so on.</p>

<p>Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.</p>

<p>For sure, there&#39;re many options to fully package a Python executable – <a href="https://www.pyinstaller.org/" rel="nofollow">PyInstaller</a> comes to my mind, but other exist. But then you&#39;ve got a kind of “build process” for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: <strong>I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines.</strong> Yes, I wouldn&#39;t do the same for “real” software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.</p>

<h3 id="the-solution-editable-python-scripts-with-isolated-dependencies">The solution: editable python scripts with isolated dependencies</h3>

<p>So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have <em>python</em> and <em>pip</em> on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.</p>

<p>This will install the dependencies in <strong>separate</strong> location in a temporary directory at first use, then reuse them when necessary.</p>

<p>So: just copy &amp; paste the following snippets, edit the two <code>USER SERVICEABLE</code> sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run <code>requests</code>, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.</p>

<pre><code class="language-python">#!/usr/bin/python3
import os
import sys
from tempfile import gettempdir, NamedTemporaryFile
import hashlib

# USER SERVICEABLE: paste here your requirements.txt
# the recommendation is to create a development virtualenv,
# install the deps with pip inside it, then do a `pip freeze`
# and paste the output here
REQUIREMENTS = &#34;&#34;&#34;
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
&#34;&#34;&#34;
# USER SERVICEABLE end

def add_custom_site_packages_directory(raise_if_failure=True):
    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()
    dep_root = os.path.join(gettempdir(), &#34;pyallinone_{}&#34;.format(digest))
    os.makedirs(dep_root, exist_ok=True)

    for dirpath, dirnames, filenames in os.walk(dep_root):
        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):
            # that&#39;s our dir!
            sys.path.insert(0, os.path.abspath(dirpath))
            return dep_root

    if raise_if_failure:
        raise ValueError(&#34;could not find our site-packages dir&#34;)

    return dep_root

dep_root = add_custom_site_packages_directory(False)

deps_installed = False

while True:
    try:
        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!
        import requests
        # USER SERVICEABLE end

        break
    except ImportError:
        if deps_installed:
            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)
        try:
            from pip import main as pipmain
        except ImportError:
            from pip._internal import main as pipmain

        with NamedTemporaryFile() as req:
            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))
            req.flush()
            pipmain([&#34;install&#34;, &#34;--prefix&#34;, dep_root, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])

        add_custom_site_packages_directory()
        deps_installed = True

# HERE you can start writing the actual code of your script

r = requests.get(&#34;https://www.google.com&#34;)
print(r.status_code)
</code></pre>

<h3 id="how-does-this-work">How does this work?</h3>
<ul><li>It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using <code>pip</code>. Such subdir name is autogenerated depending on your requirements – so, if your requirements change, a new directory is employed.</li>
<li>Then, it adds such directory to your <code>sys.path</code>, allowing Python to find modules and packages there.</li>
<li>When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.</li></ul>

<h3 id="caveats">CAVEATS:</h3>
<ul><li>Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.</li>
<li>If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).</li>
<li>If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.</li></ul>

<h3 id="alternative-approaches">Alternative approaches:</h3>
<ul><li>If you already use <code>uv</code> and you assume it&#39;s already installed on your systems, there&#39;s a <a href="https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/" rel="nofollow">great article here on how to use uv for python scripts with dependencies</a></li></ul>

<p>Photo by <a href="https://unsplash.com/@itsmiki5?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge" rel="nofollow">Milan Popovic</a> on Unsplash</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</guid>
      <pubDate>Tue, 19 Feb 2019 20:58:00 +0000</pubDate>
    </item>
    <item>
      <title>Misaligned Expectations: investigating the expectations gap</title>
      <link>https://www.franzoni.eu/alan/misaligned-expectations-investigating-the-expectations-gap</link>
      <description>&lt;![CDATA[&#xA;&#xA;As some of my followers already know, I&#39;m enrolled in the great Master&#39;s program at Georgia Tech, the OMSCS.&#xA;&#xA;As a part of my studies, I&#39;m doing some research to investigate the expectations gap between the higher education and the industry sectors; why does the university teach students this way? What do students expect? And what do employers and professional?&#xA;&#xA;Help us by answering a small survey, or just subscribe if you&#39;d like to receive the results when the research is completed:&#xA;&#xA;https://www.misalignedtech.com/&#xA;&#xA;MostlyUnixish&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2018/07/janko-ferlic-174927-unsplash-1-.jpg" alt=""></p>

<p>As some of my followers already know, I&#39;m enrolled in the great Master&#39;s program at <a href="http://www.gatech.edu/" rel="nofollow">Georgia Tech</a>, the <a href="http://www.omscs.gatech.edu/" rel="nofollow">OMSCS</a>.</p>

<p>As a part of my studies, I&#39;m doing some research to investigate the expectations gap between the higher education and the industry sectors; why does the university teach students this way? What do students expect? And what do employers and professional?</p>

<p>Help us by answering a small survey, or just subscribe if you&#39;d like to receive the results when the research is completed:</p>

<p><a href="https://www.misalignedtech.com/" rel="nofollow">https://www.misalignedtech.com/</a></p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/misaligned-expectations-investigating-the-expectations-gap</guid>
      <pubDate>Thu, 05 Jul 2018 21:12:20 +0000</pubDate>
    </item>
    <item>
      <title>SCP taming: stop local silliness</title>
      <link>https://www.franzoni.eu/alan/scp-taming-stop-local-sillyness</link>
      <description>&lt;![CDATA[Every day now and then, I get an scp command wrong. Scp is designed after commands like rcp and works totally fine for local-to-local file copy.&#xA;&#xA;While this can (or could) be useful in some contexts, It&#39;s not what I like to do these days; very often, if either hosts is not remote, it&#39;s a typo on my part, and results in spurious &#34;somebody@host.example.com&#34; files somewhere in various directories on my disk.&#xA;&#xA;So, what I like to do? I put a small script like this in my path before the actual scp executable in order to pre-check arguments, and prevent accidental local-only copies.&#xA;&#xA;!/bin/bash&#xA;ORIG=&#34;/usr/bin/scp&#34;&#xA;&#xA;for var in &#34;$@&#34;&#xA;do&#xA;    if [[ $var = &#34;:&#34; ]]; then&#xA;        $ORIG &#34;$@&#34;&#xA;        exit $?&#xA;    fi&#xA;done&#xA;&#xA;echo &#34;ERROR: Missing colon. You need to pass at least one remote host specifier in source or dest&#34;&#xA;exit 1&#xA;&#xA;#MostlyUnixish #ssh #scp&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Every <del>day</del> now and then, I get an <strong>scp</strong> command wrong. Scp is designed after commands like <a href="https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.cmds4/rcp.htm" rel="nofollow">rcp</a> and works totally fine for local-to-local file copy.</p>

<p>While this can (or could) be useful in some contexts, It&#39;s not what I like to do these days; very often, if either hosts is not remote, it&#39;s a typo on my part, and results in spurious “<a href="mailto:somebody@host.example.com" rel="nofollow">somebody@host.example.com</a>” files somewhere in various directories on my disk.</p>

<p>So, what I like to do? I put a small script like this in my path <em>before</em> the actual scp executable in order to pre-check arguments, and prevent accidental local-only copies.</p>

<pre><code class="language-bash">#!/bin/bash
ORIG=&#34;/usr/bin/scp&#34;

for var in &#34;$@&#34;
do
    if [[ $var = *&#34;:&#34;* ]]; then
        $ORIG &#34;$@&#34;
        exit $?
    fi
done

echo &#34;ERROR: Missing colon. You need to pass at least one remote host specifier in source or dest&#34;
exit 1
</code></pre>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:ssh" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">ssh</span></a> <a href="/alan/tag:scp" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">scp</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/scp-taming-stop-local-sillyness</guid>
      <pubDate>Thu, 26 Apr 2018 10:38:49 +0000</pubDate>
    </item>
    <item>
      <title>Command line data crunching with Python</title>
      <link>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</link>
      <description>&lt;![CDATA[&#xA;&#xA;Every time I&#39;m doing some data crunching on the command line, I find myself juggling between sed, awk, sort, uniq, etc. While I like the UNIX way of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.&#xA;&#xA;I know that Perl and Ruby support implicit loops / prints - see this and that. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.&#xA;&#xA;On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy - the -c switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:&#xA;&#xA;$ python -c &#39;import sys;for x in sys.stdin:print x&#39;&#xA;&#xA;won&#39;t &#34;just work&#34;:&#xA;&#xA;  File &#34;string&#34;, line 1&#xA;    import sys;for x in [1,5]:    print x;print x&#xA;                 ^&#xA;SyntaxError: invalid syntax&#xA;&#xA;But: there&#39;s a bash feature to interpret escape sequences in single-quoted strings, so this will work fine:&#xA;&#xA;$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;hello&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.&#xA;&#xA;AN IMPORTANT NOTE: if you&#39;re treating non-ascii data, I suggest you set the PYTHONIOENCODING variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:&#xA;&#xA;echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;ààà&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend this book enough.&#xA;&#xA;Photo by Daniel Cheung on Unsplash&#xA;&#xA;#Ollivander #python #datacrunching #MostlyUnixish&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2018/02/daniel-cheung-129839-1-.jpg" alt=""></p>

<p>Every time I&#39;m doing some data crunching on the command line, I find myself juggling between <em>sed, awk, sort, uniq, etc</em>. While I like <em>the UNIX way</em> of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.</p>

<p>I know that Perl and Ruby support implicit loops / prints – see <a href="http://www.wellho.net/resources/ex.php4?item=p210/spot" rel="nofollow">this</a> and <a href="http://www.wellho.net/resources/ex.php4?item=r110/ruby_awk.rb" rel="nofollow">that</a>. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.</p>

<p>On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy – the <code>-c</code> switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:</p>

<pre><code>$ python -c &#39;import sys;for x in sys.stdin:print x&#39;
</code></pre>

<p>won&#39;t “just work”:</p>

<pre><code>  File &#34;&lt;string&gt;&#34;, line 1
    import sys;for x in [1,5]:    print x;print x
                 ^
SyntaxError: invalid syntax
</code></pre>

<p>But: there&#39;s a <a href="https://stackoverflow.com/questions/11966312/how-does-the-leading-dollar-sign-affect-single-quotes-in-bash" rel="nofollow">bash feature to interpret escape sequences in single-quoted strings</a>, so this will work fine:</p>

<pre><code>$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
hello
world
this
is
me
</code></pre>

<p>I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.</p>

<p><strong>AN IMPORTANT NOTE:</strong> if you&#39;re treating non-ascii data, I suggest you set the <em>PYTHONIOENCODING</em> variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:</p>

<pre><code>echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
ààà
world
this
is
me
</code></pre>

<p>Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend <a href="https://www.datascienceatthecommandline.com/" rel="nofollow">this book</a> enough.</p>

<p><em>Photo by Daniel Cheung on <a href="https://unsplash.com/photos/cPF2nlWcMY4" rel="nofollow">Unsplash</a></em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:datacrunching" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">datacrunching</span></a> <a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</guid>
      <pubDate>Wed, 14 Feb 2018 14:44:51 +0000</pubDate>
    </item>
    <item>
      <title>Git: automatically set and use multiple commit identities</title>
      <link>https://www.franzoni.eu/alan/git-identities</link>
      <description>&lt;![CDATA[&#xA;&#xA;Sure, git is great. Sure it is possible to use multiple commit identities in git - just set local per-repo variables. If it weren&#39;t for the fact that chances of forgetting about it is about 99.9%.&#xA;&#xA;Problem context&#xA;&#xA;When installing git, it is required to configure a username and an email that end up in commit logs:&#xA;&#xA;commit b8e77cefcceccddbc74cb13348ccc51d9128872f&#xA;Author: Alan Franzoni username@franzoni.eu&#xA;Date:   Thu Aug 3 16:35:53 2017 +0200&#xA;&#xA;    Drop fc24, add fc26 support&#xA;&#xA;I&#39;ve got a work computer, and a personal one, but I often do some work from my own machine, and sometimes (i.e. during lunch breaks or commuting) I do some personal-related work from my work machine. So, when I clone a personal project on my work machine or vice-versa, then commit anything, I end up with mixed up work-related email &amp; usernames.&#xA;&#xA;Sure, nothing tragic, but annoying - especially if CI systems pickup commit logs for sending error mails, and I don&#39;t realize I&#39;ve broken a build until it&#39;s too late.&#xA;&#xA;The stupid but effective solution&#xA;&#xA;On my personal workstation, I set my personal info for the global config, then override my git executable this way:&#xA;&#xA;!/bin/bash&#xA;GITBIN=&#34;/usr/local/bin/git&#34;&#xA;include the trailing slash, not a leading one for ssh/https compatibility&#xA;GITHUBORGNAME=&#34;MyMightyOrganization/&#34;&#xA;OVERRIDEUSERNAME=&#34;Alan Franzoni&#34;&#xA;OVERRIDEEMAIL=&#34;alan@mymightyorganization.com&#34;&#xA;if [ &#34;$1&#34; != &#34;clone&#34; ] || grep -v &#34;${GITHUBORGNAME}&#34; &lt;&lt;&lt; &#34;$@&#34; ; then&#xA;    ${GITBIN} &#34;$@&#34;&#xA;    EXITCODE=&#34;$?&#34;&#xA;    exit ${EXITCODE}&#xA;fi&#xA;&#xA;shift&#xA;&#xA;${GITBIN} clone --config user.name=&#34;${OVERRIDEUSERNAME}&#34; --config user.email=&#34;${OVERRIDEEMAIL}&#34; &#34;$@&#34;&#xA;EXITCODE=&#34;$?&#34;&#xA;exit ${EXIT_CODE}&#xA;&#xA;br&#xA;I leverage the fact that it&#39;s possible to set per-repository config values that override global ones, and I hook into the clone command and set those values at clone time, so I don&#39;t risk forgetting about it. Since it&#39;s a repository property that gets set, it will work in other clients, GUIs or IDEs as well.&#xA;&#xA;Photo by Andre Hunter&#xA;&#xA;#MostlyUnixish #git&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2017/10/andre-hunter-350301-1-.jpg" alt=""></p>

<p>Sure, git is great. Sure it is <a href="http://www.thecodedself.com/Using-Multiple-Author-Identities-With-Git/" rel="nofollow">possible</a> to use multiple commit identities in git – just set local per-repo variables. If it weren&#39;t for the fact that chances of forgetting about it is about 99.9%.</p>

<h4 id="problem-context">Problem context</h4>

<p>When installing git, it is required to <a href="https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup" rel="nofollow">configure a username and an email</a> that end up in commit logs:</p>

<pre><code>commit b8e77cefcceccddbc74cb13348ccc51d9128872f
Author: Alan Franzoni &lt;username@franzoni.eu&gt;
Date:   Thu Aug 3 16:35:53 2017 +0200

    Drop fc24, add fc26 support
</code></pre>

<p>I&#39;ve got a work computer, and a personal one, but I often do some work from my own machine, and sometimes (i.e. during lunch breaks or commuting) I do some personal-related work from my work machine. So, when I clone a personal project on my work machine or vice-versa, then commit anything, <strong>I end up with mixed up work-related email &amp; usernames</strong>.</p>

<p>Sure, nothing tragic, but annoying – especially if CI systems pickup commit logs for sending error mails, and I don&#39;t realize I&#39;ve broken a build until it&#39;s too late.</p>

<h4 id="the-stupid-but-effective-solution">The stupid but effective solution</h4>

<p>On my personal workstation, I set my personal info for the global config, then override my git executable this way:</p>

<pre><code class="language-bash">#!/bin/bash
GIT_BIN=&#34;/usr/local/bin/git&#34;
# include the trailing slash, not a leading one for ssh/https compatibility
GITHUB_ORGNAME=&#34;MyMightyOrganization/&#34;
OVERRIDE_USERNAME=&#34;Alan Franzoni&#34;
OVERRIDE_EMAIL=&#34;alan@mymightyorganization.com&#34;
if [ &#34;$1&#34; != &#34;clone&#34; ] || grep -v &#34;${GITHUB_ORGNAME}&#34; &lt;&lt;&lt; &#34;$@&#34; ; then
    ${GIT_BIN} &#34;$@&#34;
    EXIT_CODE=&#34;$?&#34;
    exit ${EXIT_CODE}
fi

shift

${GIT_BIN} clone --config user.name=&#34;${OVERRIDE_USERNAME}&#34; --config user.email=&#34;${OVERRIDE_EMAIL}&#34; &#34;$@&#34;
EXIT_CODE=&#34;$?&#34;
exit ${EXIT_CODE}
</code></pre>

<p><br>
I leverage the fact that it&#39;s possible to set per-repository config values that override global ones, and I hook into the clone command and set those values at clone time, so I don&#39;t risk forgetting about it. Since it&#39;s a repository property that gets set, it will work in other clients, GUIs or IDEs as well.</p>

<p><em>Photo by <a href="https://unsplash.com/@dre0316" rel="nofollow">Andre Hunter</a></em></p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:git" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">git</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/git-identities</guid>
      <pubDate>Thu, 12 Oct 2017 13:24:21 +0000</pubDate>
    </item>
    <item>
      <title>badblocks: test your mass storage!</title>
      <link>https://www.franzoni.eu/alan/badblocks-test-your-mass-storage-2</link>
      <description>&lt;![CDATA[Are you unsure whether your mass storage works properly? Do you suspect it&#39;s faulty somewhere?&#xA;&#xA;Verify it with a nifty tool, included in most Linux distros and in macOS if you install the ext2fs tools:&#xA;&#xA;sudo badblocks -swv -t random -b 4096 -c 2048 DEVICE&#xA;&#xA;It will perform a read-write test of your mass storage device, and tell you what&#39;s wrong.&#xA;&#xA;WARNING: everything will be deleted during the test! Make sure you&#39;ve already backupped your data!&#xA;&#xA;#MostlyUnixish #tips #bash&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Are you unsure whether your mass storage works properly? Do you suspect it&#39;s faulty somewhere?</p>

<p>Verify it with a nifty tool, included in most Linux distros and in macOS if you install the ext2fs tools:</p>

<pre><code>sudo badblocks -swv -t random -b 4096 -c 2048 &lt;DEVICE&gt;
</code></pre>

<p>It will perform a read-write test of your mass storage device, and tell you what&#39;s wrong.</p>

<p><strong>WARNING:</strong> everything will be deleted during the test! Make sure you&#39;ve already backupped your data!</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:tips" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">tips</span></a> <a href="/alan/tag:bash" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">bash</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/badblocks-test-your-mass-storage-2</guid>
      <pubDate>Mon, 26 Sep 2016 10:29:58 +0000</pubDate>
    </item>
    <item>
      <title>Automated cleanup of Docker leavings</title>
      <link>https://www.franzoni.eu/alan/cleaning-up-docker-leavings</link>
      <description>&lt;![CDATA[&#xA;&#xA;As much as I like Docker, one thing it doesn&#39;t do well: cleanup whatever is left on my hard drive after running it for a while. There&#39;re generally two kinds of Docker Waste:&#xA;&#xA;Exited containers. Unless you used run --rm when running a new container from an image, docker doesn&#39;t automagically remove containers once they&#39;re stopped; you need to rm them explicitly. Since it&#39;s generally considered (at least, by me) a Docker good practice to never restart a container, they just stay there and take valuable disk space. Of course sometimes it&#39;s useful or even necessary not to start a container with --rm, because you may want to inspect the filesystem after exiting, but most of the times it&#39;s an oversight.&#xA;Dangling images. Whenever you build a brand-new Docker image with the same tag as an existing one, you&#39;ll be possibly left with an old, dangling image - i.e an image you can only use by tag, not by name. The same could happen when an upstream image changes. More or less just like a git dangling object. Small note: it doesn&#39;t necessarily happen. The new image can be a perfect child of the previous image, in that case Docker uses a kind of inheritance mechanism which produces no dangling images. But if the image changes &#34;in the middle&#34; of the Dockerfile, and hence a piece of the old image get invalidated, then you&#39;ll be left with some dangling images.&#xA;&#xA;So, here&#39;s the script. Just put it in your /etc/cron.daily directory and live quite happily ever after:&#xA;&#xA;!/bin/bash&#xA;docker ps -a -q --filter=&#34;status=created&#34; | xargs --no-run-if-empty \&#xA;docker rm&#xA;docker ps -a -q --filter=&#34;status=exited&#34; | xargs --no-run-if-empty \&#xA;docker rm&#xA;docker images -q -f dangling=true | xargs --no-run-if-empty \&#xA;docker rmi&#xA;&#xA;#MostlyUnixish #docker&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2016/09/1619438_76bcbb30.jpg" alt=""></p>

<p>As much as I like <a href="https://www.docker.com/" rel="nofollow">Docker</a>, one thing it doesn&#39;t do well: cleanup whatever is left on my hard drive after running it for a while. There&#39;re generally two kinds of Docker Waste:</p>
<ul><li>Exited containers. Unless you used <code>run --rm</code> when running a new container from an image, docker doesn&#39;t automagically remove containers once they&#39;re stopped; you need to <code>rm</code> them explicitly. Since it&#39;s generally considered (at least, by me) a Docker good practice to <em>never restart a container</em>, they just stay there and take valuable disk space. Of course sometimes it&#39;s useful or even necessary not to start a container with <code>--rm</code>, because you may want to inspect the filesystem after exiting, but most of the times it&#39;s an oversight.</li>
<li>Dangling images. Whenever you build a brand-new Docker image with the same tag as an existing one, you&#39;ll be possibly left with an old, dangling image – i.e an image you can only use by tag, not by name. The same could happen when an upstream image changes. More or less just like a <a href="https://git-scm.com/book/it/v2/Git-Internals-Maintenance-and-Data-Recovery" rel="nofollow">git dangling object</a>. <em>Small note: it doesn&#39;t necessarily happen. The new image can be a perfect child of the previous image, in that case Docker uses a kind of inheritance mechanism which produces no dangling images. But if the image changes “in the middle” of the Dockerfile, and hence a piece of the old image get invalidated, then you&#39;ll be left with some dangling images</em>.</li></ul>

<p>So, here&#39;s the script. Just put it in your <code>/etc/cron.daily</code> directory and live quite happily ever after:</p>

<pre><code>#!/bin/bash
docker ps -a -q --filter=&#34;status=created&#34; | xargs --no-run-if-empty \
docker rm
docker ps -a -q --filter=&#34;status=exited&#34; | xargs --no-run-if-empty \
docker rm
docker images -q -f dangling=true | xargs --no-run-if-empty \
docker rmi
</code></pre>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:docker" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">docker</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/cleaning-up-docker-leavings</guid>
      <pubDate>Tue, 13 Sep 2016 07:28:33 +0000</pubDate>
    </item>
    <item>
      <title>Python requests SSL and InsecurePlatformWarning</title>
      <link>https://www.franzoni.eu/alan/python-requests-ssl-and-insecureplatformwarning</link>
      <description>&lt;![CDATA[Every now and then, when using Python 2.7 \&lt; 2.7.9 and trying to access SSL resources, especially through the requests toolkit, which seems to trigger the issue frequently - but I&#39;ve seen it on some combinations of pip inside virtualenv as well - you&#39;ll get an error or a warning along those lines:&#xA;&#xA;InsecurePlatformWarning: A true SSLContext object is not&#xA;available. This prevents urllib3 from configuring SSL appropriately and &#xA;may cause certain SSL connections to fail. For more information, see &#xA;https://urllib3.readthedocs.org/en/latest  &#xA;/security.html#insecureplatformwarning.&#xA;&#xA;or&#xA;&#xA;SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.&#xA;  SNIMissingWarning&#xA;&#xA;or, even worse:&#xA;&#xA;error: SSLError: hostname &#39;xxx.franzoni.eu&#39; doesn&#39;t match either of www.franzoni.eu, franzoni.eu&#xA;&#xA;This is caused by old libraries in python \&lt; 2.7.9 and \&lt; 3.4. To fix that, just add to your current virtualenv those three packages:&#xA;&#xA;pip install pyOpenSSL ndg-httpsclient pyasn1&#xA;&#xA;You may need runtime/development packages for python and openssl as well in order for the build to succeed, e.g. python-dev libssl-dev libffi-dev on Ubuntu 14.04.&#xA;&#xA;#MostlyUnixish #python #requests #ssl&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Every now and then, when using Python 2.7 &lt; 2.7.9 and trying to access SSL resources, especially through the <a href="http://docs.python-requests.org/en/master/" rel="nofollow">requests</a> toolkit, which seems to trigger the issue frequently – but I&#39;ve seen it on some combinations of <strong>pip</strong> inside <strong>virtualenv</strong> as well – you&#39;ll get an error or a warning along those lines:</p>

<pre><code>InsecurePlatformWarning: A true SSLContext object is not
available. This prevents urllib3 from configuring SSL appropriately and 
may cause certain SSL connections to fail. For more information, see 
https://urllib3.readthedocs.org/en/latest  
/security.html#insecureplatformwarning.
</code></pre>

<p>or</p>

<pre><code>SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
</code></pre>

<p>or, even worse:</p>

<pre><code>error: SSLError: hostname &#39;xxx.franzoni.eu&#39; doesn&#39;t match either of www.franzoni.eu, franzoni.eu
</code></pre>

<p>This is caused by old libraries in python &lt; 2.7.9 and &lt; 3.4. To fix that, just add to your current virtualenv those three packages:</p>

<pre><code>pip install pyOpenSSL ndg-httpsclient pyasn1
</code></pre>

<p>You may need runtime/development packages for python and openssl as well in order for the build to succeed, e.g. <code>python-dev libssl-dev libffi-dev</code> on Ubuntu 14.04.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:requests" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">requests</span></a> <a href="/alan/tag:ssl" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">ssl</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/python-requests-ssl-and-insecureplatformwarning</guid>
      <pubDate>Thu, 14 Jul 2016 21:41:49 +0000</pubDate>
    </item>
    <item>
      <title>chmod and the capital X</title>
      <link>https://www.franzoni.eu/alan/chmod-and-the-capital-x</link>
      <description>&lt;![CDATA[This is one very, very, very interesting piece of ancient wisdom.&#xA;&#xA;Suppose you need to change permissions for some directories. Maybe the permissions are too strict, e.g. something like 700 for directories and 600 for files, and you need to give access to some other group and not just the owner.&#xA;&#xA;As you know, execute permission on directories is required for traversal. So you might think doing something like:&#xA;&#xA;chmod -R g+rx /var/somedirectory&#xA;&#xA;This would probably work, but would leave you in a sad state: you&#39;ll get the executable bit set on files that are not supposed to be executable at all, which may be just annoying but may get risky as well.&#xA;&#xA;A great alternative is:&#xA;&#xA;chmod -R g+rX /var/somedirectory&#xA;&#xA;the capital X means: set the executable bit only if the target a) is a directory b) has already at least one executable bit set for any one of user, group, others.&#xA;&#xA;There&#39;s another alternative that may be appealing in some situations, at least when you want to give the group the same permissions as the owner:&#xA;&#xA;chmod -R g=u /var/somedirectory&#xA;&#xA;This will give any file the group permissions identical to the user&#39;s permission.&#xA;&#xA;#MostlyUnixish #linux #ancientwisdom #chmod&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>This is one very, very, very interesting piece of ancient wisdom.</p>

<p>Suppose you need to change permissions for some directories. Maybe the permissions are too strict, e.g. something like 700 for directories and 600 for files, and you need to give access to some other group and not just the owner.</p>

<p>As you know, execute permission on directories is required for traversal. So you might think doing something like:</p>

<pre><code class="language-bash">chmod -R g+rx /var/somedirectory
</code></pre>

<p>This would probably work, but would leave you in a sad state: you&#39;ll get the executable bit set on files that are not supposed to be executable at all, which may be just annoying but may get risky as well.</p>

<p>A great alternative is:</p>

<pre><code class="language-bash">chmod -R g+rX /var/somedirectory
</code></pre>

<p>the capital X means: <em>set the executable bit only if the target a) is a directory b) has already at least one executable bit set for any one of user, group, others</em>.</p>

<p>There&#39;s another alternative that may be appealing in some situations, at least when you want to give the group the same permissions as the owner:</p>

<pre><code class="language-bash">chmod -R g=u /var/somedirectory
</code></pre>

<p>This will give any file the group permissions identical to the user&#39;s permission.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:linux" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">linux</span></a> <a href="/alan/tag:ancientwisdom" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">ancientwisdom</span></a> <a href="/alan/tag:chmod" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">chmod</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/chmod-and-the-capital-x</guid>
      <pubDate>Tue, 27 Jan 2015 22:31:00 +0000</pubDate>
    </item>
  </channel>
</rss>