<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>alan</title>
    <link>https://www.franzoni.eu/alan/</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:20:47 +0000</pubDate>
    <item>
      <title>AI-assisted coding trumps vibe coding</title>
      <link>https://www.franzoni.eu/alan/ai-assisted-coding-trumps-vibe-coding</link>
      <description>&lt;![CDATA[Being an AI evolutionist rather than revolutionist, I have refrained myself from posting too much about AI - a lot of people are doing so by the way, I very much doubt you’re missing my opinion.&#xA;&#xA;But, I recently got back from Pycon Italy and I realized one thing: everybody uses AI at work. This includes me.&#xA;&#xA;At the same time, in a professional software development context, nobody gives a f- about vibe coding.&#xA;&#xA;The name is cool, but the idea of leaving the AI do the all the dirty work is just silly. And, despite the hype, this is very clear to all software professionals. AI provides a lot of value right now, but generating working software from natural language requirements seems a worse version of the classical “just get a bunch of juniors to get this job done” disaster.&#xA;&#xA;Properly using AI to enhance our coding is a deeply intellectual exercise and it’s high time we recognize it as such. Insisting on cherry-picked examples where “vibe coding created my app even if I don’t know how to program” is pretty pointless - for a professional programmer. I’ll let other people, who haven’t experienced the magic feeling of something you create coming to life, or have stopped doing so, rely on vibe coding. It’s great for prototyping, or for designers, or for examples.&#xA;&#xA;I’ll keep using AI as one of the many tools I have, as a professional developer, to improve my work, just as compilers, linters, IDEs, new languages and libraries.&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Being an AI evolutionist rather than revolutionist, I have refrained myself from posting too much about AI – a lot of people are doing so by the way, I very much doubt you’re missing my opinion.</p>

<p>But, I recently got back from <a href="https://2025.pycon.it/en" rel="nofollow">Pycon Italy</a> and I realized one thing: <strong>everybody uses AI at work.</strong> This includes me.</p>

<p>At the same time, <strong>in a professional software development context</strong>, <strong>nobody gives a f- about vibe coding.</strong></p>

<p>The name is cool, but the idea of leaving the AI do the all the dirty work is just silly. And, despite the hype, this is very clear to all software professionals. AI provides a lot of value right now, but generating working software from natural language requirements seems a worse version of the classical “just get a bunch of juniors to get this job done” disaster.</p>

<p>Properly using AI to enhance our coding is a <a href="https://www.businessinsider.com/andrew-ng-vibe-coding-unfortunate-term-exhausting-job-2025-6" rel="nofollow">deeply intellectual exercise</a> and it’s high time we recognize it as such. Insisting on cherry-picked examples where “vibe coding created my app even if I don’t know how to program” is pretty pointless – <em>for a professional programmer</em>. I’ll let other people, who haven’t experienced the magic feeling of something you create coming to life, or have stopped doing so, rely on vibe coding. It’s great for prototyping, or for designers, or for examples.</p>

<p>I’ll keep using AI as one of the many tools I have, as a professional developer, to improve my work, just as compilers, linters, IDEs, new languages and libraries.</p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/ai-assisted-coding-trumps-vibe-coding</guid>
      <pubDate>Thu, 19 Jun 2025 15:04:41 +0000</pubDate>
    </item>
    <item>
      <title>How to use AI to suck at user experience</title>
      <link>https://www.franzoni.eu/alan/how-to-use-ai-to-suck-at-customer-support</link>
      <description>&lt;![CDATA[&#xA;&#xA;Photo by Alexandre Debiève / Unsplash&#xA;&#xA;Or: the perils of blindly using AI to save some money.&#xA;&#xA;IMPORTANT NOTE: while I&#39;m pointing out some specific companies in this post, it&#39;s not meant to discredit them; instead, I hope to trigger a bit of reflection on how AI should (and should not) be used, and how it should not replace your product and user experience thinking.&#xA;&#xA;I admit I was kind of waiting for it to happen; bad usage of AI was going to create some problem for me one day or another.&#xA;&#xA;Full disclosure: I&#39;m not an AI skepticist, I&#39;ve got quite a bit of machine learning and data science experience, and I think the current LLM-oriented AI wave certainly has value, but I believe it&#39;s an evolution rather than the revolution that hype-riders claim it to be.&#xA;&#xA;The facts&#xA;&#xA;Some days ago I was having a short vacation in Lienz, Austria, a nice small city from where you can follow a bike path through the mountains to San Candido, Italy and then return by train (or vice-versa). I still recommend the trip!&#xA;&#xA;We were going to take the train to San Candido at 8.50am, and we arrived at the train station parking at 8.30am, so we had quite a bit of time. But: you need to pay for the parking. No problem, right? Unless the parking meter, which looked quite modern, only accepted coins: no cards and no bills. We needed to pay a total of EUR 8,00 and hadn&#39;t got enough coins. So, what? Technology to the rescue, of course: we downloaded the official parking app, as advertised on the side of the parking meter.&#xA;&#xA;parking meter Parkster ad&#xA;&#xA;So, I duly scanned the QR code which led me to the Parkster IOS app . I installed it and looked for the area code in order to pay:&#xA;&#xA;parkster screenshot&#xA;&#xA;I tried looking manually for the parking area; no luck. I gave location access to the app, and it worked flawlessly, the map was pointing precisely to our parking spot. And, while entering the code, I could see that some parking areas indeed existed as they were partially matching:&#xA;&#xA;partial area matches&#xA;&#xA;So, the app seemed to be working indeed, it was just that our parking area wasn&#39;t showing up. By then, we were risking running out of time, so we scrambled to change some coins, and we barely made it to the train. If you have access to the app in your App Store, try yourself - the behaviour is still the same as of August, 26th 2023.&#xA;&#xA;Trying to fix the issue&#xA;&#xA;But I had took photos and screenshots, so, once I got back home, I sent an e-mail to the Lienz Town Hall, to Parkster customer service (the e-mail was right there in the ad) and to Lienz Tourist Information (to their credit, Parkster is the only addressee that answered so far), explaining the issue I found with a certain amount of details and proof:&#xA;&#xA;1st support request&#xA;&#xA;I got this response from Parkster:&#xA;&#xA;answer 1&#xA;&#xA;This is quite a generic answer, which is something that can happen with support requests; a &#34;template answer&#34; is sent at first to reduce operator burden, and only if the customer follows up then a &#34;real operator&#34; is involved. But the &#34;you need to register&#34; is either factually incorrect (I was using the app without registering) or misleading; so I followed up:&#xA;&#xA;followup 1&#xA;&#xA;and I got this:&#xA;&#xA;answer 2&#xA;&#xA;Now I was getting puzzled. Did they read my message? It appeared so, since the answer kind of made some sense. But, the area code was available both in screenshots from my first e-mail, and it was explicitly there in my followup question.&#xA;&#xA;Then, I tried registering, and... the area codes would finally show up! During registration, I was asked &#34;in which country you&#39;d like to park&#34;, so I suspect that only related area codes are shown. Why? I don&#39;t know, but I don&#39;t actually care, there may be some reason. So, I had one more followup with the customer support just to explain that this &#34;detail&#34; isn&#39;t written anywhere and this was quite a UX problem. But at this point I had some suspicion about my support agent; it seemed like he wanted to answer something to me, but he did not really care how I was communicating something which should have value to his own employer (I sincerely doubt I was the only one who had experienced such issue, Austria is quite packed with tourists):&#xA;&#xA;final interaction&#xA;&#xA;that made me think &#34;I am really chatting with a human or are they using ChatGPT to answer?!?&#34; - I just told you I have registered and now I can find the zone! And the other &#34;live tickets&#34; and invoice parts are totally unrelated!&#xA;&#xA;On such last message, I spotted a different sender than beforehand (I don&#39;t know why):&#xA;&#xA;and, ThinkOwl indeed is Customer Service Software powered by AI, whatever that means.&#xA;&#xA;Wrapping up&#xA;&#xA;For the parking app&#xA;&#xA;Some customer was trying to help you and point out a bug or a UX problem to you.&#xA;&#xA;An answer like &#34;Thank you for your message. Our development team has been informed and we&#39;ll consider possible changes to our system in the future&#34; would have both great and enough. And, the change could be something as stupid as adding &#34;Please make sure to register for the correct country or zones won&#39;t be found&#34; to the &#34;not found message&#34;.&#xA;&#xA;AI is not really helping in this situation. It probably lets some departments to meet some cost and answer time targets, but they&#39;ll miss the opportunity to learn customer problems and improve their product and user experience.&#xA;&#xA;Don&#39;t use AI this way. In fact, I expect AI to improve customer experience by empowering customer support representatives to work better and faster, answering quickly to FAQs but letting them take more time to answer more complex - but legit - answers or to interact with development teams to actually fix real, underlying issues.&#xA;&#xA;For the Lienz municipality&#xA;&#xA;Product thinking goes end-to-end. Installing a coin-only parking meter in 2023 (or 2020, I can&#39;t know the exact date, but it looked quite recent) because you&#39;re relying on a third-party app can be a good idea, but, then, you need to test your user flow end-to-end. Your city relies a lot on tourists, and you don&#39;t want to piss them off!&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://images.unsplash.com/photo-1585744134783-ae8804f900dc?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=M3wxMTc3M3wwfDF8c2VhcmNofDE1fHxzdWNrc3xlbnwwfHx8fDE2OTMwNTE0MTl8MA&amp;ixlib=rb-4.0.3&amp;q=80&amp;w=2000" alt=""></p>

<p><em>Photo by <a href="https://unsplash.com/@alexkixa?utm_source=ghost&amp;utm_medium=referral&amp;utm_campaign=api-credit" rel="nofollow">Alexandre Debiève</a> / <a href="https://unsplash.com/?utm_source=ghost&amp;utm_medium=referral&amp;utm_campaign=api-credit" rel="nofollow">Unsplash</a></em></p>

<p>Or: the perils of blindly using AI to save some money.</p>

<p><em>IMPORTANT NOTE: while I&#39;m pointing out some specific companies in this post, it&#39;s not meant to discredit them; instead, I hope to trigger a bit of reflection on how AI should (and should not) be used, and how it should not replace your product and user experience thinking.</em></p>

<p>I admit I was kind of waiting for it to happen; bad usage of AI was going to create some problem for me one day or another.</p>

<p><strong>Full disclosure</strong>: I&#39;m not an AI skepticist, I&#39;ve got quite a bit of machine learning and data science experience, and I think the current LLM-oriented AI wave certainly has value, but I believe it&#39;s an <strong>evolution</strong> rather than the revolution that hype-riders claim it to be.</p>

<h2 id="the-facts">The facts</h2>

<p>Some days ago I was having a short vacation in <a href="https://en.wikipedia.org/wiki/Lienz" rel="nofollow">Lienz, Austria</a>, a nice small city from where you can follow a bike path through the mountains to <a href="https://en.wikipedia.org/wiki/Innichen" rel="nofollow">San Candido, Italy</a> and then return by train (or vice-versa). I still recommend the trip!</p>

<p>We were going to take the train to San Candido at 8.50am, and we arrived at the train station parking at 8.30am, so we had quite a bit of time. But: you need to pay for the parking. No problem, right? Unless the parking meter, which looked quite modern, only accepted coins: <strong>no cards and no bills.</strong> We needed to pay a total of EUR 8,00 and hadn&#39;t got enough coins. So, what? Technology to the rescue, of course: we downloaded the official parking app, as advertised on the side of the parking meter.</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/parkx.jpeg" alt=""></p>

<p><em>parking meter Parkster ad</em></p>

<p>So, I duly scanned the QR code which led me to the <a href="https://apps.apple.com/it/app/parkster-smooth-parking/id533140250" rel="nofollow">Parkster IOS app</a> . I installed it and looked for the area code in order to pay:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/parkster-inside.jpg" alt=""></p>

<p><em>parkster screenshot</em></p>

<p>I tried looking manually for the parking area; no luck. I gave location access to the app, and it worked flawlessly, the map was pointing precisely to our parking spot. <strong>And, while entering the code, I could see that some parking areas indeed existed as they were partially matching:</strong></p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/parkster-partial-1.jpg" alt=""></p>

<p><em>partial area matches</em></p>

<p>So, the app seemed to be working indeed, it was just that our parking area wasn&#39;t showing up. By then, we were risking running out of time, so we scrambled to change some coins, and we barely made it to the train. If you have access to the app in your App Store, try yourself – the behaviour is still the same as of August, 26th 2023.</p>

<h2 id="trying-to-fix-the-issue">Trying to fix the issue</h2>

<p>But I had took photos and screenshots, so, once I got back home, I sent an e-mail to the Lienz Town Hall, to Parkster customer service (the e-mail was right there in the ad) and to Lienz Tourist Information (to their credit, Parkster is the only addressee that answered so far), explaining the issue I found with a certain amount of details and proof:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine.png" alt=""></p>

<p><em>1st support request</em></p>

<p>I got this response from Parkster:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine-1.png" alt=""></p>

<p><em>answer 1</em></p>

<p>This is quite a generic answer, which is something that can happen with support requests; a “template answer” is sent at first to reduce operator burden, and only if the customer follows up then a “real operator” is involved. But the “you need to register” is either factually incorrect (I was using the app without registering) or misleading; so I followed up:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine-2.png" alt=""></p>

<p><em>followup 1</em></p>

<p>and I got this:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine-3.png" alt=""></p>

<p><em>answer 2</em></p>

<p>Now I was getting puzzled. Did they read my message? It appeared so, since the answer <em>kind</em> of made some sense. But, the area code was available both in screenshots from my first e-mail, and it was explicitly there in my followup question.</p>

<p>Then, I tried registering, and... the area codes would finally show up! During registration, I was asked “in which country you&#39;d like to park”, so I suspect that only related area codes are shown. Why? I don&#39;t know, but I don&#39;t actually care, there may be some reason. So, I had one more followup with the customer support just to explain that this “detail” isn&#39;t written anywhere and this was quite a UX problem. But at this point I had some suspicion about my support agent; it seemed like he wanted to <em>answer</em> <em>something</em> to me, but he <em>did not really care</em> how I was communicating something which should have value <em>to his own employer</em> (I sincerely doubt I was the only one who had experienced such issue, Austria is quite packed with tourists):</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine-6.png" alt=""></p>

<p><em>final interaction</em></p>

<p>that made me think “I am really chatting with a human or are they using ChatGPT to answer?!?” – I just told you I have registered and now I can find the zone! And the other “live tickets” and invoice parts are totally unrelated!</p>

<p>On such last message, I spotted a different sender than beforehand (I don&#39;t know why):</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2023/08/immagine-4.png" alt=""></p>

<p>and, ThinkOwl indeed is <em>Customer Service Software powered by AI,</em> whatever that means.</p>

<h2 id="wrapping-up">Wrapping up</h2>

<h3 id="for-the-parking-app">For the parking app</h3>

<p>Some customer was trying to help you and point out a bug or a UX problem to you.</p>

<p>An answer like “Thank you for your message. Our development team has been informed and we&#39;ll consider possible changes to our system in the future” would have both great and enough. And, the change could be something as stupid as adding “Please make sure to register for the correct country or zones won&#39;t be found” to the “not found message”.</p>

<p>AI is not really helping in this situation. It probably lets some departments to meet some cost and answer time targets, but they&#39;ll miss the opportunity to learn customer problems and improve their product and user experience.</p>

<p>Don&#39;t use AI this way. In fact, I expect AI to <em>improve</em> customer experience by empowering customer support representatives to work better and faster, answering quickly to FAQs but letting them take more time to answer more complex – but legit – answers or to interact with development teams to actually fix real, underlying issues.</p>

<h3 id="for-the-lienz-municipality">For the Lienz municipality</h3>

<p>Product thinking goes end-to-end. Installing a coin-only parking meter in 2023 (or 2020, I can&#39;t know the exact date, but it looked quite recent) because you&#39;re relying on a third-party app can be a good idea, but, then, <em>you need to test your user flow end-to-end</em>. Your city relies a lot on tourists, and you don&#39;t want to piss them off!</p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/how-to-use-ai-to-suck-at-customer-support</guid>
      <pubDate>Sat, 26 Aug 2023 13:31:47 +0000</pubDate>
    </item>
    <item>
      <title>How to do a bad comparison of cloud vs colocation</title>
      <link>https://www.franzoni.eu/alan/how-to-do-a</link>
      <description>&lt;![CDATA[&#xA;&#xA;Apples and oranges&#xA;&#xA;There&#39;s this post by Ahrefs , a SEO company from Singapore, which is getting quite a lot of shares recently.&#xA;&#xA;While nobody ever said that the cloud was cheap, it&#39;s a terrible comparison. Basically, the article enumerates what they have bought (a number of Dell servers + network equipment and colocation costs) and roughly translates those to AWS cloud services.&#xA;&#xA;Some problems:&#xA;&#xA;Apparently, all of Ahrefs servers lie in a single datacenter. Good luck with disaster recovery and availability.&#xA;The cloud is nice because, you know, it&#39;s a cloud. Scalability. Flexibility. On-demand resources. Nobody forces any customer to always rent 850 EC2 servers. Ahrefs could have used EC2 machines and cloud databases with autoscaling to pay less without getting performance problems. They could have used completely managed solutions (e.g. Lambda) to totally forget about the need to manage a server, and to get a dynamic pricing.&#xA;No operational cost is included. It&#39;s true, cloud engineers aren&#39;t cheap; but a few of them can automate and manage quite a large fleet of cloud resources. On the contrary, deployment and maintenance of such a large hardware fleet is probably more complex. What was the setup cost?&#xA;&#xA;Don&#39;t do those comparisons. Choose to compare a scenario, instead, and put some numbers to estimate the risk of problems, the cost in case of downtime (which is quite easy to prevent on the cloud), and the availability of relevant resources (are you keeping your datacenter technicians on payroll even when they don&#39;t have any work to do?).&#xA;&#xA;The cloud will almost inevitably come out more expensive than raw server/colocation costs, but quite often it&#39;s going to be cheaper than a lot of &#34;enterprise&#34; managed servers that I&#39;ve seen in my life (like, some agency managing servers in a DC for a customer).&#xA;&#xA;Towards the end of the article, Ahref acknowledges that there are tradeoffs for cloud vs on premise, so I think the first part of the article is even more insincere. They know what they&#39;re doing: it&#39;s an hybrid approch where the predictable workload is kept in a private DC, but probably there&#39;s an underlying cloud infrastructure that, if everything goes wrong (e.g. the DC burns) would be able to take 100% of the required workload. Quite a different story.&#xA;&#xA;#MostlyUnixish #cloud&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2023/03/DALL-E-2023-03-13-22.40.25---server-farm-with-one-apple-and-one-orange-close-by.jpg" alt=""></p>

<p><em>Apples and oranges</em></p>

<p>There&#39;s <a href="https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8" rel="nofollow">this post by Ahrefs</a> , a SEO company from Singapore, which is getting quite a lot of shares recently.</p>

<p>While nobody ever said that the cloud was cheap, it&#39;s a terrible comparison. Basically, the article enumerates what they have bought (a number of Dell servers + network equipment and colocation costs) and roughly translates those to AWS cloud services.</p>

<p>Some problems:</p>
<ul><li>Apparently, all of Ahrefs servers lie in a single datacenter. Good luck with disaster recovery and availability.</li>
<li>The cloud is nice because, you know, it&#39;s a cloud. Scalability. Flexibility. On-demand resources. Nobody forces any customer to always rent 850 EC2 servers. Ahrefs could have used EC2 machines and cloud databases with autoscaling to pay less without getting performance problems. They could have used completely managed solutions (e.g. Lambda) to totally forget about the need to manage a server, <em>and</em> to get a dynamic pricing.</li>
<li>No operational cost is included. It&#39;s true, cloud engineers aren&#39;t cheap; but a few of them can automate and manage quite a large fleet of cloud resources. On the contrary, deployment and maintenance of such a large hardware fleet is probably more complex. What was the setup cost?</li></ul>

<p>Don&#39;t do those comparisons. Choose to compare a scenario, instead, and put some numbers to estimate the risk of problems, the cost in case of downtime (which is quite easy to prevent on the cloud), and the availability of relevant resources (are you keeping your datacenter technicians on payroll even when they don&#39;t have any work to do?).</p>

<p>The cloud will almost inevitably come out more expensive than raw server/colocation costs, but quite often it&#39;s going to be cheaper than a lot of “enterprise” managed servers that I&#39;ve seen in my life (like, some agency managing servers in a DC for a customer).</p>

<p>Towards the end of the article, Ahref acknowledges that there are tradeoffs for cloud vs on premise, so I think the first part of the article is even more insincere. They know what they&#39;re doing: it&#39;s an hybrid approch where the predictable workload is kept in a private DC, but probably there&#39;s an underlying cloud infrastructure that, if everything goes wrong (e.g. the DC burns) would be able to take 100% of the required workload. Quite a different story.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:cloud" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">cloud</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/how-to-do-a</guid>
      <pubDate>Mon, 13 Mar 2023 21:49:05 +0000</pubDate>
    </item>
    <item>
      <title>Password requirements: myths and madness</title>
      <link>https://www.franzoni.eu/alan/password-requirements-myths-madness</link>
      <description>&lt;![CDATA[More than 11 years have passed since the venerable XKCD Password Strength strip:&#xA;&#xA;img src=&#34;https://imgs.xkcd.com/comics/passwordstrength.png&#34; &#xA;&#xA;And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:&#xA;&#xA;Single\Sign\On-1&#xA;&#xA;The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.&#xA;&#xA;I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.&#xA;&#xA;Please use a mix of uppercase and lowercase letters, digits, special chars in your password&#xA;&#xA;Usual explanation: for security reasons you must choose a strong password!&#xA;&#xA;My verdict: Misunderstanding&#xA;&#xA;The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same entropy as a shorter password from a larger charset.&#xA;&#xA;If you host a web application: Enforce a password&#39;s strength, not strange user-hostile policies. There&#39;re libraries for that (e.g. https://github.com/dropbox/zxcvbn) that work both in backend and frontend.&#xA;&#xA;You cannot use whitespace, accented letters, hyphens, \[...\] in your password&#xA;&#xA;Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like &#34;your password contains unsafe characters&#34;.&#xA;&#xA;My verdict: No real motivation beyond, possibly, incompetence&#xA;&#xA;This is a terrible one. You waste a lot of time at generating a &#34;strong password&#34; that complies with a website&#39;s policy, and then you discover that you could not use &#34;-&#34; in it. Or just whitespace for XKCD-style passwords.&#xA;&#xA;There&#39;s no technical reason for restricting any character from appearing in any password. The usual approach for handling a password is:&#xA;&#xA;take the password as a string, as the user typed it;&#xA;Prepare it (see links at the end of the paragraph);&#xA;encode with a specific, well-defined encoding (e.g. utf-8);&#xA;add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an &#34;hashed&#34; password and save that salt+hash in your storage of choice.&#xA;&#xA;What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.&#xA;&#xA;Caveats:&#xA;In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.&#xA;&#xA;If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.&#xA;&#xA;EDIT:&#xA;Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called &#34;string preparation&#34;. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:&#xA;https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html&#xA;https://www.rfc-editor.org/rfc/rfc8265&#xA;http://unicode.org/reports/tr15/\#Stabilized\Strings&#xA;&#xA;So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there could be good reasons to reject some input chars since there could be no stable encoding under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.&#xA;&#xA;You cannot enter a password with fewer than X chars&#xA;&#xA;EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET&#xA;&#xA;Usual explanation: security! I want your password to be veeeery strong!&#xA;&#xA;My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.&#xA;&#xA;This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. NIST standards even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).&#xA;&#xA;This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.&#xA;&#xA;But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the BMP, my password is probably very safe already, since there are (10\\6)\\4 possible passwords, many more than (127\\8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.&#xA;&#xA;If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.&#xA;&#xA;You cannot enter a password with more than X chars&#xA;&#xA;Usual explanation: whew. It&#39;s too large we cannot handle it.&#xA;&#xA;My verdict: Largely unmotivated unless X is quite large.&#xA;&#xA;Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.&#xA;&#xA;This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the &#34;enter your password&#34; field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.&#xA;&#xA;Caveats:&#xA;An higher length limit for passwords must should exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.&#xA;&#xA;EDIT:&#xA;A reader pointed out that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a a longer password doesn&#39;t mean it&#39;s necessarily more secure.&#xA;&#xA;If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user &#34;password too long&#34; if they&#39;re pasting into it.&#xA;&#xA;You cannot paste into this field&#xA;&#xA;Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!&#xA;&#xA;My verdict: Largely unmotivated right now, and dangerous.&#xA;&#xA;Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?&#xA;&#xA;If you host a web application: please allow pasting into password fields.&#xA;&#xA;You cannot show or copy this field&#xA;&#xA;Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.&#xA;&#xA;My verdict: Security through obscurity bad idea.&#xA;&#xA;Don&#39;t do this. While setting a password, there must be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is especially true if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.&#xA;&#xA;If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!&#xA;&#xA;EDIT:&#xA;This entry was edited after some comments. I know that&#39;s not what &#34;security through obscurity&#34; mean, it was a kind of joke about &#34;obscurating&#34; the password.&#xA;&#xA;Your password has expired&#xA;&#xA;Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.&#xA;&#xA;My verdict: Old idea&#xA;&#xA;This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation&#xA;&#xA;Too many failures - your account is locked.&#xA;&#xA;Usual explanation: we don&#39;t want your account to be bruteforced&#xA;&#xA;My verdict: Bad idea especially if &#34;too many&#34; is low, like 3 or 5, and my password is complex.&#xA;&#xA;The idea is that you &#34;protect&#34; a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.&#xA;&#xA;Account locking could be used in 2FA contexts (requires the other factor to provoke a DoS - that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts - they&#39;re too few to guess a password.&#xA;&#xA;Caveats:&#xA;Some compliance policies require locking user accounts.&#xA;&#xA;If you host a web application: consider using temporary locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).&#xA;&#xA;Final thoughts&#xA;&#xA;I really hope that passwordless login will make some steps forward in the next years (e.g. https://passkeys.dev/, https://www.yubico.com/authentication-standards/fido2/). In the meantime, I&#39;d love a &#34;password complexity api&#34; so that my browser (or password manager) can already generate the right password for a website.&#xA;&#xA;References&#xA;&#xA;https://www.auditboard.com/blog/nist-password-guidelines/&#xA;https://pages.nist.gov/800-63-3/sp800-63-3.html&#xA;&#xA;#Ollivander #security #passwords&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>More than 11 years have passed since the venerable <a href="https://xkcd.com/936/" rel="nofollow">XKCD Password Strength strip</a>:</p>

<p><img src="https://imgs.xkcd.com/comics/password_strength.png"></p>

<p>And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/12/Single_Sign_On-1.jpg" alt="Single\_Sign\_On-1"></p>

<p>The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.</p>

<p>I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.</p>

<h3 id="please-use-a-mix-of-uppercase-and-lowercase-letters-digits-special-chars-in-your-password">Please use a mix of uppercase and lowercase letters, digits, special chars in your password</h3>

<p>Usual explanation: for security reasons you must choose a strong password!</p>

<p>My verdict: <strong>Misunderstanding</strong></p>

<p>The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same <strong><a href="https://en.wikipedia.org/wiki/Entropy_%28information_theory%29" rel="nofollow">entropy</a></strong> as a shorter password from a larger charset.</p>

<p>If you host a web application: Enforce a password&#39;s <strong>strength</strong>, not strange user-hostile policies. There&#39;re libraries for that (e.g. <a href="https://github.com/dropbox/zxcvbn" rel="nofollow">https://github.com/dropbox/zxcvbn</a>) that work both in backend and frontend.</p>

<h3 id="you-cannot-use-whitespace-accented-letters-hyphens-in-your-password">You cannot use whitespace, accented letters, hyphens, [...] in your password</h3>

<p>Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like “your password contains unsafe characters”.</p>

<p>My verdict: <strong>No real motivation beyond, possibly, incompetence</strong></p>

<p>This is a terrible one. You waste a lot of time at generating a “strong password” that complies with a website&#39;s policy, and then you discover that you could not use “–” in it. Or just whitespace for XKCD-style passwords.</p>

<p><strong>There&#39;s no technical reason for restricting any character from appearing in any password.</strong> The usual approach for handling a password is:</p>
<ul><li>take the password as a string, as the user typed it;</li>
<li>Prepare it (see links at the end of the paragraph);</li>
<li>encode with a specific, well-defined encoding (e.g. utf-8);</li>
<li>add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an “hashed” password and save that salt+hash in your storage of choice.</li></ul>

<p>What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.</p>

<p>Caveats:
In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.</p>

<p>If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.</p>

<p>EDIT:
<em>Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called “string preparation”. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:</em>
<a href="https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html" rel="nofollow">https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html</a>
<a href="https://www.rfc-editor.org/rfc/rfc8265" rel="nofollow">https://www.rfc-editor.org/rfc/rfc8265</a>
<a href="http://unicode.org/reports/tr15/#Stabilized_Strings" rel="nofollow">http://unicode.org/reports/tr15/#Stabilized_Strings</a></p>

<p>So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there <em>could</em> be good reasons to reject some input chars since there could be no <em>stable encoding</em> under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.</p>

<h3 id="you-cannot-enter-a-password-with-fewer-than-x-chars">You cannot enter a password with fewer than X chars</h3>

<p><em>EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET</em></p>

<p>Usual explanation: security! I want your password to be veeeery strong!</p>

<p>My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.</p>

<p>This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. <a href="https://pages.nist.gov/800-63-3/sp800-63b.html" rel="nofollow">NIST standards</a> even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).</p>

<p>This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.</p>

<p>But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the <a href="https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane" rel="nofollow">BMP</a>, my password is probably very safe already, since there are (10**6)**4 possible passwords, many more than (127**8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.</p>

<p>If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.</p>

<h3 id="you-cannot-enter-a-password-with-more-than-x-chars">You cannot enter a password with more than X chars</h3>

<p>Usual explanation: whew. It&#39;s too large we cannot handle it.</p>

<p>My verdict: <strong>Largely unmotivated</strong> unless X is quite large.</p>

<p>Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.</p>

<p>This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the “enter your password” field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.</p>

<p>Caveats:
An higher length limit for passwords <del>must</del> <strong>should</strong> exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.</p>

<p>EDIT:
A reader <a href="https://www.sjoerdlangkemper.nl/2021/07/02/long-password-denial-of-service/" rel="nofollow">pointed out</a> that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a <a href="https://crypto.stackexchange.com/questions/98505/argon2-allows-a-huge-key-length-in-input-but-does-it-really-provide-the-securit" rel="nofollow">a longer password doesn&#39;t mean it&#39;s necessarily more secure</a>.</p>

<p>If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user “password too long” if they&#39;re pasting into it.</p>

<h3 id="you-cannot-paste-into-this-field">You cannot paste into this field</h3>

<p>Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!</p>

<p>My verdict: <strong>Largely unmotivated right now, and dangerous</strong>.</p>

<p>Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?</p>

<p>If you host a web application: please allow pasting into password fields.</p>

<h3 id="you-cannot-show-or-copy-this-field">You cannot show or copy this field</h3>

<p>Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.</p>

<p>My verdict: <del>Security through obscurity</del> bad idea.</p>

<p>Don&#39;t do this. While setting a password, there <strong>must</strong> be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is <strong>especially true</strong> if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.</p>

<p>If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!</p>

<p>EDIT:
<em>This entry was edited after some comments. I know that&#39;s not what “security through obscurity” mean, it was a kind of joke about “obscurating” the password.</em></p>

<h3 id="your-password-has-expired">Your password has expired</h3>

<p>Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.</p>

<p>My verdict: <strong>Old idea</strong></p>

<p>This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation</p>

<h3 id="too-many-failures-your-account-is-locked">Too many failures – your account is locked.</h3>

<p>Usual explanation: we don&#39;t want your account to be bruteforced</p>

<p>My verdict: <strong>Bad idea</strong> especially if “too many” is low, like 3 or 5, and my password is complex.</p>

<p>The idea is that you “protect” a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.</p>

<p>Account locking <em>could</em> be used in 2FA contexts (requires the other factor to provoke a DoS – that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts – they&#39;re too few to guess a password.</p>

<p>Caveats:
Some compliance policies require locking user accounts.</p>

<p>If you host a web application: consider using <em>temporary</em> locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).</p>

<h2 id="final-thoughts">Final thoughts</h2>

<p>I really hope that passwordless login will make some steps forward in the next years (e.g. <a href="https://passkeys.dev/" rel="nofollow">https://passkeys.dev/</a>, <a href="https://www.yubico.com/authentication-standards/fido2/" rel="nofollow">https://www.yubico.com/authentication-standards/fido2/</a>). In the meantime, I&#39;d love a “password complexity api” so that my browser (or password manager) can already generate the right password for a website.</p>

<h2 id="references">References</h2>

<p><a href="https://www.auditboard.com/blog/nist-password-guidelines/" rel="nofollow">https://www.auditboard.com/blog/nist-password-guidelines/</a>
<a href="https://pages.nist.gov/800-63-3/sp800-63-3.html" rel="nofollow">https://pages.nist.gov/800-63-3/sp800-63-3.html</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a> <a href="/alan/tag:passwords" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">passwords</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/password-requirements-myths-madness</guid>
      <pubDate>Thu, 22 Dec 2022 20:41:14 +0000</pubDate>
    </item>
    <item>
      <title>Ransomware-resistant backups with duplicity and AWS S3</title>
      <link>https://www.franzoni.eu/alan/ransomware-resistant-backups</link>
      <description>&lt;![CDATA[Ransomware and backups&#xA;&#xA;Article updated: February 5th, 2022&#xA;&#xA;Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.&#xA;&#xA;Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, cryptolocker-like malwares are spreading. You&#39;re not safe just because why should anybody hack into my little server: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.&#xA;&#xA;So: people should revise their own threat models. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a failure scenario (hardware failure, accidental deletion), not for an attack scenario. But even half-serious cybercriminals, once they get access to your server, will delete all backup sets they can access, so that their encrypt-your-data-and-ask-for-money threat actually works.&#xA;&#xA;Once upon a time&#xA;&#xA;You had backups on a tape library. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated - getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.&#xA;&#xA;Then approaches like bacula were common - the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it &#34;wipe everything&#34; so again, barring any bacula server exploit, bacula server data held backups safe.&#xA;&#xA;More recently, cloud approaches like rsync.net leverage ZFS snapshots to make remote data immutable - they can be removed only according to your retention schedule (details). BorgBase works in a similar way and has a good writeup about backup strategies. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider restore costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data - at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.&#xA;&#xA;But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: restic offers rest-server, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.&#xA;&#xA;The cloud to the rescue&#xA;&#xA;So, here we&#39;ll see how to use the good old duplicity backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.&#xA;&#xA;For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your workstation and your server, and you&#39;d like to have your server backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.&#xA;&#xA;So, make yourself a favour and use two separate machines.&#xA;&#xA;Create your S3 bucket and credentials&#xA;&#xA;Those steps must be taken from your workstation&#xA;&#xA;These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.&#xA;&#xA;Create the bucket&#xA;&#xA;Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it sample-duplicity-backup for the sake of this article) in your preferred region (I will use eu-central-1) ; and use these settings:&#xA;&#xA;Disable ACLs&#xA;&#xA;S3\bucket-1-1&#xA;&#xA;Block public access&#xA;&#xA;S3\bucket-2-1&#xA;&#xA;Enable bucket versioning&#xA;&#xA;S3\bucket-3-1&#xA;&#xA;Enable object lock&#xA;&#xA;S3\bucket-4-1&#xA;&#xA;Configure object lock&#xA;&#xA;Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.&#xA;&#xA;sample-duplicity-backup\objectlock&#xA;&#xA;Create a IAM user&#xA;&#xA;Now you need to create a suitable IAM user to be used with duplicity. Let&#39;s call this sample-duplicity-user; it should only have programmatic access, and you should attach only this policy to such user:&#xA;&#xA;{&#xA;    &#34;Version&#34;: &#34;2012-10-17&#34;,&#xA;    &#34;Statement&#34;: [&#xA;        {&#xA;            &#34;Effect&#34;: &#34;Allow&#34;,&#xA;            &#34;Action&#34;: [&#xA;                &#34;s3:PutObject&#34;,&#xA;                &#34;s3:GetObject&#34;,&#xA;                &#34;s3:ListBucketMultipartUploads&#34;,&#xA;                &#34;s3:AbortMultipartUpload&#34;,&#xA;                &#34;s3:ListBucket&#34;,&#xA;                &#34;s3:ListMultipartUploadParts&#34;&#xA;            ],&#xA;            &#34;Resource&#34;: [&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup/&#34;&#xA;            ]&#xA;        }&#xA;    ]&#xA;}&#xA;&#xA;Download the access key ID and the secret access key for such user.&#xA;&#xA;Install duplicity&#xA;&#xA;The following steps must be performed on the server&#xA;&#xA;Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported - check the homepage for further info. Also make sure the boto3 library and gnupg are available on your system.&#xA;&#xA;As an example, for recent Ubuntu versions (tested on 20.04):&#xA;&#xA;Add the stable duplicity PPA&#xA;apt -y install duplicity python3-boto3 gnupg&#xA;&#xA;Create a backup script&#xA;&#xA;Put some data in /root/data for the purpose of this test.&#xA;&#xA;Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography - support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:&#xA;&#xA;!/bin/bash -e&#xA;export PASSPHRASE=&#34;XYZXYZXYZ&#34;&#xA;export AWSACCESSKEYID=&#34;XXXXXXX&#34;&#xA;export AWSSECRETACCESSKEY=&#34;YYYYYYYYY&#34;&#xA;/usr/bin/duplicity \&#xA;--s3-european-buckets \&#xA;  --s3-use-new-style  --asynchronous-upload -v 4 \&#xA;   incr --full-if-older-than 30D \&#xA;  /root/data \&#xA;  &#34;boto3+s3://sample-duplicity-backup/data&#34;&#xA;&#xA;This command will backup the contents of /root/data to your s3 bucket, using a&#xA;data prefix.&#xA;&#xA;One detail to note: the --full-if-older 30D means: create an incremental backup&#xA;since the previous one, or create a new full backup if more than 30 days passed since our last full backup. It is essential that the number of days that we set here is smaller than the object lock days.&#xA;&#xA;So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the data directory:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket&#xA;&#xA;If you like, you can run the script again. You&#39;ll notice that duplicity only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.&#xA;&#xA;Attack scenario&#xA;&#xA;Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for sample-duplicity-user, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.&#xA;&#xA;For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same sample-duplicity-user credentials and pretend you&#39;re the badguy:&#xA;&#xA;aws configure --profile badguy&#xA;&#xA;Now, try acting as the Bad Guy:&#xA;&#xA;$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/&#xA;2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg&#xA;2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg&#xA;2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg&#xA;&#xA;He can read the backup. This is expected. But can he delete the backup?&#xA;&#xA;$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied&#xA;&#xA;The attacker cannot delete files; our object lock policy prevents that!&#xA;&#xA;But, there&#39;s still one thing that the attacker could do: overwrite existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between &#34;add an object&#34; or &#34;overwrite an existing object&#34; (and it would be quite difficult and slow to implement in a distributed storage system):&#xA;&#xA;$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Ouch! This (apparently) succeeded:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-update&#xA;&#xA;But the reality is that the original data is still there: the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the &#34;Versions&#34; tab for our modified file:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-versions&#xA;&#xA;Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the s3:ListBucketVersions permission (check the API call as well) - we don&#39;t even have it for our backup IAM user.&#xA;&#xA;Retrieving the original files&#xA;&#xA;A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:&#xA;&#xA;$ aws s3api list-object-versions --bucket sample-duplicity-backup&#xA;{&#xA;    &#34;Versions&#34;: [&#xA;        ...&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,&#xA;            &#34;Size&#34;: 50,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;wZOOamRomLYpZO5Gc.Vynrl0cP6bWq&#34;,&#xA;            &#34;IsLatest&#34;: true,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;            &#34;Size&#34;: 267,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;            &#34;IsLatest&#34;: false,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        ...&#xA;    ]&#xA;}&#xA;&#xA;Then retrieve this file in its older version:&#xA;&#xA;$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;&#xA;{&#xA;    &#34;AcceptRanges&#34;: &#34;bytes&#34;,&#xA;    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;    &#34;ContentLength&#34;: 267,&#xA;    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,&#xA;    &#34;Metadata&#34;: {}&#xA;}&#xA;$ ls .gpg&#xA;duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.&#xA;&#xA;Wrapping up&#xA;&#xA;I hope this article can be useful from an implementation standpoint, but the most important takeaway is: always think about your threat model every time you take a decision. Reason in the terms of what an attacker could do when entering your system; security is never binary.&#xA;&#xA;And, please: don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set, like -rwx------ 1 root root 365 Dec  9 21:52 duplicity_run. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!&#xA;&#xA;An exercise&#xA;&#xA;An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.&#xA;&#xA;To my readers: since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.&#xA;&#xA;Updates &amp; footnotes&#xA;&#xA;I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the HN crowd as usual for commenting.&#xA;&#xA;Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.&#xA;&#xA;So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):&#xA;&#xA;You should remember to periodically verify your backups, usually by restoring them - this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, make sure you verify backups before deleting old data.&#xA;Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even replicate buckets between accounts&#xA;Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend setting a proper budget alert for your AWS account and maybe an alert for excessive S3 data usage&#xA;Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job - so you get errors if it fails - and to periodically verify them.&#xA;If you&#39;ve got a lot of data and not a lot of bandwidth - hence the initial backup may take you a lot time - make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.&#xA;&#xA;#Ollivander #backups #security&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<h2 id="ransomware-and-backups">Ransomware and backups</h2>

<p><em>Article updated: February 5th, 2022</em></p>

<p>Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.</p>

<p>Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, <a href="https://en.wikipedia.org/wiki/CryptoLocker" rel="nofollow">cryptolocker-like</a> malwares are spreading. You&#39;re not safe just because <em>why should anybody hack into my little server</em>: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.</p>

<p>So: <strong>people should revise their own threat models</strong>. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a <strong>failure scenario</strong> (hardware failure, accidental deletion), not for an <strong>attack scenario</strong>. But even half-serious cybercriminals, once they get access to your server, <strong>will delete all backup sets they can access</strong>, so that their encrypt-your-data-and-ask-for-money threat actually works.</p>

<h2 id="once-upon-a-time">Once upon a time</h2>

<p>You had backups on a <a href="https://en.wikipedia.org/wiki/Tape_drive" rel="nofollow">tape library</a>. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated – getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.</p>

<p>Then approaches like <a href="https://www.bacula.org/" rel="nofollow">bacula</a> were common – the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it “wipe everything” so again, barring any bacula server exploit, bacula server data held backups safe.</p>

<p>More recently, cloud approaches like <a href="https://www.rsync.net/" rel="nofollow">rsync.net</a> leverage ZFS snapshots to make remote data immutable – they can be removed only according to your retention schedule (<a href="https://www.rsync.net/resources/howto/snapshots.html" rel="nofollow">details</a>). <a href="https://www.borgbase.com/" rel="nofollow">BorgBase</a> works in a similar way and has a <a href="https://docs.borgbase.com/strategy/" rel="nofollow">good writeup about backup strategies</a>. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider <em>restore</em> costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data – at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.</p>

<p>But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: <a href="https://restic.net/" rel="nofollow">restic</a> offers <a href="https://github.com/restic/rest-server" rel="nofollow">rest-server</a>, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.</p>

<h2 id="the-cloud-to-the-rescue">The cloud to the rescue</h2>

<p>So, here we&#39;ll see how to use the good old <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">duplicity</a> backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.</p>

<p>For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your <strong>workstation</strong> and your <strong>server</strong>, and you&#39;d like to have your <strong>server</strong> backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.</p>

<p>So, make yourself a favour and use two separate machines.</p>

<h2 id="create-your-s3-bucket-and-credentials">Create your S3 bucket and credentials</h2>

<p><em>Those steps must be taken from your workstation</em></p>

<p>These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.</p>

<h3 id="create-the-bucket">Create the bucket</h3>

<p>Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it <em>sample-duplicity-backup</em> for the sake of this article) in your preferred region (I will use <em>eu-central-1</em>) ; and use these settings:</p>

<h4 id="disable-acls">Disable ACLs</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-1-1.jpg" alt="S3\_bucket-1-1"></p>

<h4 id="block-public-access">Block public access</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-2-1.jpg" alt="S3\_bucket-2-1"></p>

<h4 id="enable-bucket-versioning">Enable bucket versioning</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-3-1.jpg" alt="S3\_bucket-3-1"></p>

<h4 id="enable-object-lock">Enable object lock</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-4-1.jpg" alt="S3\_bucket-4-1"></p>

<h4 id="configure-object-lock">Configure object lock</h4>

<p>Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_objectlock.jpg" alt="sample-duplicity-backup\_objectlock"></p>

<h3 id="create-a-iam-user">Create a IAM user</h3>

<p>Now you need to <a href="https://console.aws.amazon.com/iamv2/home#/users" rel="nofollow">create a suitable IAM user</a> to be used with duplicity. Let&#39;s call this <code>sample-duplicity-user</code>; it should only have programmatic access, and you should attach <strong>only</strong> this policy to such user:</p>

<pre><code>{
    &#34;Version&#34;: &#34;2012-10-17&#34;,
    &#34;Statement&#34;: [
        {
            &#34;Effect&#34;: &#34;Allow&#34;,
            &#34;Action&#34;: [
                &#34;s3:PutObject&#34;,
                &#34;s3:GetObject&#34;,
                &#34;s3:ListBucketMultipartUploads&#34;,
                &#34;s3:AbortMultipartUpload&#34;,
                &#34;s3:ListBucket&#34;,
                &#34;s3:ListMultipartUploadParts&#34;
            ],
            &#34;Resource&#34;: [
                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,
                &#34;arn:aws:s3:::sample-duplicity-backup/*&#34;
            ]
        }
    ]
}
</code></pre>

<p>Download the access key ID and the secret access key for such user.</p>

<h2 id="install-duplicity">Install duplicity</h2>

<p><em>The following steps must be performed on the server</em></p>

<p>Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported – check <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">the homepage</a> for further info. Also make sure the <code>boto3</code> library and <code>gnupg</code> are available on your system.</p>

<p>As an example, for recent Ubuntu versions (tested on 20.04):</p>
<ul><li>Add the <a href="https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git" rel="nofollow">stable duplicity PPA</a></li>
<li><code>apt -y install duplicity python3-boto3 gnupg</code></li></ul>

<h2 id="create-a-backup-script">Create a backup script</h2>

<p>Put some data in <code>/root/data</code> for the purpose of this test.</p>

<p>Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography – support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:</p>

<pre><code>#!/bin/bash -e
export PASSPHRASE=&#34;XYZXYZXYZ&#34;
export AWS_ACCESS_KEY_ID=&#34;XXXXXXX&#34;
export AWS_SECRET_ACCESS_KEY=&#34;YYYYYYYYY&#34;
/usr/bin/duplicity \
--s3-european-buckets \
  --s3-use-new-style  --asynchronous-upload -v 4 \
   incr --full-if-older-than 30D \
  /root/data \
  &#34;boto3+s3://sample-duplicity-backup/data&#34;
</code></pre>

<p>This command will backup the contents of <code>/root/data</code> to your s3 bucket, using a
<code>data</code> prefix.</p>

<p>One detail to note: the <code>--full-if-older 30D</code> means: <em>create an incremental backup
since the previous one, or create a new full backup if more than 30 days passed since our last full backup</em>. It is <strong>essential</strong> that the number of days that we set here is smaller than the object lock days.</p>

<p>So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the <em>data</em> directory:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket"></p>

<p>If you like, you can run the script again. You&#39;ll notice that duplicity <em>only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.</em></p>

<h2 id="attack-scenario">Attack scenario</h2>

<p>Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for <code>sample-duplicity-user</code>, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.</p>

<p>For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same <code>sample-duplicity-user</code> credentials and pretend you&#39;re the badguy:</p>

<p><code>aws configure --profile badguy</code></p>

<p>Now, try acting as the Bad Guy:</p>

<pre><code>$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/
2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg
2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg
2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg
</code></pre>

<p>He can read the backup. This is expected. But can he <strong>delete</strong> the backup?</p>

<pre><code>$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied
</code></pre>

<p>The attacker cannot delete files; our object lock policy prevents that!</p>

<p>But, there&#39;s still one thing that the attacker could do: <strong>overwrite</strong> existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between “add an object” or “overwrite an existing object” (and it would be quite difficult and slow to implement in a distributed storage system):</p>

<pre><code>$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Ouch! This (apparently) succeeded:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-update.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-update"></p>

<p>But the reality is that <strong>the original data is still there:</strong> the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the “Versions” tab for our modified file:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-versions.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-versions"></p>

<p>Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the <em>s3:ListBucketVersions</em> permission (check the <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html" rel="nofollow">API call as well</a>) – we don&#39;t even have it for our backup IAM user.</p>

<h2 id="retrieving-the-original-files">Retrieving the original files</h2>

<p>A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:</p>

<pre><code>$ aws s3api list-object-versions --bucket sample-duplicity-backup
{
    &#34;Versions&#34;: [
        ...
        {
            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,
            &#34;Size&#34;: 50,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;w_ZOOamRomLYpZO5Gc.Vy_nrl0cP6bWq&#34;,
            &#34;IsLatest&#34;: true,
            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        {
            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
            &#34;Size&#34;: 267,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
            &#34;IsLatest&#34;: false,
            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        ...
    ]
}
</code></pre>

<p>Then retrieve this file in its older version:</p>

<pre><code>$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;
{
    &#34;AcceptRanges&#34;: &#34;bytes&#34;,
    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
    &#34;ContentLength&#34;: 267,
    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,
    &#34;Metadata&#34;: {}
}
$ ls *.gpg
duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.</p>

<h2 id="wrapping-up">Wrapping up</h2>

<p>I hope this article can be useful from an implementation standpoint, but the most important takeaway is: <strong>always think about your threat model every time you take a decision</strong>. Reason in the terms of what an attacker could do when entering your system; <strong>security is never binary</strong>.</p>

<p>And, please: <strong>don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set</strong>, like <code>-rwx------ 1 root root 365 Dec  9 21:52 duplicity_run</code>. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!</p>

<h3 id="an-exercise">An exercise</h3>

<p>An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.</p>

<p><em>To my readers:</em> since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.</p>

<h3 id="updates-footnotes">Updates &amp; footnotes</h3>

<p>I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the <a href="https://news.ycombinator.com/item?id=30105538" rel="nofollow">HN crowd</a> as usual for commenting.</p>

<p>Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.</p>

<p>So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):</p>
<ul><li>You should remember to periodically verify your backups, usually by restoring them – this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, <strong>make sure you verify backups before deleting old data</strong>.</li>
<li>Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html" rel="nofollow">replicate buckets between accounts</a></li>
<li>Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/" rel="nofollow">setting a proper budget alert for your AWS account</a> and maybe an <a href="https://stackoverflow.com/questions/51541751/alarm-when-object-size-in-s3-bucket-exceeds-threshold" rel="nofollow">alert for excessive S3 data usage</a></li>
<li>Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job – so you get errors if it fails – and to periodically verify them.</li>
<li>If you&#39;ve got a lot of data and not a lot of bandwidth – hence the initial backup may take you a lot time – make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.</li></ul>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:backups" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">backups</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/ransomware-resistant-backups</guid>
      <pubDate>Thu, 27 Jan 2022 19:50:24 +0000</pubDate>
    </item>
    <item>
      <title>Log4j haters: just STFU</title>
      <link>https://www.franzoni.eu/alan/log4j-haters-please</link>
      <description>&lt;![CDATA[I think the behaviour of many people towards log4j developers and towards the project is simply ridiculous. I understand the memes; it&#39;s the internet, after all. But I can read posts and tweets by many IT professionals - developers, managers, security engineers - that treat the log4j project and the people that work on it as absolute shit.&#xA;&#xA;Such library is being used by (probably) thousands of projects since ~20 years, and it&#39;s available for free. The impact of this vulnerability is a testimony of Log4j success.&#xA;&#xA;So, my dear haters, I need to inform you that Log4j comes with a license that says &#34;You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License&#34;. You shouldn&#39;t blame Log4j developers; you should blame all the users and developers who chose it and used it, for free. Because they accepted that license. And you should blame all those people that didn&#39;t realize the first fix was incomplete. Log4j developers had absolutely no obligation.&#xA;&#xA;If I ever release some open source project that achieves the same success as Log4j, and, at any time, gets a similarly dangerous vulnerability, I&#39;ll make sure not to release the hotfix for free. You&#39;ll be able to get it with a special, one-time, paid-for commercial license. The community won&#39;t love me anymore, but I&#39;ll probably be able to retire in some nice place with all the money that I&#39;ll get for sure (because you won&#39;t rely on a patch from a random stranger on the internet, right? You want the fix from me, and you don&#39;t want to check what&#39;s in it).&#xA;&#xA;P.S.&#xA;&#xA;I&#39;ve read very few articles about how bad the disclosure process for Log4shell was. Apache Software Foundation had less than three weeks to patch before the disclosure went completely public (Google Project Zero usually has a 90 days deadline), and there&#39;re articles indicating that some vendors detected active exploitation just a few days after the first private report. I&#39;d speculate somebody tried to get rich here by selling a very interesting 0-day.&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>I think the behaviour of many people towards log4j developers and towards the project is simply ridiculous. I understand the memes; it&#39;s the internet, after all. But I can read posts and tweets by many IT professionals – developers, managers, security engineers – that treat the log4j project and the people that work on it as absolute shit.</p>

<p>Such library is being used by (probably) thousands of projects since ~20 years, and it&#39;s available for free. <strong>The impact of this vulnerability is a testimony of Log4j success</strong>.</p>

<p>So, my dear haters, I need to inform you that Log4j comes with a license that says <em>“You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License”</em>. You shouldn&#39;t blame Log4j developers; you should blame all the users and developers who chose it and used it, for free. Because they accepted that license. And you should blame all those people that didn&#39;t realize the first fix was incomplete. Log4j developers had absolutely no obligation.</p>

<p>If I ever release some open source project that achieves the same success as Log4j, and, at any time, gets a similarly dangerous vulnerability, <strong>I&#39;ll make sure not to release the hotfix for free</strong>. You&#39;ll be able to get it with a special, one-time, paid-for commercial license. The community won&#39;t love me anymore, but I&#39;ll probably be able to retire in some nice place with all the money that I&#39;ll get for sure (because you won&#39;t rely on a patch from a random stranger on the internet, right? You want the fix <em>from me</em>, and you <em>don&#39;t want to check what&#39;s in it).</em></p>

<p>P.S.</p>

<p>I&#39;ve read very few articles about how bad the disclosure process for Log4shell was. Apache Software Foundation had less than three weeks to patch before the disclosure went completely public (Google Project Zero usually has a 90 days deadline), and there&#39;re articles indicating that some vendors detected active exploitation just a few days after the first private report. I&#39;d speculate somebody tried to get rich here by selling a very interesting 0-day.</p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/log4j-haters-please</guid>
      <pubDate>Thu, 16 Dec 2021 21:25:01 +0000</pubDate>
    </item>
    <item>
      <title>Consistent Hashing for Dummies</title>
      <link>https://www.franzoni.eu/alan/consistent-hashing-for-dummies</link>
      <description>&lt;![CDATA[Today I&#39;ll discuss about an interesting concept: consistent hashing. It&#39;s a widely employed technique to properly perform sharding in distributed storage systems. I&#39;m not aiming at a rigorous explanation (please don&#39;t use the raw snippets I provide in production code!), but I hope I can make the concept simple enough.&#xA;&#xA;The problem&#xA;&#xA;What problem are we aiming to solve?&#xA;Let&#39;s suppose we need to handle more data than it&#39;s possible to store in a single server. What&#39;s the problem? Just create a number of shards and distribute the data!&#xA;&#xA;But, then, clients need to know where to connect in order to store or retrieve a value. So we pick a primary key and feed it into a function, an hashing function in fact, since it maps our data to a fixed size output; such function will tell us where to connect, by returning a shard index so that 0 &lt;= shardindex &lt; shardcount, e.g:&#xA;&#xA;def getshard(primarykey, shardcount) -  int&#xA;    pass&#xA;&#xA;Let&#39;s suppose our primary key is a 64-bit unsigned integer, and that we want to create 10 shards. Our first thought could be: let&#39;s split the keyspace in ten consecutive parts of (almost) identical size, check where our primary key belongs, and return the position of such part as the shard index, that is:&#xA;&#xA;MAXUINT64 = 2*64 - 1&#xA;def getshardlinear(primarykey: int, shardcount: int):&#xA;    shardsize = -(-MAXUINT64 // shardcount) # ceil division, we don&#39;t want the shard size to be accidentally too small&#xA;    return primarykey // shardsize&#xA;&#xA;This would work. But there&#39;s a problem:&#xA;What happens if there&#39;s a lot of data with a primary key around a certain value, and very little elsewhere? We&#39;ll have one or two shards doing far too much work, while the others sit idle.&#xA;&#xA;Modulo to the rescue! What if we just do primarykey modulo shardcount?&#xA;&#xA;def getshardmod(primarykey: int, shardcount: int):&#xA;    return primarykey % shardcount&#xA;&#xA;Looks nice! Unless the primary keys are sorted modulo-wise, it will probably distribute our load evenly.&#xA;&#xA;But then, what happens if we discover that our system gets overloaded, and we&#39;d like to add another shard? Our indexes will change, and we&#39;ll need to move some data around. But how much data, and which?.&#xA;&#xA;With the linear sharding, the shard size will shrink, we&#39;ll need to recalculate the shard index, and probably move a lot of objects around for most shards; many objects that were towards the end of shard 0 will be shifted to shard 1, and so on; that&#39;s quite a lot of work:&#xA;&#xA;tenshardsize = -(-MAXUINT64 // 10)&#xA;for n in range(tenshardsize-3, tenshardsize):&#xA;    print(&#34;10-shard: {} -  11-shard: {}&#34;.format(getshardlinear(n, 10), getshardlinear(n, 11)))&#xA;&#xA;for n in range(tenshardsize2-3, tenshardsize2):&#xA;    print(&#34;10-shard: {} -  11-shard: {}&#34;.format(getshardlinear(n, 10), getshardlinear(n, 11)))&#xA;&#xA;Remember, when using ten shards, indexes go from 0 to 9; when using eleven, indexes go from 0 to 10.&#xA;&#xA;Output:&#xA;&#xA;10-shard: 0 -  11-shard: 1&#xA;10-shard: 0 -  11-shard: 1&#xA;10-shard: 0 -  11-shard: 1&#xA;10-shard: 1 -  11-shard: 2&#xA;10-shard: 1 -  11-shard: 2&#xA;10-shard: 1 -  11-shard: 2&#xA;&#xA;But even the modulo sharding won&#39;t help us: the shard index key will change for most values.&#xA;&#xA;for n in range(20, 30):&#xA;    print(&#34;10-shard: {} -  11-shard: {}&#34;.format(getshardmod(n, 10), getshardmod(n, 11)))&#xA;&#xA;Output:&#xA;&#xA;10-shard: 0 -  11-shard: 9&#xA;10-shard: 1 -  11-shard: 10&#xA;10-shard: 2 -  11-shard: 0&#xA;10-shard: 3 -  11-shard: 1&#xA;10-shard: 4 -  11-shard: 2&#xA;10-shard: 5 -  11-shard: 3&#xA;10-shard: 6 -  11-shard: 4&#xA;10-shard: 7 -  11-shard: 5&#xA;10-shard: 8 -  11-shard: 6&#xA;10-shard: 9 -  11-shard: 7&#xA;&#xA;The solution&#xA;&#xA;So, the properties we&#39;d like to get from our ideal hashing function are balance - objects should be distributed evenly across our shards, and monotonicity - if a shard is added, objects should flow only from existing shards to the new one; there should be no need of internal reshuffling.&#xA;&#xA;And, guess what? That&#39;s exactly what consistent hashing does!&#xA;&#xA;A simple implementation of this algorithm goes like that:&#xA;&#xA;Python implementation by Peter Lithammer&#xA;def getshardlampingveach(primarykey: int, shardcount: int):&#xA;    b, j = -1, 0.0&#xA;&#xA;    if shardcount &lt; 1:&#xA;        raise ValueError(&#xA;            f&#34;&#39;numbuckets&#39; must be a positive number, got {shardcount}&#34;&#xA;        )&#xA;&#xA;    while j &lt; shardcount:&#xA;        b = int(j)&#xA;        primarykey = ((primarykey  int(2862933555777941757)) + 1) &amp; 0xFFFFFFFFFFFFFFFF&#xA;        j = float(b + 1)  (float(1 &lt; 31) / float((primarykey   33) + 1))&#xA;&#xA;    return int(b)&#xA;&#xA;See it in action:&#xA;&#xA;import random&#xA;random.seed(1024910)&#xA;movedtonew = 0&#xA;for x in range(0, 10000):&#xA;    n = random.randrange(0, 264)&#xA;    tenshard = getshardlampingveach(n, 10)&#xA;    elevenshard = getshardlampingveach(n, 11)&#xA;    if tenshard != elevenshard:&#xA;        if elevenshard != 10:&#xA;            raise ValueError(&#34;object flows to non-new index&#34;)&#xA;        movedtonew += 1&#xA;&#xA;print(f&#34;{movedto_new} objects changed index to 10&#34;)&#xA;&#xA;Output:&#xA;&#xA;898 objects changed index to 10&#xA;&#xA;We picked 10,000 random indexes. As you can see, there were no objects whose shard index changed to a value different than 10, the newly-added shard index, and a reasonable amount of objects (quite close to 1/11 of 10,000 objects, in fact) were moved to the new shard index.&#xA;&#xA;Explaining how this algorithm works is beyond the scope of this post; take a look at the last paper in the references if you&#39;re interested! But I hope you now understand what consistent hashing does: a consistent hashing function maps its input to evenly distributed outputs, and if the number of shards changes slightly, the output location changes only slightly*. I haven&#39;t tested how the Lamping-Veach algorithm behaves if you wildly modify the number of shards (e.g. go from 10 to 20).&#xA;&#xA;References&#xA;&#xA;Source for the Python implementation for Lamping-Veach consistent hashing&#xA;Karger, Lehman, Leighton, et al: Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web. This is the original paper which introduced consistent hashing to the world. The original algorithm is somewhat more complex but exhibits additional properties, since it was originally designed for situations where each client could have a different view of what working shards were.&#xA;Lamping, Veach: A Fast, Minimal Memory, Consistent Hash Algorithm. This paper introduces the original C version of the aforementioned consistent hashing algorithm, along with an explanation of how it works.&#xA;&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>Today I&#39;ll discuss about an interesting concept: consistent hashing. It&#39;s a widely employed technique to properly perform <em>sharding</em> in distributed storage systems. I&#39;m not aiming at a rigorous explanation (please don&#39;t use the raw snippets I provide in production code!), but I hope I can make the concept simple enough.</p>

<h2 id="the-problem">The problem</h2>

<p>What problem are we aiming to solve?
Let&#39;s suppose we need to handle more data than it&#39;s possible to store in a single server. What&#39;s the problem? Just create a number of shards and distribute the data!</p>

<p>But, then, clients need to know where to connect in order to store or retrieve a value. So we pick a primary key and feed it into a function, an hashing function in fact, since it maps our data to a fixed size output; such function will tell us where to connect, by returning a shard index so that <code>0 &lt;= shard_index &lt; shard_count</code>, e.g:</p>

<pre><code class="language-python">def get_shard(primary_key, shard_count) -&gt; int
    pass
</code></pre>

<p>Let&#39;s suppose our primary key is a 64-bit unsigned integer, and that we want to create 10 shards. Our first thought could be: let&#39;s split the keyspace in ten consecutive parts of (almost) identical size, check where our primary key belongs, and return the position of such part as the shard index, that is:</p>

<pre><code class="language-python">MAX_UINT64 = 2**64 - 1
def get_shard_linear(primary_key: int, shard_count: int):
    shard_size = -(-MAX_UINT64 // shard_count) # ceil division, we don&#39;t want the shard size to be accidentally too small
    return primary_key // shard_size
</code></pre>

<p>This would work. But there&#39;s a problem:
What happens if there&#39;s a lot of data with a primary key around a certain value, and very little elsewhere? We&#39;ll have one or two shards doing far too much work, while the others sit idle.</p>

<p>Modulo to the rescue! What if we just do <code>primary_key modulo shard_count</code>?</p>

<pre><code class="language-python">def get_shard_mod(primary_key: int, shard_count: int):
    return primary_key % shard_count
</code></pre>

<p>Looks nice! Unless the primary keys are sorted modulo-wise, it will probably distribute our load evenly.</p>

<p>But then, what happens if we discover that our system gets overloaded, and we&#39;d like to add another shard? Our indexes will change, and we&#39;ll need to move some data around. But <em>how much data, and which?</em>.</p>

<p>With the linear sharding, the shard size will shrink, we&#39;ll need to recalculate the shard index, and probably move a lot of objects around for most shards; many objects that were towards the end of shard 0 will be shifted to shard 1, and so on; that&#39;s quite a lot of work:</p>

<pre><code class="language-python">ten_shard_size = -(-MAX_UINT64 // 10)
for n in range(ten_shard_size-3, ten_shard_size):
    print(&#34;10-shard: {} -&gt; 11-shard: {}&#34;.format(get_shard_linear(n, 10), get_shard_linear(n, 11)))

for n in range(ten_shard_size*2-3, ten_shard_size*2):
    print(&#34;10-shard: {} -&gt; 11-shard: {}&#34;.format(get_shard_linear(n, 10), get_shard_linear(n, 11)))
</code></pre>

<p>Remember, when using ten shards, indexes go from 0 to 9; when using eleven, indexes go from 0 to 10.</p>

<p>Output:</p>

<pre><code>10-shard: 0 -&gt; 11-shard: 1
10-shard: 0 -&gt; 11-shard: 1
10-shard: 0 -&gt; 11-shard: 1
10-shard: 1 -&gt; 11-shard: 2
10-shard: 1 -&gt; 11-shard: 2
10-shard: 1 -&gt; 11-shard: 2
</code></pre>

<p>But even the modulo sharding won&#39;t help us: the shard index key will change for most values.</p>

<pre><code class="language-python">for n in range(20, 30):
    print(&#34;10-shard: {} -&gt; 11-shard: {}&#34;.format(get_shard_mod(n, 10), get_shard_mod(n, 11)))
</code></pre>

<p>Output:</p>

<pre><code>10-shard: 0 -&gt; 11-shard: 9
10-shard: 1 -&gt; 11-shard: 10
10-shard: 2 -&gt; 11-shard: 0
10-shard: 3 -&gt; 11-shard: 1
10-shard: 4 -&gt; 11-shard: 2
10-shard: 5 -&gt; 11-shard: 3
10-shard: 6 -&gt; 11-shard: 4
10-shard: 7 -&gt; 11-shard: 5
10-shard: 8 -&gt; 11-shard: 6
10-shard: 9 -&gt; 11-shard: 7
</code></pre>

<h2 id="the-solution">The solution</h2>

<p>So, the properties we&#39;d like to get from our ideal hashing function are <strong>balance</strong> – objects should be distributed evenly across our shards, and <strong>monotonicity</strong> – if a shard is added, objects should flow only from existing shards to the new one; there should be no need of internal reshuffling.</p>

<p>And, guess what? That&#39;s exactly what consistent hashing does!</p>

<p>A simple implementation of this algorithm goes like that:</p>

<pre><code class="language-python"># Python implementation by Peter Lithammer
def get_shard_lamping_veach(primary_key: int, shard_count: int):
    b, j = -1, 0.0

    if shard_count &lt; 1:
        raise ValueError(
            f&#34;&#39;num_buckets&#39; must be a positive number, got {shard_count}&#34;
        )

    while j &lt; shard_count:
        b = int(j)
        primary_key = ((primary_key * int(2862933555777941757)) + 1) &amp; 0xFFFFFFFFFFFFFFFF
        j = float(b + 1) * (float(1 &lt;&lt; 31) / float((primary_key &gt;&gt; 33) + 1))

    return int(b)
</code></pre>

<p>See it in action:</p>

<pre><code class="language-python">import random
random.seed(1024910)
moved_to_new = 0
for x in range(0, 10000):
    n = random.randrange(0, 2**64)
    ten_shard = get_shard_lamping_veach(n, 10)
    eleven_shard = get_shard_lamping_veach(n, 11)
    if ten_shard != eleven_shard:
        if eleven_shard != 10:
            raise ValueError(&#34;object flows to non-new index&#34;)
        moved_to_new += 1

print(f&#34;{moved_to_new} objects changed index to 10&#34;)
</code></pre>

<p>Output:</p>

<pre><code>898 objects changed index to 10
</code></pre>

<p>We picked 10,000 random indexes. As you can see, <strong>there were no objects whose shard index changed to a value different than 10, the newly-added shard index</strong>, and a reasonable amount of objects (quite close to 1/11 of 10,000 objects, in fact) were moved to the new shard index.</p>

<p>Explaining <em>how</em> this algorithm works is beyond the scope of this post; take a look at the last paper in the references if you&#39;re interested! But I hope you now understand <em>what</em> consistent hashing does: <em>a consistent hashing function maps its input to evenly distributed outputs, and if the number of shards changes slightly, the output location changes only slightly</em>. I haven&#39;t tested how the Lamping-Veach algorithm behaves if you wildly modify the number of shards (e.g. go from 10 to 20).</p>

<h2 id="references">References</h2>
<ul><li>Source for the <a href="https://github.com/lithammer/python-jump-consistent-hash/blob/master/src/jump/__init__.py" rel="nofollow">Python implementation for Lamping-Veach consistent hashing</a></li>
<li>Karger, Lehman, Leighton, et al: <a href="https://www.cs.princeton.edu/courses/archive/fall09/cos518/papers/chash.pdf" rel="nofollow">Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web</a>. This is the original paper which introduced consistent hashing to the world. The original algorithm is somewhat more complex but exhibits additional properties, since it was originally designed for situations where each client could have a different view of what working shards were.</li>
<li>Lamping, Veach: <a href="https://arxiv.org/ftp/arxiv/papers/1406/1406.2294.pdf" rel="nofollow">A Fast, Minimal Memory, Consistent Hash Algorithm</a>. This paper introduces the original C version of the aforementioned consistent hashing algorithm, along with an explanation of how it works.</li></ul>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/consistent-hashing-for-dummies</guid>
      <pubDate>Thu, 28 Oct 2021 12:28:02 +0000</pubDate>
    </item>
    <item>
      <title>The curse of the downvote</title>
      <link>https://www.franzoni.eu/alan/the-curse-of-the-downvote</link>
      <description>&lt;![CDATA[&#xA;&#xA;I don&#39;t have strong opinions on Facebook - I&#39;m not even a user anymore - but I think that the &#34;like/dislike&#34; mania is going a bit too far. I&#39;ve read yesterday that an engineer was fired from Facebook for having a YouTube channel, but that&#39;s beyond the scope of my post today. In one of his videos, he says that FB culture is driven by likes - you need to be popular, not to be good.&#xA;&#xA;Well, I actually think that may be appropriate for Facebook. Eat your own dogfood - isn&#39;t that one of the golden rules for just any product?&#xA;&#xA;But, what I hate nowadays is the curse of the downvote (or the upvote). It&#39;s everywhere: on Facebook, on Reddit, even on Hacker News - you get downvoted, then probably your comment is hidden from most people. The most upvoted comment (or article) gets more impressions.&#xA;&#xA;But what does an upvote, or a like, or a share, or a retweet, or a downvote, or a flag, actually mean? I think we&#39;ve lost the context.&#xA;&#xA;When I still was on Facebook, and some of my friends shared an hoax or just some falsehood, I was usually quick to point that out - with due references. Then they just told me &#34;hey, I shared the article. I&#39;m not the one who wrote that. I may not even agree with that&#34;. So... what does &#34;share&#34; mean?&#xA;&#xA;I think we need better connections for our actions. What&#39;s an upvote (or a retweet or a reshare) ? Does it mean &#34;I agree&#34;? Does it mean &#34;I have reasons to believe the poster is factually correct&#34;? Does it mean &#34;I like it for no particular reason&#34;?&#xA;&#xA;What&#39;s a downvote? Does it mean &#34;I think the poster is factually incorrect&#34;? Or &#34;I don&#39;t like this opinion&#34;?&#xA;&#xA;What&#39;s a flag (in HN) or a report in other sites ? Does it mean &#34;The poster suggests something illegal&#34;? Does it mean &#34;I think it&#39;s an hoax, fake news, just trolling&#34;? Or it just means &#34;I wouldn&#39;t like to see it there, throw it away&#34;. What does that REALLY imply?&#xA;&#xA;Yes, maybe that&#39;s complex. &#34;Upvote&#34; or &#34;Downvote&#34; seems easier. But that&#39;s not just our opinion anymore. Upvotes and downvotes do shape conversations and discussions, and using them unwisely just perpetrates echo chambers and unhealthy ideological silos.&#xA;&#xA;I think we can - and we should - do better. Much better.&#xA;&#xA;Maybe we should show comments / articles in a random fashion, and let people &#34;grade&#34; them by quality of the opinion and/or correctness. And only after a while, when commenting stops, we should create a &#34;top ten&#34; for our comments; or something like that; we should make sure that more than one idea gets exposure, rather than going from the start with a &#34;winner takes it all&#34; mentality. It&#39;s not good for anybody.&#xA;&#xA;Photo by Fabian Gieske on Unsplash&#xA;&#xA;#Ollivander #rants #thoughts&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://images.unsplash.com/photo-1551730459-92db2a308d6a?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjExNzczfQ" alt=""></p>

<p>I don&#39;t have strong opinions on Facebook – I&#39;m not even a user anymore – but I think that the “like/dislike” mania is going a bit too far. I&#39;ve read yesterday that an engineer <a href="https://www.cnbc.com/2019/09/19/ex-facebook-engineer-patrick-shyu-makes-fun-of-company-on-youtube.html" rel="nofollow">was fired from Facebook for having a YouTube channel</a>, but that&#39;s beyond the scope of my post today. In one of his videos, he says that FB culture is driven by likes – you need to be popular, not to be good.</p>

<p>Well, I actually think that may be appropriate for Facebook. Eat your own dogfood – isn&#39;t that one of the golden rules for just any product?</p>

<p>But, what I hate nowadays is the curse of the downvote (or the upvote). It&#39;s everywhere: on Facebook, on Reddit, even on Hacker News – you get downvoted, then probably your comment is hidden from most people. The most upvoted comment (or article) gets more impressions.</p>

<p>But what does an upvote, or a like, or a share, or a retweet, or a downvote, or a flag, actually mean? I think we&#39;ve lost the context.</p>

<p>When I still was on Facebook, and some of my friends shared an hoax or just some falsehood, I was usually quick to point that out – with due references. Then they just told me “hey, I shared the article. I&#39;m not the one who wrote that. I may not even agree with that”. So... what does “share” mean?</p>

<p>I think we need better connections for our actions. What&#39;s an upvote (or a retweet or a reshare) ? Does it mean “I agree”? Does it mean “I have reasons to believe the poster is factually correct”? Does it mean “I like it for no particular reason”?</p>

<p>What&#39;s a downvote? Does it mean “I think the poster is factually incorrect”? Or “I don&#39;t like this opinion”?</p>

<p>What&#39;s a flag (in HN) or a report in other sites ? Does it mean “The poster suggests something illegal”? Does it mean “I think it&#39;s an hoax, fake news, just trolling”? Or it just means “I wouldn&#39;t like to see it there, throw it away”. What does that REALLY imply?</p>

<p>Yes, maybe that&#39;s complex. “Upvote” or “Downvote” seems easier. <strong>But that&#39;s not just our opinion anymore.</strong> Upvotes and downvotes do shape conversations and discussions, and using them unwisely just perpetrates echo chambers and unhealthy ideological silos.</p>

<p>I think we can – <strong>and we should -</strong> do better. Much better.</p>

<p>Maybe we should show comments / articles in a random fashion, and let people “grade” them by quality of the opinion and/or correctness. And only after a while, when commenting stops, we should create a “top ten” for our comments; or something like that; we should make sure that more than one idea gets exposure, rather than going from the start with a “winner takes it all” mentality. It&#39;s not good for anybody.</p>

<p>Photo by <a href="https://unsplash.com/@fbngsk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Fabian Gieske</a> on <a href="https://unsplash.com/s/photos/agree?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Unsplash</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:rants" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">rants</span></a> <a href="/alan/tag:thoughts" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">thoughts</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/the-curse-of-the-downvote</guid>
      <pubDate>Sat, 21 Sep 2019 12:07:46 +0000</pubDate>
    </item>
    <item>
      <title>Machine Learning: a sound primer</title>
      <link>https://www.franzoni.eu/alan/machine-learning-a-sound-primer</link>
      <description>&lt;![CDATA[&#xA;&#xA;I see many people who would like to take a glimpse at machine learning, and try to understand a bit how it works. Very often, they can either get pre-baked examples with very specific (and possibly too advanced) approaches - like deep learning - or math-oriented explanations that can be dry or just uninteresting.&#xA;&#xA;I recently discovered a rather famous free textbook that I hadn&#39;t touched before: An Introduction to Statistical Learning . As you may infer by the non-glamorous title, that&#39;s a book that doesn&#39;t try to sell you something fancy about machine learning. It&#39;s quite a practical and non math-heavy introduction to most useful machine learning topics, which will lead the reader to develop an intuition for what ML methods do. SPOILER: neural networks aren&#39;t covered! So, if you&#39;re just running after the hype, that&#39;s not the book for you.&#xA;&#xA;The only real drawback from the original book is that most examples and demos are coded in R. I don&#39;t especially like the language, as it is highly specialized and, most probably, you&#39;ll need to know another language beyond it for general-purpose processing.&#xA;&#xA;So, I&#39;m happy to link a couple of repositories that offer most examples from the book, but coded in Python; those should be more accessible to most people, as the language is very widespread:&#xA;&#xA;https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning&#xA;&#xA;https://github.com/JWarmenhoven/ISLR-python&#xA;&#xA;Happy StatLearning!&#xA;&#xA;EDIT:&#xA;&#xA;There&#39;s a MOOC as well covering most of the topics from the book, by the same original authors&#xA;&#xA;Photo by Lacie Slezak on Unsplash&#xA;&#xA;#Ollivander #machinelearning #statistics&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/07/lacie-slezak-yHG6llFLjS0-unsplash-1-.jpg" alt=""></p>

<p>I see many people who would like to take a glimpse at machine learning, and try to understand a bit how it works. Very often, they can either get pre-baked examples with very specific (and possibly too advanced) approaches – like deep learning – or math-oriented explanations that can be dry or just uninteresting.</p>

<p>I recently discovered a rather famous free textbook that I hadn&#39;t touched before: <a href="https://www.statlearning.com/" rel="nofollow">An Introduction to Statistical Learning</a> . As you may infer by the non-glamorous title, that&#39;s a book that doesn&#39;t try to sell you something <em>fancy</em> about machine learning. It&#39;s quite a practical and non math-heavy introduction to most useful machine learning topics, which will lead the reader to develop an intuition for what ML methods do. SPOILER: neural networks aren&#39;t covered! So, if you&#39;re just running after the hype, that&#39;s not the book for you.</p>

<p>The only real drawback from the original book is that most examples and demos are coded in R. I don&#39;t especially like the language, as it is highly specialized and, most probably, you&#39;ll need to know another language beyond it for general-purpose processing.</p>

<p>So, I&#39;m happy to link a couple of repositories that offer most examples from the book, but coded in Python; those should be more accessible to most people, as the language is very widespread:</p>

<p><a href="https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning" rel="nofollow">https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning</a></p>

<p><a href="https://github.com/JWarmenhoven/ISLR-python" rel="nofollow">https://github.com/JWarmenhoven/ISLR-python</a></p>

<p>Happy StatLearning!</p>

<p>EDIT:</p>

<p>There&#39;s a <a href="https://www.edx.org/course/statistical-learning" rel="nofollow">MOOC as well covering most of the topics from the book</a>, by the same original authors</p>

<p><em>Photo by <a href="https://unsplash.com/@nbb_photos?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Lacie Slezak</a> on <a href="https://unsplash.com/search/photos/learning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Unsplash</a></em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:machinelearning" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">machinelearning</span></a> <a href="/alan/tag:statistics" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">statistics</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/machine-learning-a-sound-primer</guid>
      <pubDate>Wed, 31 Jul 2019 08:40:54 +0000</pubDate>
    </item>
    <item>
      <title>Standalone, single-file, editable Python scripts WITH DEPENDENCIES</title>
      <link>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</link>
      <description>&lt;![CDATA[&#xA;&#xA;How badly I wanted something like that?&#xA;&#xA;The problem: Python for scripting&#xA;&#xA;Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing virtualenv and so on.&#xA;&#xA;Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.&#xA;&#xA;For sure, there&#39;re many options to fully package a Python executable - PyInstaller comes to my mind, but other exist. But then you&#39;ve got a kind of &#34;build process&#34; for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines. Yes, I wouldn&#39;t do the same for &#34;real&#34; software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.&#xA;&#xA;The solution: editable python scripts with isolated dependencies&#xA;&#xA;So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have python and pip on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.&#xA;&#xA;This will install the dependencies in separate location in a temporary directory at first use, then reuse them when necessary.&#xA;&#xA;So: just copy &amp; paste the following snippets, edit the two USER SERVICEABLE sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run requests, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.&#xA;&#xA;!/usr/bin/python3&#xA;import os&#xA;import sys&#xA;from tempfile import gettempdir, NamedTemporaryFile&#xA;import hashlib&#xA;&#xA;USER SERVICEABLE: paste here your requirements.txt&#xA;the recommendation is to create a development virtualenv,&#xA;install the deps with pip inside it, then do a pip freeze&#xA;and paste the output here&#xA;REQUIREMENTS = &#34;&#34;&#34;&#xA;certifi==2018.11.29&#xA;chardet==3.0.4&#xA;idna==2.8&#xA;requests==2.21.0&#xA;urllib3==1.24.1&#xA;&#34;&#34;&#34;&#xA;USER SERVICEABLE end&#xA;&#xA;def addcustomsitepackagesdirectory(raiseiffailure=True):&#xA;    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()&#xA;    deproot = os.path.join(gettempdir(), &#34;pyallinone{}&#34;.format(digest))&#xA;    os.makedirs(deproot, existok=True)&#xA;&#xA;    for dirpath, dirnames, filenames in os.walk(deproot):&#xA;        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):&#xA;            # that&#39;s our dir!&#xA;            sys.path.insert(0, os.path.abspath(dirpath))&#xA;            return deproot&#xA;&#xA;    if raiseiffailure:&#xA;        raise ValueError(&#34;could not find our site-packages dir&#34;)&#xA;&#xA;    return deproot&#xA;&#xA;deproot = addcustomsitepackagesdirectory(False)&#xA;&#xA;depsinstalled = False&#xA;&#xA;while True:&#xA;    try:&#xA;        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!&#xA;        import requests&#xA;        # USER SERVICEABLE end&#xA;&#xA;        break&#xA;    except ImportError:&#xA;        if depsinstalled:&#xA;            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)&#xA;        try:&#xA;            from pip import main as pipmain&#xA;        except ImportError:&#xA;            from pip.internal import main as pipmain&#xA;&#xA;        with NamedTemporaryFile() as req:&#xA;            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))&#xA;            req.flush()&#xA;            pipmain([&#34;install&#34;, &#34;--prefix&#34;, deproot, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])&#xA;&#xA;        addcustomsitepackagesdirectory()&#xA;        depsinstalled = True&#xA;&#xA;HERE you can start writing the actual code of your script&#xA;&#xA;r = requests.get(&#34;https://www.google.com&#34;)&#xA;print(r.statuscode)&#xA;&#xA;How does this work?&#xA;&#xA;It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using pip. Such subdir name is autogenerated depending on your requirements - so, if your requirements change, a new directory is employed.&#xA;Then, it adds such directory to your sys.path, allowing Python to find modules and packages there.&#xA;When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.&#xA;&#xA;CAVEATS:&#xA;&#xA;Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.&#xA;If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).&#xA;If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.&#xA;&#xA;Alternative approaches:&#xA;&#xA;If you already use uv and you assume it&#39;s already installed on your systems, there&#39;s a great article here on how to use uv for python scripts with dependencies&#xA;&#xA;Photo by Milan Popovic on Unsplash&#xA;&#xA;#MostlyUnixish #python #packaging #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/02/milan-popovic-674483-unsplash-1-.jpg" alt=""></p>

<p>How badly I wanted something like that?</p>

<h3 id="the-problem-python-for-scripting">The problem: Python for scripting</h3>

<p>Beside programming and data science, I find Python to be a very useful glue language; I think it&#39;s great for shell replacement when bash/zsh scripts get too complex, but there&#39;s one caveat: as long as you can work with its standard library, you&#39;re in the sweet spot. As soon as you&#39;d like to use an external dependency, that can be a problem, because if you don&#39;t want to contaminate your system with external dependencies, you&#39;ll either a) hope that your system packages a proper version for such library or b) start needing <a href="https://virtualenv.pypa.io/en/latest/" rel="nofollow">virtualenv</a> and so on.</p>

<p>Both options are ok for manual development, a bit less ok if you&#39;re willing to deliver such scripts to multiple servers for automating some kind of process.</p>

<p>For sure, there&#39;re many options to fully package a Python executable – <a href="https://www.pyinstaller.org/" rel="nofollow">PyInstaller</a> comes to my mind, but other exist. But then you&#39;ve got a kind of “build process” for your script, and you cannot edit it directly on a server. But I find that, very often, for internal tasks and scripts, my process is exactly that: <strong>I do edit the script on the server, then, when I get it right, I copy it on my version control system and deliver it to other machines.</strong> Yes, I wouldn&#39;t do the same for “real” software, but as I said, those are often internal scripts, used for reporting, cron jobs, other small automated tasks.</p>

<h3 id="the-solution-editable-python-scripts-with-isolated-dependencies">The solution: editable python scripts with isolated dependencies</h3>

<p>So what? That&#39;s what I baked. Not a perfect solution, but a decent one. Just have <em>python</em> and <em>pip</em> on your system, add a REQUIREMENTS string (equivalent to the content from requirements.txt), then import everything.</p>

<p>This will install the dependencies in <strong>separate</strong> location in a temporary directory at first use, then reuse them when necessary.</p>

<p>So: just copy &amp; paste the following snippets, edit the two <code>USER SERVICEABLE</code> sections, then start writing your desired code at the bottom. The snippet here includes an example of how to run <code>requests</code>, so you can just delete the requirements, imports and requests call at the bottom if you don&#39;t need that.</p>

<pre><code class="language-python">#!/usr/bin/python3
import os
import sys
from tempfile import gettempdir, NamedTemporaryFile
import hashlib

# USER SERVICEABLE: paste here your requirements.txt
# the recommendation is to create a development virtualenv,
# install the deps with pip inside it, then do a `pip freeze`
# and paste the output here
REQUIREMENTS = &#34;&#34;&#34;
certifi==2018.11.29
chardet==3.0.4
idna==2.8
requests==2.21.0
urllib3==1.24.1
&#34;&#34;&#34;
# USER SERVICEABLE end

def add_custom_site_packages_directory(raise_if_failure=True):
    digest = hashlib.sha256(REQUIREMENTS.encode(&#34;utf8&#34;)).hexdigest()
    dep_root = os.path.join(gettempdir(), &#34;pyallinone_{}&#34;.format(digest))
    os.makedirs(dep_root, exist_ok=True)

    for dirpath, dirnames, filenames in os.walk(dep_root):
        if dirpath.endswith(os.path.sep + &#34;site-packages&#34;):
            # that&#39;s our dir!
            sys.path.insert(0, os.path.abspath(dirpath))
            return dep_root

    if raise_if_failure:
        raise ValueError(&#34;could not find our site-packages dir&#34;)

    return dep_root

dep_root = add_custom_site_packages_directory(False)

deps_installed = False

while True:
    try:
        # USER SERVICEABLE: import all your required deps in this block! and keep the break at the end!
        import requests
        # USER SERVICEABLE end

        break
    except ImportError:
        if deps_installed:
            raise ValueError(&#34;Something was broken, could not install dependencies&#34;)
        try:
            from pip import main as pipmain
        except ImportError:
            from pip._internal import main as pipmain

        with NamedTemporaryFile() as req:
            req.write(REQUIREMENTS.encode(&#34;utf-8&#34;))
            req.flush()
            pipmain([&#34;install&#34;, &#34;--prefix&#34;, dep_root, &#34;--upgrade&#34;, &#34;--no-cache-dir&#34;, &#34;--no-deps&#34;, &#34;-r&#34;, req.name])

        add_custom_site_packages_directory()
        deps_installed = True

# HERE you can start writing the actual code of your script

r = requests.get(&#34;https://www.google.com&#34;)
print(r.status_code)
</code></pre>

<h3 id="how-does-this-work">How does this work?</h3>
<ul><li>It creates a subdir in the directory where temporary files are held on your filesystem, and downloads the dependencies there using <code>pip</code>. Such subdir name is autogenerated depending on your requirements – so, if your requirements change, a new directory is employed.</li>
<li>Then, it adds such directory to your <code>sys.path</code>, allowing Python to find modules and packages there.</li>
<li>When you restart the script, it first tries to find the libraries that were previously downloaded, and only if that fails it goes downloading the libraries again.</li></ul>

<h3 id="caveats">CAVEATS:</h3>
<ul><li>Of course the target system must have internet access, at least to pypi or github or other vcs (depending on your requirements format), and python and pip must be installed.</li>
<li>If your requirements change, nothing deletes files in your temp directory. But that&#39;s usually sweeped at system boot or by cronjobs, so it&#39;s not a real problem. BUT: if your cronjob only partially sweeps files in the subdir, it could break something (check if anything like that exists on your system, I can remember some older Redhat/Centos doing that).</li>
<li>If your packages have binary dependencies and/or require to build extensions, you still need the shared libs (for runtime) AND the proper header files / -dev packages. There&#39;s no silver bullet for that in this recipe.</li></ul>

<h3 id="alternative-approaches">Alternative approaches:</h3>
<ul><li>If you already use <code>uv</code> and you assume it&#39;s already installed on your systems, there&#39;s a <a href="https://thisdavej.com/share-python-scripts-like-a-pro-uv-and-pep-723-for-easy-deployment/" rel="nofollow">great article here on how to use uv for python scripts with dependencies</a></li></ul>

<p>Photo by <a href="https://unsplash.com/@itsmiki5?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge" rel="nofollow">Milan Popovic</a> on Unsplash</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/single-file-editable-python-scripts-with-dependencies</guid>
      <pubDate>Tue, 19 Feb 2019 20:58:00 +0000</pubDate>
    </item>
  </channel>
</rss>