<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>ollivander &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:ollivander</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:40:58 +0000</pubDate>
    <item>
      <title>Password requirements: myths and madness</title>
      <link>https://www.franzoni.eu/alan/password-requirements-myths-madness</link>
      <description>&lt;![CDATA[More than 11 years have passed since the venerable XKCD Password Strength strip:&#xA;&#xA;img src=&#34;https://imgs.xkcd.com/comics/passwordstrength.png&#34; &#xA;&#xA;And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:&#xA;&#xA;Single\Sign\On-1&#xA;&#xA;The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.&#xA;&#xA;I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.&#xA;&#xA;Please use a mix of uppercase and lowercase letters, digits, special chars in your password&#xA;&#xA;Usual explanation: for security reasons you must choose a strong password!&#xA;&#xA;My verdict: Misunderstanding&#xA;&#xA;The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same entropy as a shorter password from a larger charset.&#xA;&#xA;If you host a web application: Enforce a password&#39;s strength, not strange user-hostile policies. There&#39;re libraries for that (e.g. https://github.com/dropbox/zxcvbn) that work both in backend and frontend.&#xA;&#xA;You cannot use whitespace, accented letters, hyphens, \[...\] in your password&#xA;&#xA;Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like &#34;your password contains unsafe characters&#34;.&#xA;&#xA;My verdict: No real motivation beyond, possibly, incompetence&#xA;&#xA;This is a terrible one. You waste a lot of time at generating a &#34;strong password&#34; that complies with a website&#39;s policy, and then you discover that you could not use &#34;-&#34; in it. Or just whitespace for XKCD-style passwords.&#xA;&#xA;There&#39;s no technical reason for restricting any character from appearing in any password. The usual approach for handling a password is:&#xA;&#xA;take the password as a string, as the user typed it;&#xA;Prepare it (see links at the end of the paragraph);&#xA;encode with a specific, well-defined encoding (e.g. utf-8);&#xA;add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an &#34;hashed&#34; password and save that salt+hash in your storage of choice.&#xA;&#xA;What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.&#xA;&#xA;Caveats:&#xA;In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.&#xA;&#xA;If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.&#xA;&#xA;EDIT:&#xA;Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called &#34;string preparation&#34;. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:&#xA;https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html&#xA;https://www.rfc-editor.org/rfc/rfc8265&#xA;http://unicode.org/reports/tr15/\#Stabilized\Strings&#xA;&#xA;So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there could be good reasons to reject some input chars since there could be no stable encoding under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.&#xA;&#xA;You cannot enter a password with fewer than X chars&#xA;&#xA;EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET&#xA;&#xA;Usual explanation: security! I want your password to be veeeery strong!&#xA;&#xA;My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.&#xA;&#xA;This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. NIST standards even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).&#xA;&#xA;This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.&#xA;&#xA;But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the BMP, my password is probably very safe already, since there are (10\\6)\\4 possible passwords, many more than (127\\8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.&#xA;&#xA;If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.&#xA;&#xA;You cannot enter a password with more than X chars&#xA;&#xA;Usual explanation: whew. It&#39;s too large we cannot handle it.&#xA;&#xA;My verdict: Largely unmotivated unless X is quite large.&#xA;&#xA;Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.&#xA;&#xA;This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the &#34;enter your password&#34; field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.&#xA;&#xA;Caveats:&#xA;An higher length limit for passwords must should exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.&#xA;&#xA;EDIT:&#xA;A reader pointed out that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a a longer password doesn&#39;t mean it&#39;s necessarily more secure.&#xA;&#xA;If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user &#34;password too long&#34; if they&#39;re pasting into it.&#xA;&#xA;You cannot paste into this field&#xA;&#xA;Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!&#xA;&#xA;My verdict: Largely unmotivated right now, and dangerous.&#xA;&#xA;Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?&#xA;&#xA;If you host a web application: please allow pasting into password fields.&#xA;&#xA;You cannot show or copy this field&#xA;&#xA;Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.&#xA;&#xA;My verdict: Security through obscurity bad idea.&#xA;&#xA;Don&#39;t do this. While setting a password, there must be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is especially true if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.&#xA;&#xA;If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!&#xA;&#xA;EDIT:&#xA;This entry was edited after some comments. I know that&#39;s not what &#34;security through obscurity&#34; mean, it was a kind of joke about &#34;obscurating&#34; the password.&#xA;&#xA;Your password has expired&#xA;&#xA;Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.&#xA;&#xA;My verdict: Old idea&#xA;&#xA;This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation&#xA;&#xA;Too many failures - your account is locked.&#xA;&#xA;Usual explanation: we don&#39;t want your account to be bruteforced&#xA;&#xA;My verdict: Bad idea especially if &#34;too many&#34; is low, like 3 or 5, and my password is complex.&#xA;&#xA;The idea is that you &#34;protect&#34; a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.&#xA;&#xA;Account locking could be used in 2FA contexts (requires the other factor to provoke a DoS - that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts - they&#39;re too few to guess a password.&#xA;&#xA;Caveats:&#xA;Some compliance policies require locking user accounts.&#xA;&#xA;If you host a web application: consider using temporary locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).&#xA;&#xA;Final thoughts&#xA;&#xA;I really hope that passwordless login will make some steps forward in the next years (e.g. https://passkeys.dev/, https://www.yubico.com/authentication-standards/fido2/). In the meantime, I&#39;d love a &#34;password complexity api&#34; so that my browser (or password manager) can already generate the right password for a website.&#xA;&#xA;References&#xA;&#xA;https://www.auditboard.com/blog/nist-password-guidelines/&#xA;https://pages.nist.gov/800-63-3/sp800-63-3.html&#xA;&#xA;#Ollivander #security #passwords&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>More than 11 years have passed since the venerable <a href="https://xkcd.com/936/" rel="nofollow">XKCD Password Strength strip</a>:</p>

<p><img src="https://imgs.xkcd.com/comics/password_strength.png"></p>

<p>And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/12/Single_Sign_On-1.jpg" alt="Single\_Sign\_On-1"></p>

<p>The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.</p>

<p>I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.</p>

<h3 id="please-use-a-mix-of-uppercase-and-lowercase-letters-digits-special-chars-in-your-password">Please use a mix of uppercase and lowercase letters, digits, special chars in your password</h3>

<p>Usual explanation: for security reasons you must choose a strong password!</p>

<p>My verdict: <strong>Misunderstanding</strong></p>

<p>The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same <strong><a href="https://en.wikipedia.org/wiki/Entropy_%28information_theory%29" rel="nofollow">entropy</a></strong> as a shorter password from a larger charset.</p>

<p>If you host a web application: Enforce a password&#39;s <strong>strength</strong>, not strange user-hostile policies. There&#39;re libraries for that (e.g. <a href="https://github.com/dropbox/zxcvbn" rel="nofollow">https://github.com/dropbox/zxcvbn</a>) that work both in backend and frontend.</p>

<h3 id="you-cannot-use-whitespace-accented-letters-hyphens-in-your-password">You cannot use whitespace, accented letters, hyphens, [...] in your password</h3>

<p>Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like “your password contains unsafe characters”.</p>

<p>My verdict: <strong>No real motivation beyond, possibly, incompetence</strong></p>

<p>This is a terrible one. You waste a lot of time at generating a “strong password” that complies with a website&#39;s policy, and then you discover that you could not use “–” in it. Or just whitespace for XKCD-style passwords.</p>

<p><strong>There&#39;s no technical reason for restricting any character from appearing in any password.</strong> The usual approach for handling a password is:</p>
<ul><li>take the password as a string, as the user typed it;</li>
<li>Prepare it (see links at the end of the paragraph);</li>
<li>encode with a specific, well-defined encoding (e.g. utf-8);</li>
<li>add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an “hashed” password and save that salt+hash in your storage of choice.</li></ul>

<p>What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.</p>

<p>Caveats:
In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.</p>

<p>If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.</p>

<p>EDIT:
<em>Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called “string preparation”. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:</em>
<a href="https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html" rel="nofollow">https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html</a>
<a href="https://www.rfc-editor.org/rfc/rfc8265" rel="nofollow">https://www.rfc-editor.org/rfc/rfc8265</a>
<a href="http://unicode.org/reports/tr15/#Stabilized_Strings" rel="nofollow">http://unicode.org/reports/tr15/#Stabilized_Strings</a></p>

<p>So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there <em>could</em> be good reasons to reject some input chars since there could be no <em>stable encoding</em> under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.</p>

<h3 id="you-cannot-enter-a-password-with-fewer-than-x-chars">You cannot enter a password with fewer than X chars</h3>

<p><em>EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET</em></p>

<p>Usual explanation: security! I want your password to be veeeery strong!</p>

<p>My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.</p>

<p>This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. <a href="https://pages.nist.gov/800-63-3/sp800-63b.html" rel="nofollow">NIST standards</a> even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).</p>

<p>This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.</p>

<p>But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the <a href="https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane" rel="nofollow">BMP</a>, my password is probably very safe already, since there are (10**6)**4 possible passwords, many more than (127**8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.</p>

<p>If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.</p>

<h3 id="you-cannot-enter-a-password-with-more-than-x-chars">You cannot enter a password with more than X chars</h3>

<p>Usual explanation: whew. It&#39;s too large we cannot handle it.</p>

<p>My verdict: <strong>Largely unmotivated</strong> unless X is quite large.</p>

<p>Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.</p>

<p>This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the “enter your password” field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.</p>

<p>Caveats:
An higher length limit for passwords <del>must</del> <strong>should</strong> exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.</p>

<p>EDIT:
A reader <a href="https://www.sjoerdlangkemper.nl/2021/07/02/long-password-denial-of-service/" rel="nofollow">pointed out</a> that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a <a href="https://crypto.stackexchange.com/questions/98505/argon2-allows-a-huge-key-length-in-input-but-does-it-really-provide-the-securit" rel="nofollow">a longer password doesn&#39;t mean it&#39;s necessarily more secure</a>.</p>

<p>If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user “password too long” if they&#39;re pasting into it.</p>

<h3 id="you-cannot-paste-into-this-field">You cannot paste into this field</h3>

<p>Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!</p>

<p>My verdict: <strong>Largely unmotivated right now, and dangerous</strong>.</p>

<p>Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?</p>

<p>If you host a web application: please allow pasting into password fields.</p>

<h3 id="you-cannot-show-or-copy-this-field">You cannot show or copy this field</h3>

<p>Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.</p>

<p>My verdict: <del>Security through obscurity</del> bad idea.</p>

<p>Don&#39;t do this. While setting a password, there <strong>must</strong> be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is <strong>especially true</strong> if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.</p>

<p>If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!</p>

<p>EDIT:
<em>This entry was edited after some comments. I know that&#39;s not what “security through obscurity” mean, it was a kind of joke about “obscurating” the password.</em></p>

<h3 id="your-password-has-expired">Your password has expired</h3>

<p>Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.</p>

<p>My verdict: <strong>Old idea</strong></p>

<p>This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation</p>

<h3 id="too-many-failures-your-account-is-locked">Too many failures – your account is locked.</h3>

<p>Usual explanation: we don&#39;t want your account to be bruteforced</p>

<p>My verdict: <strong>Bad idea</strong> especially if “too many” is low, like 3 or 5, and my password is complex.</p>

<p>The idea is that you “protect” a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.</p>

<p>Account locking <em>could</em> be used in 2FA contexts (requires the other factor to provoke a DoS – that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts – they&#39;re too few to guess a password.</p>

<p>Caveats:
Some compliance policies require locking user accounts.</p>

<p>If you host a web application: consider using <em>temporary</em> locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).</p>

<h2 id="final-thoughts">Final thoughts</h2>

<p>I really hope that passwordless login will make some steps forward in the next years (e.g. <a href="https://passkeys.dev/" rel="nofollow">https://passkeys.dev/</a>, <a href="https://www.yubico.com/authentication-standards/fido2/" rel="nofollow">https://www.yubico.com/authentication-standards/fido2/</a>). In the meantime, I&#39;d love a “password complexity api” so that my browser (or password manager) can already generate the right password for a website.</p>

<h2 id="references">References</h2>

<p><a href="https://www.auditboard.com/blog/nist-password-guidelines/" rel="nofollow">https://www.auditboard.com/blog/nist-password-guidelines/</a>
<a href="https://pages.nist.gov/800-63-3/sp800-63-3.html" rel="nofollow">https://pages.nist.gov/800-63-3/sp800-63-3.html</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a> <a href="/alan/tag:passwords" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">passwords</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/password-requirements-myths-madness</guid>
      <pubDate>Thu, 22 Dec 2022 20:41:14 +0000</pubDate>
    </item>
    <item>
      <title>Ransomware-resistant backups with duplicity and AWS S3</title>
      <link>https://www.franzoni.eu/alan/ransomware-resistant-backups</link>
      <description>&lt;![CDATA[Ransomware and backups&#xA;&#xA;Article updated: February 5th, 2022&#xA;&#xA;Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.&#xA;&#xA;Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, cryptolocker-like malwares are spreading. You&#39;re not safe just because why should anybody hack into my little server: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.&#xA;&#xA;So: people should revise their own threat models. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a failure scenario (hardware failure, accidental deletion), not for an attack scenario. But even half-serious cybercriminals, once they get access to your server, will delete all backup sets they can access, so that their encrypt-your-data-and-ask-for-money threat actually works.&#xA;&#xA;Once upon a time&#xA;&#xA;You had backups on a tape library. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated - getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.&#xA;&#xA;Then approaches like bacula were common - the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it &#34;wipe everything&#34; so again, barring any bacula server exploit, bacula server data held backups safe.&#xA;&#xA;More recently, cloud approaches like rsync.net leverage ZFS snapshots to make remote data immutable - they can be removed only according to your retention schedule (details). BorgBase works in a similar way and has a good writeup about backup strategies. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider restore costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data - at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.&#xA;&#xA;But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: restic offers rest-server, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.&#xA;&#xA;The cloud to the rescue&#xA;&#xA;So, here we&#39;ll see how to use the good old duplicity backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.&#xA;&#xA;For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your workstation and your server, and you&#39;d like to have your server backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.&#xA;&#xA;So, make yourself a favour and use two separate machines.&#xA;&#xA;Create your S3 bucket and credentials&#xA;&#xA;Those steps must be taken from your workstation&#xA;&#xA;These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.&#xA;&#xA;Create the bucket&#xA;&#xA;Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it sample-duplicity-backup for the sake of this article) in your preferred region (I will use eu-central-1) ; and use these settings:&#xA;&#xA;Disable ACLs&#xA;&#xA;S3\bucket-1-1&#xA;&#xA;Block public access&#xA;&#xA;S3\bucket-2-1&#xA;&#xA;Enable bucket versioning&#xA;&#xA;S3\bucket-3-1&#xA;&#xA;Enable object lock&#xA;&#xA;S3\bucket-4-1&#xA;&#xA;Configure object lock&#xA;&#xA;Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.&#xA;&#xA;sample-duplicity-backup\objectlock&#xA;&#xA;Create a IAM user&#xA;&#xA;Now you need to create a suitable IAM user to be used with duplicity. Let&#39;s call this sample-duplicity-user; it should only have programmatic access, and you should attach only this policy to such user:&#xA;&#xA;{&#xA;    &#34;Version&#34;: &#34;2012-10-17&#34;,&#xA;    &#34;Statement&#34;: [&#xA;        {&#xA;            &#34;Effect&#34;: &#34;Allow&#34;,&#xA;            &#34;Action&#34;: [&#xA;                &#34;s3:PutObject&#34;,&#xA;                &#34;s3:GetObject&#34;,&#xA;                &#34;s3:ListBucketMultipartUploads&#34;,&#xA;                &#34;s3:AbortMultipartUpload&#34;,&#xA;                &#34;s3:ListBucket&#34;,&#xA;                &#34;s3:ListMultipartUploadParts&#34;&#xA;            ],&#xA;            &#34;Resource&#34;: [&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup/&#34;&#xA;            ]&#xA;        }&#xA;    ]&#xA;}&#xA;&#xA;Download the access key ID and the secret access key for such user.&#xA;&#xA;Install duplicity&#xA;&#xA;The following steps must be performed on the server&#xA;&#xA;Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported - check the homepage for further info. Also make sure the boto3 library and gnupg are available on your system.&#xA;&#xA;As an example, for recent Ubuntu versions (tested on 20.04):&#xA;&#xA;Add the stable duplicity PPA&#xA;apt -y install duplicity python3-boto3 gnupg&#xA;&#xA;Create a backup script&#xA;&#xA;Put some data in /root/data for the purpose of this test.&#xA;&#xA;Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography - support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:&#xA;&#xA;!/bin/bash -e&#xA;export PASSPHRASE=&#34;XYZXYZXYZ&#34;&#xA;export AWSACCESSKEYID=&#34;XXXXXXX&#34;&#xA;export AWSSECRETACCESSKEY=&#34;YYYYYYYYY&#34;&#xA;/usr/bin/duplicity \&#xA;--s3-european-buckets \&#xA;  --s3-use-new-style  --asynchronous-upload -v 4 \&#xA;   incr --full-if-older-than 30D \&#xA;  /root/data \&#xA;  &#34;boto3+s3://sample-duplicity-backup/data&#34;&#xA;&#xA;This command will backup the contents of /root/data to your s3 bucket, using a&#xA;data prefix.&#xA;&#xA;One detail to note: the --full-if-older 30D means: create an incremental backup&#xA;since the previous one, or create a new full backup if more than 30 days passed since our last full backup. It is essential that the number of days that we set here is smaller than the object lock days.&#xA;&#xA;So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the data directory:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket&#xA;&#xA;If you like, you can run the script again. You&#39;ll notice that duplicity only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.&#xA;&#xA;Attack scenario&#xA;&#xA;Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for sample-duplicity-user, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.&#xA;&#xA;For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same sample-duplicity-user credentials and pretend you&#39;re the badguy:&#xA;&#xA;aws configure --profile badguy&#xA;&#xA;Now, try acting as the Bad Guy:&#xA;&#xA;$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/&#xA;2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg&#xA;2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg&#xA;2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg&#xA;&#xA;He can read the backup. This is expected. But can he delete the backup?&#xA;&#xA;$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied&#xA;&#xA;The attacker cannot delete files; our object lock policy prevents that!&#xA;&#xA;But, there&#39;s still one thing that the attacker could do: overwrite existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between &#34;add an object&#34; or &#34;overwrite an existing object&#34; (and it would be quite difficult and slow to implement in a distributed storage system):&#xA;&#xA;$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Ouch! This (apparently) succeeded:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-update&#xA;&#xA;But the reality is that the original data is still there: the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the &#34;Versions&#34; tab for our modified file:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-versions&#xA;&#xA;Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the s3:ListBucketVersions permission (check the API call as well) - we don&#39;t even have it for our backup IAM user.&#xA;&#xA;Retrieving the original files&#xA;&#xA;A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:&#xA;&#xA;$ aws s3api list-object-versions --bucket sample-duplicity-backup&#xA;{&#xA;    &#34;Versions&#34;: [&#xA;        ...&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,&#xA;            &#34;Size&#34;: 50,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;wZOOamRomLYpZO5Gc.Vynrl0cP6bWq&#34;,&#xA;            &#34;IsLatest&#34;: true,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;            &#34;Size&#34;: 267,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;            &#34;IsLatest&#34;: false,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        ...&#xA;    ]&#xA;}&#xA;&#xA;Then retrieve this file in its older version:&#xA;&#xA;$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;&#xA;{&#xA;    &#34;AcceptRanges&#34;: &#34;bytes&#34;,&#xA;    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;    &#34;ContentLength&#34;: 267,&#xA;    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,&#xA;    &#34;Metadata&#34;: {}&#xA;}&#xA;$ ls .gpg&#xA;duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.&#xA;&#xA;Wrapping up&#xA;&#xA;I hope this article can be useful from an implementation standpoint, but the most important takeaway is: always think about your threat model every time you take a decision. Reason in the terms of what an attacker could do when entering your system; security is never binary.&#xA;&#xA;And, please: don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set, like -rwx------ 1 root root 365 Dec  9 21:52 duplicity_run. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!&#xA;&#xA;An exercise&#xA;&#xA;An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.&#xA;&#xA;To my readers: since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.&#xA;&#xA;Updates &amp; footnotes&#xA;&#xA;I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the HN crowd as usual for commenting.&#xA;&#xA;Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.&#xA;&#xA;So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):&#xA;&#xA;You should remember to periodically verify your backups, usually by restoring them - this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, make sure you verify backups before deleting old data.&#xA;Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even replicate buckets between accounts&#xA;Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend setting a proper budget alert for your AWS account and maybe an alert for excessive S3 data usage&#xA;Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job - so you get errors if it fails - and to periodically verify them.&#xA;If you&#39;ve got a lot of data and not a lot of bandwidth - hence the initial backup may take you a lot time - make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.&#xA;&#xA;#Ollivander #backups #security&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<h2 id="ransomware-and-backups">Ransomware and backups</h2>

<p><em>Article updated: February 5th, 2022</em></p>

<p>Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.</p>

<p>Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, <a href="https://en.wikipedia.org/wiki/CryptoLocker" rel="nofollow">cryptolocker-like</a> malwares are spreading. You&#39;re not safe just because <em>why should anybody hack into my little server</em>: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.</p>

<p>So: <strong>people should revise their own threat models</strong>. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a <strong>failure scenario</strong> (hardware failure, accidental deletion), not for an <strong>attack scenario</strong>. But even half-serious cybercriminals, once they get access to your server, <strong>will delete all backup sets they can access</strong>, so that their encrypt-your-data-and-ask-for-money threat actually works.</p>

<h2 id="once-upon-a-time">Once upon a time</h2>

<p>You had backups on a <a href="https://en.wikipedia.org/wiki/Tape_drive" rel="nofollow">tape library</a>. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated – getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.</p>

<p>Then approaches like <a href="https://www.bacula.org/" rel="nofollow">bacula</a> were common – the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it “wipe everything” so again, barring any bacula server exploit, bacula server data held backups safe.</p>

<p>More recently, cloud approaches like <a href="https://www.rsync.net/" rel="nofollow">rsync.net</a> leverage ZFS snapshots to make remote data immutable – they can be removed only according to your retention schedule (<a href="https://www.rsync.net/resources/howto/snapshots.html" rel="nofollow">details</a>). <a href="https://www.borgbase.com/" rel="nofollow">BorgBase</a> works in a similar way and has a <a href="https://docs.borgbase.com/strategy/" rel="nofollow">good writeup about backup strategies</a>. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider <em>restore</em> costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data – at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.</p>

<p>But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: <a href="https://restic.net/" rel="nofollow">restic</a> offers <a href="https://github.com/restic/rest-server" rel="nofollow">rest-server</a>, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.</p>

<h2 id="the-cloud-to-the-rescue">The cloud to the rescue</h2>

<p>So, here we&#39;ll see how to use the good old <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">duplicity</a> backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.</p>

<p>For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your <strong>workstation</strong> and your <strong>server</strong>, and you&#39;d like to have your <strong>server</strong> backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.</p>

<p>So, make yourself a favour and use two separate machines.</p>

<h2 id="create-your-s3-bucket-and-credentials">Create your S3 bucket and credentials</h2>

<p><em>Those steps must be taken from your workstation</em></p>

<p>These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.</p>

<h3 id="create-the-bucket">Create the bucket</h3>

<p>Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it <em>sample-duplicity-backup</em> for the sake of this article) in your preferred region (I will use <em>eu-central-1</em>) ; and use these settings:</p>

<h4 id="disable-acls">Disable ACLs</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-1-1.jpg" alt="S3\_bucket-1-1"></p>

<h4 id="block-public-access">Block public access</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-2-1.jpg" alt="S3\_bucket-2-1"></p>

<h4 id="enable-bucket-versioning">Enable bucket versioning</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-3-1.jpg" alt="S3\_bucket-3-1"></p>

<h4 id="enable-object-lock">Enable object lock</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-4-1.jpg" alt="S3\_bucket-4-1"></p>

<h4 id="configure-object-lock">Configure object lock</h4>

<p>Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_objectlock.jpg" alt="sample-duplicity-backup\_objectlock"></p>

<h3 id="create-a-iam-user">Create a IAM user</h3>

<p>Now you need to <a href="https://console.aws.amazon.com/iamv2/home#/users" rel="nofollow">create a suitable IAM user</a> to be used with duplicity. Let&#39;s call this <code>sample-duplicity-user</code>; it should only have programmatic access, and you should attach <strong>only</strong> this policy to such user:</p>

<pre><code>{
    &#34;Version&#34;: &#34;2012-10-17&#34;,
    &#34;Statement&#34;: [
        {
            &#34;Effect&#34;: &#34;Allow&#34;,
            &#34;Action&#34;: [
                &#34;s3:PutObject&#34;,
                &#34;s3:GetObject&#34;,
                &#34;s3:ListBucketMultipartUploads&#34;,
                &#34;s3:AbortMultipartUpload&#34;,
                &#34;s3:ListBucket&#34;,
                &#34;s3:ListMultipartUploadParts&#34;
            ],
            &#34;Resource&#34;: [
                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,
                &#34;arn:aws:s3:::sample-duplicity-backup/*&#34;
            ]
        }
    ]
}
</code></pre>

<p>Download the access key ID and the secret access key for such user.</p>

<h2 id="install-duplicity">Install duplicity</h2>

<p><em>The following steps must be performed on the server</em></p>

<p>Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported – check <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">the homepage</a> for further info. Also make sure the <code>boto3</code> library and <code>gnupg</code> are available on your system.</p>

<p>As an example, for recent Ubuntu versions (tested on 20.04):</p>
<ul><li>Add the <a href="https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git" rel="nofollow">stable duplicity PPA</a></li>
<li><code>apt -y install duplicity python3-boto3 gnupg</code></li></ul>

<h2 id="create-a-backup-script">Create a backup script</h2>

<p>Put some data in <code>/root/data</code> for the purpose of this test.</p>

<p>Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography – support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:</p>

<pre><code>#!/bin/bash -e
export PASSPHRASE=&#34;XYZXYZXYZ&#34;
export AWS_ACCESS_KEY_ID=&#34;XXXXXXX&#34;
export AWS_SECRET_ACCESS_KEY=&#34;YYYYYYYYY&#34;
/usr/bin/duplicity \
--s3-european-buckets \
  --s3-use-new-style  --asynchronous-upload -v 4 \
   incr --full-if-older-than 30D \
  /root/data \
  &#34;boto3+s3://sample-duplicity-backup/data&#34;
</code></pre>

<p>This command will backup the contents of <code>/root/data</code> to your s3 bucket, using a
<code>data</code> prefix.</p>

<p>One detail to note: the <code>--full-if-older 30D</code> means: <em>create an incremental backup
since the previous one, or create a new full backup if more than 30 days passed since our last full backup</em>. It is <strong>essential</strong> that the number of days that we set here is smaller than the object lock days.</p>

<p>So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the <em>data</em> directory:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket"></p>

<p>If you like, you can run the script again. You&#39;ll notice that duplicity <em>only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.</em></p>

<h2 id="attack-scenario">Attack scenario</h2>

<p>Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for <code>sample-duplicity-user</code>, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.</p>

<p>For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same <code>sample-duplicity-user</code> credentials and pretend you&#39;re the badguy:</p>

<p><code>aws configure --profile badguy</code></p>

<p>Now, try acting as the Bad Guy:</p>

<pre><code>$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/
2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg
2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg
2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg
</code></pre>

<p>He can read the backup. This is expected. But can he <strong>delete</strong> the backup?</p>

<pre><code>$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied
</code></pre>

<p>The attacker cannot delete files; our object lock policy prevents that!</p>

<p>But, there&#39;s still one thing that the attacker could do: <strong>overwrite</strong> existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between “add an object” or “overwrite an existing object” (and it would be quite difficult and slow to implement in a distributed storage system):</p>

<pre><code>$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Ouch! This (apparently) succeeded:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-update.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-update"></p>

<p>But the reality is that <strong>the original data is still there:</strong> the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the “Versions” tab for our modified file:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-versions.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-versions"></p>

<p>Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the <em>s3:ListBucketVersions</em> permission (check the <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html" rel="nofollow">API call as well</a>) – we don&#39;t even have it for our backup IAM user.</p>

<h2 id="retrieving-the-original-files">Retrieving the original files</h2>

<p>A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:</p>

<pre><code>$ aws s3api list-object-versions --bucket sample-duplicity-backup
{
    &#34;Versions&#34;: [
        ...
        {
            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,
            &#34;Size&#34;: 50,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;w_ZOOamRomLYpZO5Gc.Vy_nrl0cP6bWq&#34;,
            &#34;IsLatest&#34;: true,
            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        {
            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
            &#34;Size&#34;: 267,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
            &#34;IsLatest&#34;: false,
            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        ...
    ]
}
</code></pre>

<p>Then retrieve this file in its older version:</p>

<pre><code>$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;
{
    &#34;AcceptRanges&#34;: &#34;bytes&#34;,
    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
    &#34;ContentLength&#34;: 267,
    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,
    &#34;Metadata&#34;: {}
}
$ ls *.gpg
duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.</p>

<h2 id="wrapping-up">Wrapping up</h2>

<p>I hope this article can be useful from an implementation standpoint, but the most important takeaway is: <strong>always think about your threat model every time you take a decision</strong>. Reason in the terms of what an attacker could do when entering your system; <strong>security is never binary</strong>.</p>

<p>And, please: <strong>don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set</strong>, like <code>-rwx------ 1 root root 365 Dec  9 21:52 duplicity_run</code>. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!</p>

<h3 id="an-exercise">An exercise</h3>

<p>An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.</p>

<p><em>To my readers:</em> since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.</p>

<h3 id="updates-footnotes">Updates &amp; footnotes</h3>

<p>I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the <a href="https://news.ycombinator.com/item?id=30105538" rel="nofollow">HN crowd</a> as usual for commenting.</p>

<p>Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.</p>

<p>So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):</p>
<ul><li>You should remember to periodically verify your backups, usually by restoring them – this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, <strong>make sure you verify backups before deleting old data</strong>.</li>
<li>Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html" rel="nofollow">replicate buckets between accounts</a></li>
<li>Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/" rel="nofollow">setting a proper budget alert for your AWS account</a> and maybe an <a href="https://stackoverflow.com/questions/51541751/alarm-when-object-size-in-s3-bucket-exceeds-threshold" rel="nofollow">alert for excessive S3 data usage</a></li>
<li>Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job – so you get errors if it fails – and to periodically verify them.</li>
<li>If you&#39;ve got a lot of data and not a lot of bandwidth – hence the initial backup may take you a lot time – make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.</li></ul>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:backups" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">backups</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/ransomware-resistant-backups</guid>
      <pubDate>Thu, 27 Jan 2022 19:50:24 +0000</pubDate>
    </item>
    <item>
      <title>The curse of the downvote</title>
      <link>https://www.franzoni.eu/alan/the-curse-of-the-downvote</link>
      <description>&lt;![CDATA[&#xA;&#xA;I don&#39;t have strong opinions on Facebook - I&#39;m not even a user anymore - but I think that the &#34;like/dislike&#34; mania is going a bit too far. I&#39;ve read yesterday that an engineer was fired from Facebook for having a YouTube channel, but that&#39;s beyond the scope of my post today. In one of his videos, he says that FB culture is driven by likes - you need to be popular, not to be good.&#xA;&#xA;Well, I actually think that may be appropriate for Facebook. Eat your own dogfood - isn&#39;t that one of the golden rules for just any product?&#xA;&#xA;But, what I hate nowadays is the curse of the downvote (or the upvote). It&#39;s everywhere: on Facebook, on Reddit, even on Hacker News - you get downvoted, then probably your comment is hidden from most people. The most upvoted comment (or article) gets more impressions.&#xA;&#xA;But what does an upvote, or a like, or a share, or a retweet, or a downvote, or a flag, actually mean? I think we&#39;ve lost the context.&#xA;&#xA;When I still was on Facebook, and some of my friends shared an hoax or just some falsehood, I was usually quick to point that out - with due references. Then they just told me &#34;hey, I shared the article. I&#39;m not the one who wrote that. I may not even agree with that&#34;. So... what does &#34;share&#34; mean?&#xA;&#xA;I think we need better connections for our actions. What&#39;s an upvote (or a retweet or a reshare) ? Does it mean &#34;I agree&#34;? Does it mean &#34;I have reasons to believe the poster is factually correct&#34;? Does it mean &#34;I like it for no particular reason&#34;?&#xA;&#xA;What&#39;s a downvote? Does it mean &#34;I think the poster is factually incorrect&#34;? Or &#34;I don&#39;t like this opinion&#34;?&#xA;&#xA;What&#39;s a flag (in HN) or a report in other sites ? Does it mean &#34;The poster suggests something illegal&#34;? Does it mean &#34;I think it&#39;s an hoax, fake news, just trolling&#34;? Or it just means &#34;I wouldn&#39;t like to see it there, throw it away&#34;. What does that REALLY imply?&#xA;&#xA;Yes, maybe that&#39;s complex. &#34;Upvote&#34; or &#34;Downvote&#34; seems easier. But that&#39;s not just our opinion anymore. Upvotes and downvotes do shape conversations and discussions, and using them unwisely just perpetrates echo chambers and unhealthy ideological silos.&#xA;&#xA;I think we can - and we should - do better. Much better.&#xA;&#xA;Maybe we should show comments / articles in a random fashion, and let people &#34;grade&#34; them by quality of the opinion and/or correctness. And only after a while, when commenting stops, we should create a &#34;top ten&#34; for our comments; or something like that; we should make sure that more than one idea gets exposure, rather than going from the start with a &#34;winner takes it all&#34; mentality. It&#39;s not good for anybody.&#xA;&#xA;Photo by Fabian Gieske on Unsplash&#xA;&#xA;#Ollivander #rants #thoughts&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://images.unsplash.com/photo-1551730459-92db2a308d6a?ixlib=rb-1.2.1&amp;q=80&amp;fm=jpg&amp;crop=entropy&amp;cs=tinysrgb&amp;w=1080&amp;fit=max&amp;ixid=eyJhcHBfaWQiOjExNzczfQ" alt=""></p>

<p>I don&#39;t have strong opinions on Facebook – I&#39;m not even a user anymore – but I think that the “like/dislike” mania is going a bit too far. I&#39;ve read yesterday that an engineer <a href="https://www.cnbc.com/2019/09/19/ex-facebook-engineer-patrick-shyu-makes-fun-of-company-on-youtube.html" rel="nofollow">was fired from Facebook for having a YouTube channel</a>, but that&#39;s beyond the scope of my post today. In one of his videos, he says that FB culture is driven by likes – you need to be popular, not to be good.</p>

<p>Well, I actually think that may be appropriate for Facebook. Eat your own dogfood – isn&#39;t that one of the golden rules for just any product?</p>

<p>But, what I hate nowadays is the curse of the downvote (or the upvote). It&#39;s everywhere: on Facebook, on Reddit, even on Hacker News – you get downvoted, then probably your comment is hidden from most people. The most upvoted comment (or article) gets more impressions.</p>

<p>But what does an upvote, or a like, or a share, or a retweet, or a downvote, or a flag, actually mean? I think we&#39;ve lost the context.</p>

<p>When I still was on Facebook, and some of my friends shared an hoax or just some falsehood, I was usually quick to point that out – with due references. Then they just told me “hey, I shared the article. I&#39;m not the one who wrote that. I may not even agree with that”. So... what does “share” mean?</p>

<p>I think we need better connections for our actions. What&#39;s an upvote (or a retweet or a reshare) ? Does it mean “I agree”? Does it mean “I have reasons to believe the poster is factually correct”? Does it mean “I like it for no particular reason”?</p>

<p>What&#39;s a downvote? Does it mean “I think the poster is factually incorrect”? Or “I don&#39;t like this opinion”?</p>

<p>What&#39;s a flag (in HN) or a report in other sites ? Does it mean “The poster suggests something illegal”? Does it mean “I think it&#39;s an hoax, fake news, just trolling”? Or it just means “I wouldn&#39;t like to see it there, throw it away”. What does that REALLY imply?</p>

<p>Yes, maybe that&#39;s complex. “Upvote” or “Downvote” seems easier. <strong>But that&#39;s not just our opinion anymore.</strong> Upvotes and downvotes do shape conversations and discussions, and using them unwisely just perpetrates echo chambers and unhealthy ideological silos.</p>

<p>I think we can – <strong>and we should -</strong> do better. Much better.</p>

<p>Maybe we should show comments / articles in a random fashion, and let people “grade” them by quality of the opinion and/or correctness. And only after a while, when commenting stops, we should create a “top ten” for our comments; or something like that; we should make sure that more than one idea gets exposure, rather than going from the start with a “winner takes it all” mentality. It&#39;s not good for anybody.</p>

<p>Photo by <a href="https://unsplash.com/@fbngsk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Fabian Gieske</a> on <a href="https://unsplash.com/s/photos/agree?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Unsplash</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:rants" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">rants</span></a> <a href="/alan/tag:thoughts" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">thoughts</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/the-curse-of-the-downvote</guid>
      <pubDate>Sat, 21 Sep 2019 12:07:46 +0000</pubDate>
    </item>
    <item>
      <title>Machine Learning: a sound primer</title>
      <link>https://www.franzoni.eu/alan/machine-learning-a-sound-primer</link>
      <description>&lt;![CDATA[&#xA;&#xA;I see many people who would like to take a glimpse at machine learning, and try to understand a bit how it works. Very often, they can either get pre-baked examples with very specific (and possibly too advanced) approaches - like deep learning - or math-oriented explanations that can be dry or just uninteresting.&#xA;&#xA;I recently discovered a rather famous free textbook that I hadn&#39;t touched before: An Introduction to Statistical Learning . As you may infer by the non-glamorous title, that&#39;s a book that doesn&#39;t try to sell you something fancy about machine learning. It&#39;s quite a practical and non math-heavy introduction to most useful machine learning topics, which will lead the reader to develop an intuition for what ML methods do. SPOILER: neural networks aren&#39;t covered! So, if you&#39;re just running after the hype, that&#39;s not the book for you.&#xA;&#xA;The only real drawback from the original book is that most examples and demos are coded in R. I don&#39;t especially like the language, as it is highly specialized and, most probably, you&#39;ll need to know another language beyond it for general-purpose processing.&#xA;&#xA;So, I&#39;m happy to link a couple of repositories that offer most examples from the book, but coded in Python; those should be more accessible to most people, as the language is very widespread:&#xA;&#xA;https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning&#xA;&#xA;https://github.com/JWarmenhoven/ISLR-python&#xA;&#xA;Happy StatLearning!&#xA;&#xA;EDIT:&#xA;&#xA;There&#39;s a MOOC as well covering most of the topics from the book, by the same original authors&#xA;&#xA;Photo by Lacie Slezak on Unsplash&#xA;&#xA;#Ollivander #machinelearning #statistics&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/07/lacie-slezak-yHG6llFLjS0-unsplash-1-.jpg" alt=""></p>

<p>I see many people who would like to take a glimpse at machine learning, and try to understand a bit how it works. Very often, they can either get pre-baked examples with very specific (and possibly too advanced) approaches – like deep learning – or math-oriented explanations that can be dry or just uninteresting.</p>

<p>I recently discovered a rather famous free textbook that I hadn&#39;t touched before: <a href="https://www.statlearning.com/" rel="nofollow">An Introduction to Statistical Learning</a> . As you may infer by the non-glamorous title, that&#39;s a book that doesn&#39;t try to sell you something <em>fancy</em> about machine learning. It&#39;s quite a practical and non math-heavy introduction to most useful machine learning topics, which will lead the reader to develop an intuition for what ML methods do. SPOILER: neural networks aren&#39;t covered! So, if you&#39;re just running after the hype, that&#39;s not the book for you.</p>

<p>The only real drawback from the original book is that most examples and demos are coded in R. I don&#39;t especially like the language, as it is highly specialized and, most probably, you&#39;ll need to know another language beyond it for general-purpose processing.</p>

<p>So, I&#39;m happy to link a couple of repositories that offer most examples from the book, but coded in Python; those should be more accessible to most people, as the language is very widespread:</p>

<p><a href="https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning" rel="nofollow">https://github.com/tdpetrou/Machine-Learning-Books-With-Python/tree/master/Introduction%20to%20Statistical%20Learning</a></p>

<p><a href="https://github.com/JWarmenhoven/ISLR-python" rel="nofollow">https://github.com/JWarmenhoven/ISLR-python</a></p>

<p>Happy StatLearning!</p>

<p>EDIT:</p>

<p>There&#39;s a <a href="https://www.edx.org/course/statistical-learning" rel="nofollow">MOOC as well covering most of the topics from the book</a>, by the same original authors</p>

<p><em>Photo by <a href="https://unsplash.com/@nbb_photos?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Lacie Slezak</a> on <a href="https://unsplash.com/search/photos/learning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText" rel="nofollow">Unsplash</a></em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:machinelearning" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">machinelearning</span></a> <a href="/alan/tag:statistics" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">statistics</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/machine-learning-a-sound-primer</guid>
      <pubDate>Wed, 31 Jul 2019 08:40:54 +0000</pubDate>
    </item>
    <item>
      <title>Application authors: please don&#39;t force users into your language or packaging details</title>
      <link>https://www.franzoni.eu/alan/software-authors-please-dont-force-me-into-your-packaging-woes</link>
      <description>&lt;![CDATA[&#xA;&#xA;This story has been boiling in my head since long; today I chose to (finally) publish it.&#xA;&#xA;Long story short: in order to use a certain application, I should not need to understand how to use the language or its packaging ecosystem. Delivery and distribution is a relevant part of your app.&#xA;&#xA;This does not apply to libraries, frameworks, or tools that are highly contextual for a certain language/environment ecosystem, and would be used only by a developer, in any case.&#xA;&#xA;What do I mean? I&#39;ll start with an example. It&#39;s chronological, it&#39;s just the latest thing I came into; here&#39;s b2, the command line tool to get access to Backblaze B2 backup repositories. And those are its installation instructions:&#xA;&#xA;GitHub\-\Backblaze\B2\Command\Line\Tool\\The\command-line\tool\that\gives\easy\access\to\all\of\the\capabilities\of\B2\Cloud\Storage&#xA;&#xA;What is pip?&#xA;Is such command safe to use?&#xA;Will it work in all situations?&#xA;  answers:&#xA;Python&#39;s package manager. It makes the b2 library available, and such library has the so-called b2 script, which exposes the b2 CLI executable.&#xA;possibly safe, but will install b2 within your global/per-user python dependencies, and may alter or install additional packages as dependencies, and such deps may be picked by other, unrelated software in the system;&#xA;it may require root to be run correctly.&#xA;&#xA;Should every user of Backblaze B2 CLI know how to code in Python and understand how the Python environment works? I think that should not be necessary.&#xA;&#xA;b2 is just an example, other famous tools - awscli and fpm come to my mind - just follow suit.&#xA;&#xA;Most probably, if the tool you&#39;re downloading is written in C or C++, you&#39;ll probably expect to find a compiled binary, at least for some OSes and architectures; The same applies for Go, which makes binary creation for multiple platforms exceptionally easy. You should strive to provide some kind of equivalent piece of software for your users. .&#xA;&#xA;But, even for tools written in Python or JavaScript (they look to be the ones that suffer most from the problem I&#39;m describing), in most situations, the &#34;right thing&#34; to do is to provide a standalone binary, regardless of how it&#39;s written. You can provide standalone binaries for some archs (this is what borg does - it&#39;s a great backup utility written in Python), or you can rely on native or add-on packaging (additional repositories for Linux, choco for Windows, homebrew for Mac), like httpie. And, when you create such binary or package, you should make sure it&#39;s totally standalone (i.e. it doesn&#39;t alter global system state).&#xA;&#xA;Why? Because leveraging packaging tools that were originally meant for developers of a language is brittle and risky, and can be hard to understand. You may end up compromising the system integrity if the wrong dependency slips into globally installed packages - this is especially true for Python, which is often used by many, many system tools. And, if all third-party apps did it that way, we could have continuous breakages. You can&#39;t assume yours is the only application installed in a certain system!&#xA;&#xA;And, if you&#39;re writing an app, you can have full control of your execution environment and dependencies. You can choose a single Python or NodeJS version, and all of its dependencies. You don&#39;t need to support tons of variations!&#xA;&#xA;In some situations, a CLI evolved out of a library. This is the case for pygments, a very nice syntax highlighting library. It&#39;s meant to be used as a library from inside python projects, but it also exposes a widely used pygmentize binary that is employed by many other tools. I&#39;d love to have that CLI tool available as an independent, standalone package!&#xA;&#xA;Final considerations:&#xA;&#xA;As a software author, think whether you&#39;re building a library, a framework, or an application. If you&#39;re writing an application, think how you can deliver it to your final users, without the need for them to understand how you&#39;ve written it.&#xA;If you&#39;re a maintainer for a package, think about what you&#39;re maintaining. If something is both a library and a command line tool, consider creating multiple (possibly independent) packages.&#xA;&#xA;In the meantime:&#xA;If you find a Python CLI tool that you&#39;d like to use, and it uses the pip way and you don&#39;t to mess with system dependencies, I recommend pipsi. I don&#39;t know if something similar exists for Node or other environments.&#xA;&#xA;FAQ:&#xA;Q: But I&#39;m a solo open source developer! Packaging this way would steal too much of my time!&#xA;A: That&#39;s totally fine. Just make sure you don&#39;t suggest dangerous things to people. If you&#39;re using a developer-only approach, just state that as your target; but the tools I&#39;ve written about above (b2 and awscli) certainly are of a different class.&#xA;&#xA;Q: Any suggestion for easy packaging?&#xA;A: I find homebrew very nice for Mac packaging and distribution. For Linux, I use fpm with docker - I&#39;ve actually created a specific integration project fpm-within-docker. For Windows... no idea! I know about chocolatey and that&#39;s about it.&#xA;&#xA;Photo by Malcolm Lightbody on Unsplash&#xA;&#xA;#Ollivander #packaging #software #delivery&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2019/01/malcolm-lightbody-1081282-unsplash.jpg" alt=""></p>

<p>This story has been boiling in my head since long; today I chose to (finally) publish it.</p>

<p>Long story short: <strong>in order to use a certain application, I should not need to understand how to use the language or its packaging ecosystem. Delivery and distribution is a relevant part of your app.</strong></p>

<p>This does <strong>not</strong> apply to libraries, frameworks, or tools that are highly contextual for a certain language/environment ecosystem, and would be used <strong>only</strong> by a developer, in any case.</p>

<p>What do I mean? I&#39;ll start with an example. It&#39;s chronological, it&#39;s just the latest thing I came into; here&#39;s <a href="https://github.com/Backblaze/B2_Command_Line_Tool" rel="nofollow">b2</a>, the <strong>command line tool</strong> to get access to Backblaze B2 backup repositories. And those are its installation instructions:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2019/01/GitHub_-_Backblaze_B2_Command_Line_Tool__The_command-line_tool_that_gives_easy_access_to_all_of_the_capabilities_of_B2_Cloud_Storage.png" alt="GitHub\_-\_Backblaze\_B2\_Command\_Line\_Tool\_\_The\_command-line\_tool\_that\_gives\_easy\_access\_to\_all\_of\_the\_capabilities\_of\_B2\_Cloud\_Storage"></p>
<ul><li>What is pip?</li>
<li>Is such command safe to use?</li>
<li>Will it work in all situations?
<em>answers:</em></li>
<li>Python&#39;s package manager. It makes the b2 library available, and such library has the so-called b2 script, which exposes the b2 CLI executable.</li>
<li>possibly safe, but will install b2 within your global/per-user python dependencies, and may alter or install additional packages as dependencies, and such deps may be picked by other, unrelated software in the system;</li>
<li>it may require root to be run correctly.</li></ul>

<p>Should every user of Backblaze B2 CLI know how to code in Python and understand how the Python environment works? I think that should <strong>not</strong> be necessary.</p>

<p>b2 is just an example, other famous tools – <a href="https://aws.amazon.com/cli/" rel="nofollow">awscli</a> and <a href="https://fpm.readthedocs.io/en/latest/installing.html" rel="nofollow">fpm</a> come to my mind – just follow suit.</p>

<p>Most probably, if the tool you&#39;re downloading is written in C or C++, you&#39;ll probably expect to find a compiled binary, at least for some OSes and architectures; The same applies for Go, which makes binary creation for multiple platforms exceptionally easy. <strong>You should strive to provide some kind of equivalent piece of software for your users.</strong> .</p>

<p>But, even for tools written in Python or JavaScript (they look to be the ones that suffer most from the problem I&#39;m describing), in most situations, the “right thing” to do is to provide a standalone binary, <strong>regardless of how it&#39;s written.</strong> You can provide standalone binaries for some archs (this is what <a href="https://github.com/borgbackup/borg" rel="nofollow">borg</a> does – it&#39;s a great backup utility written in Python), or you can rely on native or add-on packaging (additional repositories for Linux, choco for Windows, homebrew for Mac), like <a href="https://httpie.org/" rel="nofollow">httpie</a>. And, when you create such binary or package, you should make sure it&#39;s totally standalone (i.e. it doesn&#39;t alter global system state).</p>

<p>Why? Because leveraging packaging tools that were originally meant for developers of a language is <strong>brittle and risky</strong>, and can be hard to understand. You may end up compromising the system integrity if the wrong dependency slips into globally installed packages – this is especially true for Python, which is often used by many, many system tools. And, if all third-party apps did it that way, we could have continuous breakages. <strong>You can&#39;t assume yours is the only application installed in a certain system!</strong></p>

<p>And, if you&#39;re writing an app, you can have <strong>full control</strong> of your execution environment and dependencies. You can choose a single Python or NodeJS version, and all of its dependencies. You don&#39;t need to support tons of variations!</p>

<p>In some situations, a CLI evolved out of a library. This is the case for <a href="http://pygments.org/" rel="nofollow">pygments</a>, a very nice syntax highlighting library. It&#39;s meant to be used as a library from inside python projects, but it also exposes a widely used <a href="http://pygments.org/docs/cmdline/" rel="nofollow">pygmentize</a> binary that is employed by many other tools. I&#39;d love to have that CLI tool available as an independent, standalone package!</p>

<p>Final considerations:</p>
<ul><li>As a software author, think whether you&#39;re building a library, a framework, or an application. If you&#39;re writing an application, think how you can deliver it to your final users, without the need for them to understand how you&#39;ve written it.</li>
<li>If you&#39;re a maintainer for a package, think about what you&#39;re maintaining. If something is both a library and a command line tool, consider creating multiple (possibly independent) packages.</li></ul>

<p>In the meantime:
If you find a Python CLI tool that you&#39;d like to use, and it uses the <em>pip way</em> and you don&#39;t to mess with system dependencies, I recommend <a href="https://github.com/mitsuhiko/pipsi" rel="nofollow">pipsi</a>. I don&#39;t know if something similar exists for Node or other environments.</p>

<p>FAQ:
Q: But I&#39;m a solo open source developer! Packaging this way would steal too much of my time!
A: That&#39;s totally fine. Just make sure you don&#39;t suggest dangerous things to people. If you&#39;re using a developer-only approach, just state that as your target; but the tools I&#39;ve written about above (b2 and awscli) certainly are of a different class.</p>

<p>Q: Any suggestion for easy packaging?
A: I find <a href="https://brew.sh/" rel="nofollow">homebrew</a> very nice for Mac packaging and distribution. For Linux, I use <a href="https://github.com/jordansissel/fpm" rel="nofollow">fpm</a> with <a href="https://www.docker.com/" rel="nofollow">docker</a> – I&#39;ve actually created a specific integration project <a href="https://github.com/alanfranz/fpm-within-docker" rel="nofollow">fpm-within-docker</a>. For Windows... no idea! I know about <a href="https://chocolatey.org/" rel="nofollow">chocolatey</a> and that&#39;s about it.</p>

<p>Photo by Malcolm Lightbody on <a href="https://unsplash.com/photos/401OD83Ke6o" rel="nofollow">Unsplash</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:packaging" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">packaging</span></a> <a href="/alan/tag:software" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">software</span></a> <a href="/alan/tag:delivery" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">delivery</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/software-authors-please-dont-force-me-into-your-packaging-woes</guid>
      <pubDate>Thu, 10 Jan 2019 17:10:00 +0000</pubDate>
    </item>
    <item>
      <title>Productivity, the office, and the open floor plan</title>
      <link>https://www.franzoni.eu/alan/productivity-the-office-and-an-open-floor-plan</link>
      <description>&lt;![CDATA[&#xA;&#xA;UPDATE: this article was written at a time when remote work wasn&#39;t so widespread. After having worked remotely myself, I don&#39;t think that the office is so critical anymore, even though I believe it&#39;s still useful to have a nice office somewhere where you can meet people in person, every now and then. But a very nice office is even more important nowadays; if your office sucks, there&#39;re zero chance that people will want to go there!&#xA;&#xA;There&#39;s one pattern that, nowadays, I find amusing; the productivity mantra is repeated everywhere. Everybody wants to get more productive, every company is trying to make their employees more productive. Robotics, AI: everything calls for it.&#xA;&#xA;From Wikipedia:&#xA;&#xA;  Productivity describes various measures of the efficiency of production. A productivity measure is expressed as the ratio of output to inputs used in a production process, i.e. output per unit of input&#xA;&#xA;So: productivity is a ratio.&#xA;&#xA;That&#39;s a great idea! Let&#39;s do more work in less time, and let&#39;s use the rest of our time in other activities - be it study, research, exercise, leisure time, or whatever.&#xA;&#xA;Or, let&#39;s do more work in the same time as before! That&#39;s a good idea as well; we&#39;ll achieve more, and maybe, if our company recognizes such additional value, get paid more.&#xA;&#xA;But then. Open floor plans. Shared desks.&#xA;&#xA;annie-spratt-604131-unsplash-1-&#xA;&#xA;How can those things go together?&#xA;&#xA;Focusing on software development/tech firms, there seem to exist a widespread discontent around open floor plans, and yet the number of companies adopting them seem to be growing - I don&#39;t have exact data, but it seems to be a new trend. Traditional corporations that want to start behaving like startups start ditching private offices. The usual reason: an open office workspace promotes collaboration. Yes, maybe it does. But do all the employees in a company just collaborate between each other all the time?&#xA;&#xA;Most probably, they won&#39;t. Most probably, they&#39;ll need plenty of time to focus and do deep work. The risk is to overoptimize for a single aim, and forget the whole picture.&#xA;&#xA;More often than not, an open office plan is a marketing synonym for let&#39;s cram how many people we can, quite at random, in the smallest possible space.&#xA;&#xA;Incidentally, I think that open office workspaces are one of the reasonssup[1]/sup that actually make 100% remote work positions effective: the productivity drop of open workspaces is larger than the drop caused by collaboration/communication difficulties in remote work situations.&#xA;&#xA;Personally, I think that a good office trumps any other work environment/style in terms of productivity. Of course, life is a multi-objective optimization; can you find all the employees you need around a specific location? Are you paying high enough for those employees to live comfortably around that office? If any of those answers are not a definite yes, hybrid or full remote makes a lot of sense!&#xA;&#xA;But, how should a good office be organized? I have experienced a variety of work environments and, well, in my opinion a sort-of open space rooms work fine, even great, but there&#39;re some dispositions:&#xA;&#xA;Don&#39;t make them too large. Around 100/150msup2/sup (1000-1500 sqft) should be ok to accommodate enough people working together.&#xA;Reserve enough space for each worker. Usually 7-10msup2/sup (70-100 sqft) for each person is a good guidance.&#xA;Make sure desks are personal, and they&#39;re deep and wide enough. No desk should be narrower than 150cm (60 inches), and ideally you should aim at 180/200cm (70-80 inches). It should be possible for two people to sit at the same desk at any time, without messing with table legs, other people&#39;s legs, or any other item. Pursue no friction for collaboration.&#xA;Provide some place (lockers, cabinets, whatever) to let people put their personal and work belongings in, so the desktop area can be tidy and clear.&#xA;Provide good tools. Large screens, powerful workstations, if needed.&#xA;Make sure that both visual and auditory noises are minimal. While at his desk, any worker should not be able to see their peers&#39; monitors. Phones should be silenced. People requiring to make constant noise (e.g. salesmen, tech support) should not be put in the same room as deep-working people (e.g. software developers). Carpeted floors and doors that can actually be closed can be very useful.&#xA;Respect privacy and safety. People should not fear shoulder surfing or that anybody could approach them without being seen, because that prevents many workers from getting relaxed enough to enter a flow state of mind.&#xA;Offices should not be hallways. There should be no need for somebody to cross a room just to reach another place.&#xA;Enforce a behavioural code. Talking is totally permitted, albeit in a low voice and not shouting around the room. I find pair programming 100% fine in such environments.&#xA;Provide other community space, with whiteboards, large screens, coffee, snacks, where people can meet &amp; discuss when they need to collaborate rather than focus.&#xA;&#xA;Using this kind of office space, you can accommodate 10-15 developers in a room, and change their position when they need to constantly work with somebody else. Collaboration turns out to work out properly, but distractions are limited. A decent trade-off.&#xA;&#xA;What I finally argue is: if you&#39;re using an open floor plan just to cram more people in the same space, you&#39;re not improving productivity. You&#39;re (possibly) just improving total production. Because more people working in a bad environment could still produce more than fewer people in a good environment.&#xA;&#xA;But, take some time to do some calculations. Your engineers are possibly the most expensive part of your budget. Are you totally sure that you&#39;d want to spend a conspicuous amount of money for their wage, and then let them work at a fraction of their potential efficiency? This is especially relevant within the context of remote or hybrid work. Try comparing hybrid/full remote expenses to the ones of having everybody in good office, not in a terrible one.&#xA;&#xA;Sure, real estate and rents are a cost. A distracting work environment won&#39;t reduce their abilities 1% or 2%. I&#39;d speculate that the productivity drop can easily reach 20%-50%. Take note of your &#34;developer density&#34; and when it&#39;s approaching the limit, start looking for new office space. Don&#39;t wait to be Too Crammed To Do Anything Useful.&#xA;&#xA;Sometimes I hear somebody say that &#34;hey, that&#39;s how it works nowadays, adapt or be ejected, I can work with noise and distractions&#34;. Let&#39;s suppose that such people actually exist and do a lot of great work; how many of them can you recruit in the current state of talent crunch? And, are you really sacrificing a lot people that could do great work on the altar of smaller offices? Remember: when creating an office space for a team, you should think about the whole team, a lot of different people. Even though it works for you, it may not work for them. That&#39;s true for hybrid/remote as well; you need to select the right people to work at the office, and those may not be the best remote workers; and vice-versa.&#xA;&#xA;UPDATE July, 20th 2018:&#xA;There&#39;re more and more people that agree with me about open floor plans, in that they&#39;re detrimental to productivity and conversation:&#xA;&#xA;https://theconversation.com/a-new-study-should-be-the-final-nail-for-open-plan-offices-99756&#xA;https://code.likeagirl.io/a-research-roundup-to-show-that-your-office-layout-is-toxic-and-some-tips-for-making-it-better-8434864b0ab2&#xA;https://m.signalvnoise.com/the-open-plan-office-is-a-terrible-horrible-no-good-very-bad-idea-42bd9cd294e3&#xA;https://joshuatdean.com/wp-content/uploads/2020/02/NoiseCognitiveFunctionandWorkerProductivity.pdf&#xA;&#xA;Photos by Breather and Annie Spratt&#xA;&#xA;------------------------------------------------------------------------&#xA;&#xA;The other one being long commutes that waste time and destroy workers&#39; morale.&#xA;&#xA;#Ollivander #productivity #industry&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2018/04/breather-176228-unsplash-1-.jpg" alt=""></p>

<p><em>UPDATE: this article was written at a time when remote work wasn&#39;t so widespread. After having worked remotely myself, I don&#39;t think that the office is so critical anymore, even though I believe it&#39;s still useful to have a nice office somewhere where you can meet people in person, every now and then. But a <strong>very nice</strong> office is even more important nowadays; if your office sucks, there&#39;re zero chance that people will want to go there!</em></p>

<p>There&#39;s one pattern that, nowadays, I find amusing; the <strong>productivity</strong> mantra is repeated everywhere. Everybody wants to get more productive, every company is trying to make their employees more productive. Robotics, AI: everything calls for it.</p>

<p>From Wikipedia:</p>

<blockquote><p>Productivity describes various measures of the efficiency of production. A productivity measure is expressed as the ratio of output to inputs used in a production process, i.e. output per unit of input</p></blockquote>

<p>So: <em>productivity is a ratio</em>.</p>

<p><strong>That&#39;s a great idea!</strong> Let&#39;s do more work in less time, and let&#39;s use the rest of our time in other activities – be it study, research, exercise, leisure time, or whatever.</p>

<p>Or, <strong>let&#39;s do more work in the same time as before!</strong> That&#39;s a good idea as well; we&#39;ll achieve more, and maybe, if our company recognizes such additional value, get paid more.</p>

<p>But then. <a href="https://www.inc.com/geoffrey-james/apple-employees-hate-apples-5-billion-open-plan-o.html" rel="nofollow">Open floor plans</a>. Shared desks.</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2018/04/annie-spratt-604131-unsplash-1-.jpg" alt="annie-spratt-604131-unsplash-1-"></p>

<p>How can those things go together?</p>

<p>Focusing on software development/tech firms, there seem to exist a <a href="https://news.ycombinator.com/item?id=16864925" rel="nofollow">widespread</a> <a href="https://www.dezeen.com/2017/08/10/apple-park-campus-employees-rebel-over-open-plan-offices-architecture-news/" rel="nofollow">discontent</a> around open floor plans, and yet the number of companies adopting them seem to be growing – I don&#39;t have exact data, but it seems to be a new trend. Traditional corporations that want to start behaving like startups start ditching private offices. The usual reason: <em>an open office workspace promotes collaboration</em>. Yes, maybe it does. But <strong>do all the employees in a company just collaborate between each other all the time?</strong></p>

<p>Most probably, they won&#39;t. Most probably, they&#39;ll need plenty of time to <strong>focus</strong> and do <a href="https://www.amazon.com/Deep-Work-Focused-Success-Distracted/dp/1455586692" rel="nofollow"><strong>deep work</strong></a>. The risk is to <strong>overoptimize</strong> for a single aim, and forget the whole picture.</p>

<p>More often than not, an open office plan is a marketing synonym for <em>let&#39;s cram how many people we can, quite at random, in the smallest possible space</em>.</p>

<p>Incidentally, I think that <strong>open office workspaces are one of the reasons<sup>[1]</sup> that actually make 100% remote work positions effective:</strong> the productivity drop of open workspaces is larger than the drop caused by collaboration/communication difficulties in remote work situations.</p>

<p>Personally, I think that a <strong>good office trumps any other work environment/style in terms of productivity</strong>. Of course, life is a multi-objective optimization; can you find all the employees you need around a specific location? Are you paying high enough for those employees to live comfortably around that office? <strong>If any of those answers are not a definite yes, hybrid or full remote makes a lot of sense</strong>!</p>

<p>But, how should a good office be organized? I have experienced a variety of work environments and, well, in my opinion a sort-of open space rooms work fine, even great, but there&#39;re some dispositions:</p>
<ul><li>Don&#39;t make them too large. Around 100/150m<sup>2</sup> (1000-1500 sqft) should be ok to accommodate enough people working together.</li>
<li>Reserve enough space for each worker. Usually 7-10m<sup>2</sup> (70-100 sqft) for each person is a good guidance.</li>
<li>Make sure desks are personal, and they&#39;re deep and wide enough. No desk should be narrower than 150cm (60 inches), and ideally you should aim at 180/200cm (70-80 inches). <strong>It should be possible for two people to sit at the same desk at any time</strong>, without messing with table legs, other people&#39;s legs, or any other item. <strong>Pursue no friction for collaboration.</strong></li>
<li>Provide some place (lockers, cabinets, whatever) to let people put their personal and work belongings in, so the desktop area can be tidy and clear.</li>
<li>Provide good tools. Large screens, powerful workstations, if needed.</li>
<li>Make sure that both <em>visual</em> and <em>auditory</em> noises are minimal. While at his desk, any worker should <em>not</em> be able to see their peers&#39; monitors. Phones should be silenced. People requiring to make constant noise (e.g. salesmen, tech support) should not be put in the same room as deep-working people (e.g. software developers). Carpeted floors and doors that can actually be closed can be very useful.</li>
<li>Respect <strong>privacy</strong> and <strong>safety</strong>. People should not fear shoulder surfing or that anybody could approach them without being seen, because that prevents many workers from getting relaxed enough to enter a <em>flow</em> state of mind.</li>
<li>Offices <strong>should not be hallways</strong>. There should be no need for somebody to cross a room just to reach another place.</li>
<li>Enforce a behavioural code. <strong>Talking is totally permitted</strong>, albeit in a low voice and not shouting around the room. I find pair programming 100% fine in such environments.</li>
<li>Provide other community space, with whiteboards, large screens, coffee, snacks, where people can meet &amp; discuss when they need to collaborate rather than focus.</li></ul>

<p>Using this kind of office space, you can accommodate 10-15 developers in a room, and change their position when they need to constantly work with somebody else. Collaboration turns out to work out properly, but distractions are limited. A decent trade-off.</p>

<p>What I finally argue is: if you&#39;re using an open floor plan just to cram more people in the same space, <strong>you&#39;re not improving productivity. You&#39;re (possibly) just improving total production</strong>. Because more people working in a bad environment <em>could</em> still produce more than fewer people in a good environment.</p>

<p>But, <strong>take some time to do some calculations</strong>. Your engineers are possibly the most expensive part of your budget. Are you totally sure that you&#39;d want to spend a conspicuous amount of money for their wage, and then let them work at a fraction of their potential efficiency? This is <strong>especially relevant</strong> within the context of remote or hybrid work. Try comparing hybrid/full remote expenses to the ones of having everybody in <strong>good office</strong>, not in a terrible one.</p>

<p>Sure, real estate and rents are a cost. A distracting work environment won&#39;t reduce their abilities 1% or 2%. I&#39;d speculate that the productivity drop can easily reach 20%-50%. Take note of your “developer density” and when it&#39;s approaching the limit, start looking for new office space. Don&#39;t wait to be Too Crammed To Do Anything Useful.</p>

<p>Sometimes I hear somebody say that “hey, that&#39;s how it works nowadays, adapt or be ejected, I can work with noise and distractions”. Let&#39;s suppose that such people actually exist and do a lot of great work; how many of them can you recruit in the current state of talent crunch? And, are you really sacrificing a lot people that could do great work on the altar of smaller offices? Remember: when creating an office space for a team, you should think about <strong>the whole team</strong>, a lot of different people. Even though it works for you, it may not work for them. That&#39;s true for hybrid/remote as well; you need to select the right people to work at the office, and those may not be the best remote workers; and vice-versa.</p>

<p>UPDATE July, 20th 2018:
There&#39;re more and more people that agree with me about open floor plans, in that they&#39;re detrimental to productivity and conversation:</p>

<p><a href="https://theconversation.com/a-new-study-should-be-the-final-nail-for-open-plan-offices-99756" rel="nofollow">https://theconversation.com/a-new-study-should-be-the-final-nail-for-open-plan-offices-99756</a>
<a href="https://code.likeagirl.io/a-research-roundup-to-show-that-your-office-layout-is-toxic-and-some-tips-for-making-it-better-8434864b0ab2" rel="nofollow">https://code.likeagirl.io/a-research-roundup-to-show-that-your-office-layout-is-toxic-and-some-tips-for-making-it-better-8434864b0ab2</a>
<a href="https://m.signalvnoise.com/the-open-plan-office-is-a-terrible-horrible-no-good-very-bad-idea-42bd9cd294e3" rel="nofollow">https://m.signalvnoise.com/the-open-plan-office-is-a-terrible-horrible-no-good-very-bad-idea-42bd9cd294e3</a>
<a href="https://joshuatdean.com/wp-content/uploads/2020/02/NoiseCognitiveFunctionandWorkerProductivity.pdf" rel="nofollow">https://joshuatdean.com/wp-content/uploads/2020/02/NoiseCognitiveFunctionandWorkerProductivity.pdf</a></p>

<p>Photos by <a href="https://unsplash.com/@breather?utm_medium=referral&amp;utm_campaign=photographer-credit&amp;utm_content=creditBadge" rel="nofollow">Breather</a> and <a href="https://unsplash.com/@anniespratt" rel="nofollow">Annie Spratt</a></p>

<hr>
<ol><li>The other one being long commutes that waste time and destroy workers&#39; morale.</li></ol>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:productivity" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">productivity</span></a> <a href="/alan/tag:industry" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">industry</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/productivity-the-office-and-an-open-floor-plan</guid>
      <pubDate>Wed, 18 Apr 2018 12:43:37 +0000</pubDate>
    </item>
    <item>
      <title>Command line data crunching with Python</title>
      <link>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</link>
      <description>&lt;![CDATA[&#xA;&#xA;Every time I&#39;m doing some data crunching on the command line, I find myself juggling between sed, awk, sort, uniq, etc. While I like the UNIX way of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.&#xA;&#xA;I know that Perl and Ruby support implicit loops / prints - see this and that. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.&#xA;&#xA;On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy - the -c switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:&#xA;&#xA;$ python -c &#39;import sys;for x in sys.stdin:print x&#39;&#xA;&#xA;won&#39;t &#34;just work&#34;:&#xA;&#xA;  File &#34;string&#34;, line 1&#xA;    import sys;for x in [1,5]:    print x;print x&#xA;                 ^&#xA;SyntaxError: invalid syntax&#xA;&#xA;But: there&#39;s a bash feature to interpret escape sequences in single-quoted strings, so this will work fine:&#xA;&#xA;$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;hello&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.&#xA;&#xA;AN IMPORTANT NOTE: if you&#39;re treating non-ascii data, I suggest you set the PYTHONIOENCODING variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:&#xA;&#xA;echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;&#xA;ààà&#xA;world&#xA;this&#xA;is&#xA;me&#xA;&#xA;Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend this book enough.&#xA;&#xA;Photo by Daniel Cheung on Unsplash&#xA;&#xA;#Ollivander #python #datacrunching #MostlyUnixish&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2018/02/daniel-cheung-129839-1-.jpg" alt=""></p>

<p>Every time I&#39;m doing some data crunching on the command line, I find myself juggling between <em>sed, awk, sort, uniq, etc</em>. While I like <em>the UNIX way</em> of having one tool doing one thing well, I sometimes find it slightly boring to put all the tools together, sometimes stretching their features a bit too much.</p>

<p>I know that Perl and Ruby support implicit loops / prints – see <a href="http://www.wellho.net/resources/ex.php4?item=p210/spot" rel="nofollow">this</a> and <a href="http://www.wellho.net/resources/ex.php4?item=r110/ruby_awk.rb" rel="nofollow">that</a>. Those switches makes it easy to work with data on the command line, but I don&#39;t use those languages a lot anymore, so I always need to lookup something in online manuals before performing something useful. And I never took my time to learn awk properly, so maybe I wouldn&#39;t need al that.</p>

<p>On the contrary, I still use Python quite a lot, and it&#39;s becoming the de-facto standard for data science purposes. Using it on the command line by piping something in &amp; out of it, by the way, isn&#39;t always so easy – the <code>-c</code> switch allows passing a command in, but it&#39;s not always easy to understand whether a char is being interpreted by bash or by the python interpreter, and Python is whitespace-sensitive, too. So a command line like:</p>

<pre><code>$ python -c &#39;import sys;for x in sys.stdin:print x&#39;
</code></pre>

<p>won&#39;t “just work”:</p>

<pre><code>  File &#34;&lt;string&gt;&#34;, line 1
    import sys;for x in [1,5]:    print x;print x
                 ^
SyntaxError: invalid syntax
</code></pre>

<p>But: there&#39;s a <a href="https://stackoverflow.com/questions/11966312/how-does-the-leading-dollar-sign-affect-single-quotes-in-bash" rel="nofollow">bash feature to interpret escape sequences in single-quoted strings</a>, so this will work fine:</p>

<pre><code>$ echo -e &#34;hello\nworld\nthis\nis\nme&#34; | python -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
hello
world
this
is
me
</code></pre>

<p>I find Python string manipulation to be great and usually fast-enough for not-so-large datasets, so you can do very interesting things and shell out to standard unix commands only if and when you actually need to. As long as you rely on the standard lib only, you&#39;re quite safe about portability, too.</p>

<p><strong>AN IMPORTANT NOTE:</strong> if you&#39;re treating non-ascii data, I suggest you set the <em>PYTHONIOENCODING</em> variable, especially if you&#39;re using Python3, since that interpreter version converts to unicode objects wherever it is possible:</p>

<pre><code>echo -e &#34;ààà\nworld\nthis\nis\nme&#34; | PYTHONIOENCODING=&#39;utf-8&#39; python3 -c $&#39;import sys\nfor x in sys.stdin:\n    print(x.strip())&#39;
ààà
world
this
is
me
</code></pre>

<p>Enjoy your command line! And if you want to become a command line data processing guru, I cannot recommend <a href="https://www.datascienceatthecommandline.com/" rel="nofollow">this book</a> enough.</p>

<p><em>Photo by Daniel Cheung on <a href="https://unsplash.com/photos/cPF2nlWcMY4" rel="nofollow">Unsplash</a></em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:python" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">python</span></a> <a href="/alan/tag:datacrunching" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">datacrunching</span></a> <a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/command-line-data-crunching-with-python</guid>
      <pubDate>Wed, 14 Feb 2018 14:44:51 +0000</pubDate>
    </item>
    <item>
      <title>Shell scripting: short or long format options?</title>
      <link>https://www.franzoni.eu/alan/shell-scripting-when-to-use-the-long-or-the-shor</link>
      <description>&lt;![CDATA[&#xA;&#xA;This is something I get asked quite a lot, so I wanted to write a piece about it.&#xA;&#xA;This is an extract from the manpage from GNU grep:&#xA;&#xA;NAME&#xA;       grep, egrep, fgrep, rgrep - print lines matching a pattern&#xA;&#xA;SYNOPSIS&#xA;       grep [OPTIONS] PATTERN [FILE...]&#xA;       grep [OPTIONS] [-e PATTERN]...  [-f FILE]...  [FILE...]&#xA;&#xA;DESCRIPTION&#xA;       grep  searches  the  named  input  FILEs  for lines containing a match to the given PATTERN.  If no files are specified, or if the file “-” is given, grep searches standard input.  By&#xA;       default, grep prints the matching lines.&#xA;&#xA;       In addition, the variant programs egrep, fgrep and rgrep are the same as grep -E, grep -F, and grep -r, respectively.  These variants are deprecated, but  are  provided  for  backward&#xA;       compatibility.&#xA;&#xA;OPTIONS&#xA;   Generic Program Information&#xA;       --help Output a usage message and exit.&#xA;&#xA;       -V, --version&#xA;              Output the version number of grep and exit.&#xA;&#xA;   Matcher Selection&#xA;       -E, --extended-regexp&#xA;              Interpret PATTERN as an extended regular expression (ERE, see below).&#xA;&#xA;       -F, --fixed-strings&#xA;              Interpret PATTERN as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.&#xA;&#xA;       -G, --basic-regexp&#xA;              Interpret PATTERN as a basic regular expression (BRE, see below).  This is the default.&#xA;&#xA;       -P, --perl-regexp&#xA;              Interpret the pattern as a Perl-compatible regular expression (PCRE).  This is highly experimental and grep -P may warn of unimplemented features.&#xA;&#xA;So, -F is a short format form, while --fixed-strings is a long format form.&#xA;&#xA;When should I use either form?&#xA;&#xA;My take is: when working interactively, do whatever you like. Whenever you are writing a script to be reused, strive to use the long format options as much as possible.&#xA;&#xA;Why?&#xA;&#xA;Nobody cares about what you do on your computer. But whenever you write a script that should, later on, be used and read by other people - or just yourself in the future - the long form is often self-explanatory. Instead of opening the manpage and looking for cryptic -K -X -i, you can just read the option! It will for sure save more time than what you spend while typing a few more chars.&#xA;&#xA;There&#39;re a few exceptions to this rule:&#xA;&#xA;if the option is very common. Feel free to use -E and -P for grep, as they&#39;re very well known and widespread options for this command.&#xA;if you need portability between different OSes (e.g. MacOS+Linux+BSD) and the long format option is not available somewhere; this sometimes happens.&#xA;&#xA;Why is this cool?&#xA;&#xA;Consider this command, which is the recommended way to install the great MacOS package manager Homebrew:&#xA;&#xA;/usr/bin/ruby -e &#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)&#34;&#xA;&#xA;Such curl command is quite common in many installers; can you tell what such options do? Can&#39;t you? Consider this line:&#xA;&#xA;/usr/bin/ruby -e &#34;$(curl --fail --silent --show-error --location https://raw.githubusercontent.com/Homebrew/install/master/install)&#34;&#xA;&#xA;I bet you can now understand what it does; maybe the last flag is a bit vague (it tells curl to follow redirects via Location headers), but the overall feeling is simply... great!&#xA;&#xA;Photo by Alex Holyoake on Unsplash&#xA;&#xA;#Ollivander #shell #bash&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2017/09/alex-holyoake-213910--1-.jpg" alt=""></p>

<p>This is something I get asked quite a lot, so I wanted to write a piece about it.</p>

<p>This is an extract from the manpage from GNU grep:</p>

<pre><code>NAME
       grep, egrep, fgrep, rgrep - print lines matching a pattern

SYNOPSIS
       grep [OPTIONS] PATTERN [FILE...]
       grep [OPTIONS] [-e PATTERN]...  [-f FILE]...  [FILE...]

DESCRIPTION
       grep  searches  the  named  input  FILEs  for lines containing a match to the given PATTERN.  If no files are specified, or if the file “-” is given, grep searches standard input.  By
       default, grep prints the matching lines.

       In addition, the variant programs egrep, fgrep and rgrep are the same as grep -E, grep -F, and grep -r, respectively.  These variants are deprecated, but  are  provided  for  backward
       compatibility.

OPTIONS
   Generic Program Information
       --help Output a usage message and exit.

       -V, --version
              Output the version number of grep and exit.

   Matcher Selection
       -E, --extended-regexp
              Interpret PATTERN as an extended regular expression (ERE, see below).

       -F, --fixed-strings
              Interpret PATTERN as a list of fixed strings (instead of regular expressions), separated by newlines, any of which is to be matched.

       -G, --basic-regexp
              Interpret PATTERN as a basic regular expression (BRE, see below).  This is the default.

       -P, --perl-regexp
              Interpret the pattern as a Perl-compatible regular expression (PCRE).  This is highly experimental and grep -P may warn of unimplemented features.
</code></pre>

<p>So, <code>-F</code> is a short format form, while <code>--fixed-strings</code> is a long format form.</p>

<p>When should I use either form?</p>

<p>My take is: when working interactively, do whatever you like. Whenever you are writing a <strong>script to be reused</strong>, strive to use the <strong>long format options as much as possible</strong>.</p>

<p>Why?</p>

<p>Nobody cares about what you do on your computer. But whenever you write a script that should, later on, be used and read by other people – or just yourself in the future – <strong>the long form is often self-explanatory</strong>. Instead of opening the manpage and looking for cryptic <code>-K -X -i</code>, you can just read the option! It will for sure save more time than what you spend while typing a few more chars.</p>

<p>There&#39;re a few exceptions to this rule:</p>
<ul><li>if the option is very common. Feel free to use <code>-E</code> and <code>-P</code> for <code>grep</code>, as they&#39;re very well known and widespread options for this command.</li>
<li>if you need portability between different OSes (e.g. MacOS+Linux+BSD) and the long format option is not available somewhere; this sometimes happens.</li></ul>

<h4 id="why-is-this-cool">Why is this cool?</h4>

<p>Consider this command, which is the recommended way to install the great MacOS package manager <a href="https://brew.sh" rel="nofollow">Homebrew</a>:</p>

<pre><code>/usr/bin/ruby -e &#34;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)&#34;
</code></pre>

<p>Such <code>curl</code> command is quite common in many installers; <strong>can you tell what such options do?</strong> Can&#39;t you? Consider this line:</p>

<pre><code>/usr/bin/ruby -e &#34;$(curl --fail --silent --show-error --location https://raw.githubusercontent.com/Homebrew/install/master/install)&#34;
</code></pre>

<p>I bet you can now understand what it does; maybe the last flag is a bit vague (it tells <code>curl</code> to follow redirects via <code>Location</code> headers), but the overall feeling is simply... great!</p>

<p><em>Photo by Alex Holyoake on Unsplash</em></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:shell" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">shell</span></a> <a href="/alan/tag:bash" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">bash</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/shell-scripting-when-to-use-the-long-or-the-shor</guid>
      <pubDate>Thu, 07 Sep 2017 13:23:00 +0000</pubDate>
    </item>
    <item>
      <title>Stopping The Internet Of Noise - A Useful Internet Back Again</title>
      <link>https://www.franzoni.eu/alan/stopping-the-internet-of-noise</link>
      <description>&lt;![CDATA[&#xA;&#xA;The internet is getting noisy. Too noisy. Having grown up in the nineties, with 56k dial-up, I sometimes struggle to understand how little I&#39;m accomplishing today with all the bandwidth I can leverage.&#xA;&#xA;There were some key factors that made the old internet so productive, by the way, and many of those factors are just gone.&#xA;&#xA;This is not just a rant. I have some proposals as well.&#xA;&#xA;Discussing&#xA;&#xA;Most discussions on the internet once happened in Usenet newsgroups and mailing lists. You could even access mailing lists via NNTP (the newsgroup protocol) using a great service like gmane.&#xA;&#xA;40tude Dialog&#xA;&#xA;40tude Dialog was my favourite newsreader at the time. I could subscribe to a lot of different newsgroups, without needing to sign up for each of them, and it was very easy to catch up with lots of traffic - just a matter of checking which messages had still the title in bold.&#xA;&#xA;There was a single application with a consistent interface to access the most disparate groups, and topics were widely enforced - probably the most common complaint from newsgroup users was that somebody was off topic - such discussions were quickly criticized and got little attention.&#xA;&#xA;Chatting&#xA;&#xA;There once was IRC, the Internet Relay Chat. mIRC was the king of clients for a while.&#xA;&#xA;an irc chat&#xA;&#xA;IRC, sometimes, was not great, lacking some advanced features for multimedia and emojis, but it was good. There were plenty of channels with specific topics, and you could join them in a second.&#xA;&#xA;Then, other instant messengers picked up - ICQ, MSN, AOL, etc - but it wasn&#39;t a great issue; the protocols they used were mostly simple, and tools like Trillian and Pidgin/Adium quickly reversed them and let the users just pick one client with multiple connections. Skype was a bit of a white fly because its original protocol was harder to reverse.&#xA;&#xA;Again, there was an easy way to get a consistent interface in a single application to chat on IRC, Google Chat, Facebook Chat, generic XMPP, and so on.&#xA;&#xA;Getting notified about updates&#xA;&#xA;Oh, this is was easy, wasn&#39;t it? RSS and Atom just ruled the world. Bloglines and the beloved Google Reader were the first stop, every morning, for many developers.&#xA;&#xA;Some blogger just putting too much shit on its blog? We remove him from our feed! No matter what!&#xA;&#xA;google reader&#xA;&#xA;We could simply pull our update sources within one single application, and check all the updates when we had the time to.&#xA;&#xA;I can remember my workflow: scroll through the sources, open any interesting article in a new tab, mark all feeds as read; then go through the tabs until I finished. Job done. I could, at the same time, read a lot of interesting thing and discard a lot of uninteresting shit.&#xA;&#xA;What happened&#xA;&#xA;If you take a look at was the state of things, you can notice that:&#xA;&#xA;Focus is lost; once, everything went around topics. Channels had topics; newsgroups had topics; blog had topics (one great rule for a blog was to have a &#39;razor sharp&#39; focus). *Nowadays we have people instead of topics. I have nothing against people, but maybe, if I follow a great software architect, I&#39;d like to hear what he&#39;s got to say about software, not about other shits.&#xA;There are multiple platforms lacking an API: nowadays I&#39;m forced to switch between Slack, Telegram, Whatsapp, Discord, Skype and whatever IM is going to be the most hip tomorrow. There are hundreds of forums - maybe the retain a topic, but they don&#39;t offer a consistent way of pulling content into a single application. Facebook and Twitter - while there used to be some apps that allowed integrating such feeds into a single app, they&#39;re long gone as far as I know. StackOverflow mitigates the issue with an acceptable RSS feed, even though it&#39;s not as customizable as I&#39;d like.&#xA;  Many websites just stopped offering RSS feeds at all, or stopped making them customizable, and just push their updates - ALL of their updates - on Facebook and Twitter. I used to be an avid Lifehacker reader, back when they had category-based RSS feeds, but at once... they just stopped providing such service.&#xA;Can&#39;t mark things as read. Have you ever noticed that? Inbox Zero once was a good practice for every workflow. Facebook and Twitter simply don&#39;t allow this functionality. You cannot mark a tweet or post as read, it could resurface at any time. And you cannot see which posts or tweets you&#39;ve not read yet - probably because the amount would be simple overwhelming. You need to waste your time on such websites.&#xA;&#xA;Where we want to go&#xA;&#xA;We want to reverse all this. We need:&#xA;&#xA;Topics. Google Plus created somethings similar to that with Collections (without RSS, of course); or we could just create a blog or username for each of our topics - I think most of us won&#39;t discuss about so many totally unrelated different fields. It&#39;s a change of mentality - we shouldn&#39;t write something just because we can. Unless we are celebrities, people, especially strangers, won&#39;t follow us just for the sake of it - we need actual, quality content. Smallchat is fine on FB or Twitter.&#xA;APIs. I&#39;m not saying we should get back to IRC or to NNTP. But we need a common API for Instant Messaging and forum-like software, so that people can use their favourite tools to organize their data sources. Installing tens of apps or visiting tens of websites every day is not an option.&#xA;&#xA;Incidentally, this is not a call for the open internet; I could not care less if there&#39;s a leading provider for content, as long as such content is accessible in a standard way.&#xA;&#xA;This is a call for a useful internet back again*.&#xA;&#xA;(Incidentally, this is not the usual focus of this blog, which is mostly technical. But I hope my audience will forgive me, this is something important, IMHO).&#xA;&#xA;EDIT:&#xA;an interesting discussion arose on Hacker News&#xA;&#xA;#Ollivander #rants #thoughts&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2017/07/jonathan-velasquez-160775--1-.jpg" alt=""></p>

<p>The internet is getting noisy. Too noisy. Having grown up in the nineties, with 56k dial-up, I sometimes struggle to understand how little I&#39;m accomplishing today with all the bandwidth I can leverage.</p>

<p>There were some key factors that made the old internet so productive, by the way, and many of those factors are just gone.</p>

<p>This is not just a rant. I have some proposals as well.</p>

<h5 id="discussing">Discussing</h5>

<p>Most discussions on the internet once happened in <a href="https://en.wikipedia.org/wiki/Usenet" rel="nofollow">Usenet</a> newsgroups and mailing lists. You could even access mailing lists via NNTP (the newsgroup protocol) using a great service like <a href="http://gmane.org/" rel="nofollow">gmane</a>.</p>

<p><img src="https://upload.wikimedia.org/wikipedia/commons/8/87/40tude_Dialog.png" alt="40tude Dialog"></p>

<p>40tude Dialog was my favourite newsreader at the time. I could subscribe to a lot of different newsgroups, <strong>without needing to sign up for each of them</strong>, and it was very easy to catch up with lots of traffic – just a matter of checking which messages had still the title in bold.</p>

<p><strong>There was a single application with a consistent interface</strong> to access the most disparate groups, and <strong>topics were widely enforced</strong> – probably the most common complaint from newsgroup users was that somebody was <em>off topic</em> – such discussions were quickly criticized and got little attention.</p>

<h5 id="chatting">Chatting</h5>

<p>There once was IRC, the Internet Relay Chat. <a href="http://www.mirc.com/get.html" rel="nofollow">mIRC</a> was the king of clients for a while.</p>

<p><img src="https://upload.wikimedia.org/wikipedia/commons/f/ff/Smuxi-0.7.2-linx-main-window.png" alt="an irc chat"></p>

<p>IRC, sometimes, was not great, lacking some advanced features for multimedia and emojis, but it was good. There were plenty of channels with specific topics, and you could join them in a second.</p>

<p>Then, other instant messengers picked up – ICQ, MSN, AOL, etc – but it wasn&#39;t a great issue; the protocols they used were mostly simple, and tools like <a href="https://www.trillian.im/" rel="nofollow">Trillian</a> and <a href="https://pidgin.im/" rel="nofollow">Pidgin/Adium</a> quickly reversed them and let the users just pick one client with multiple connections. Skype was a bit of a white fly because its original protocol was harder to reverse.</p>

<p>Again, <strong>there was an easy way to get a consistent interface in a single application</strong> to chat on IRC, Google Chat, Facebook Chat, generic XMPP, and so on.</p>

<h5 id="getting-notified-about-updates">Getting notified about updates</h5>

<p>Oh, this is was easy, wasn&#39;t it? <a href="https://en.wikipedia.org/wiki/RSS" rel="nofollow">RSS</a> and <a href="https://en.wikipedia.org/wiki/Atom_%28standard%29" rel="nofollow">Atom</a> just ruled the world. <a href="https://en.wikipedia.org/wiki/Bloglines" rel="nofollow">Bloglines</a> and the beloved <a href="https://www.google.com/reader/about/" rel="nofollow">Google Reader</a> were the first stop, every morning, for many developers.</p>

<p>Some blogger just putting too much shit on its blog? We remove him from our feed! No matter what!</p>

<p><img src="https://upload.wikimedia.org/wikipedia/commons/b/b7/BazQux_Reader_screenshot.png" alt="google reader"></p>

<p>We could simply pull <strong>our update sources</strong> within one single application, and check all the updates when we had the time to.</p>

<p>I can remember my workflow: scroll through the sources, open any interesting article in a new tab, mark all feeds as read; then go through the tabs until I finished. Job done. I could, at the same time, read a lot of interesting thing and discard a lot of uninteresting shit.</p>

<h5 id="what-happened">What happened</h5>

<p>If you take a look at was the state of things, you can notice that:</p>
<ul><li><strong>Focus is lost</strong>; once, everything went around <strong>topics</strong>. Channels had topics; newsgroups had topics; blog had topics (one great rule for a blog was to have a &#39;razor sharp&#39; focus). <strong>Nowadays we have <em>people</em> instead of topics</strong>. I have nothing against people, but maybe, if I follow a great software architect, I&#39;d like to hear what he&#39;s got to say about software, not about other shits.</li>
<li><strong>There are multiple platforms lacking an API</strong>: nowadays I&#39;m forced to switch between Slack, Telegram, Whatsapp, Discord, Skype and whatever IM is going to be the most hip tomorrow. There are hundreds of forums – maybe the retain a topic, but they don&#39;t offer a consistent way of pulling content into a single application. Facebook and Twitter – while there used to be some apps that allowed integrating such feeds into a single app, they&#39;re long gone as far as I know. StackOverflow mitigates the issue with an acceptable RSS feed, even though it&#39;s not as customizable as I&#39;d like.
Many websites just <strong>stopped offering RSS feeds at all</strong>, or stopped making them customizable, and just push their updates – ALL of their updates – on Facebook and Twitter. I used to be an avid <a href="http://lifehacker.com/" rel="nofollow">Lifehacker</a> reader, back when they had category-based RSS feeds, but at once... they just stopped providing such service.</li>
<li><strong>Can&#39;t mark things as read</strong>. Have you ever noticed that? <a href="http://www.lifehack.org/articles/lifehack/ultimate-way-inbox-zero.html" rel="nofollow">Inbox Zero</a> once was a good practice for every workflow. <strong>Facebook and Twitter simply don&#39;t allow this functionality</strong>. You cannot mark a tweet or post as read, it could resurface at any time. And you cannot see which posts or tweets you&#39;ve not read yet – probably because the amount would be simple overwhelming. <strong>You need to waste your time on such websites</strong>.</li></ul>

<h5 id="where-we-want-to-go">Where we want to go</h5>

<p>We want to reverse all this. We need:</p>
<ul><li><strong>Topics</strong>. Google Plus created somethings similar to that with Collections (without RSS, of course); or <strong>we could just create a blog or username for each of our topics</strong> – I think most of us won&#39;t discuss about so many totally unrelated different fields. It&#39;s a change of mentality – we shouldn&#39;t write something just because we can. Unless we are celebrities, people, especially strangers, won&#39;t follow us just for the sake of it – <strong>we need actual, quality content</strong>. Smallchat is fine on FB or Twitter.</li>
<li><strong>APIs</strong>. I&#39;m not saying we should get back to IRC or to NNTP. <strong>But we need a common API for Instant Messaging and forum-like software</strong>, so that people can use their favourite tools to organize their data sources. Installing tens of apps or visiting tens of websites every day is not an option.</li></ul>

<p>Incidentally, this is not a call for the open internet; I could not care less if there&#39;s a leading provider for content, <strong>as long as such content is accessible in a standard way</strong>.</p>

<p>This is a call for <strong>a useful internet back again</strong>.</p>

<p>(Incidentally, this is not the usual focus of this blog, which is mostly technical. But I hope my audience will forgive me, this is something important, IMHO).</p>

<p>EDIT:
an interesting discussion arose on <a href="https://news.ycombinator.com/item?id=14698545" rel="nofollow">Hacker News</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:rants" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">rants</span></a> <a href="/alan/tag:thoughts" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">thoughts</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/stopping-the-internet-of-noise</guid>
      <pubDate>Tue, 04 Jul 2017 21:48:00 +0000</pubDate>
    </item>
    <item>
      <title>Students that don&#39;t &#34;get&#34; computer science - bimodality as a teaching failure</title>
      <link>https://www.franzoni.eu/alan/students-that-dont-get-computer-science-bimodality</link>
      <description>&lt;![CDATA[&#xA;&#xA;There seems to be a common belief about computer science: people either get it, or don&#39;t get it.&#xA;&#xA;A recent paper by the University of Toronto, Evidence That Computer Science Grades Are Not Bimodal, dispels such myth.&#xA;&#xA;The paper even suggests some explanations for this myth: teaching failure. We don&#39;t know how to teach computer science; so, it seems that some people succeed at learning despite teachers&#39; inability to convey CS concepts.&#xA;&#xA;I think the idea is interesting and very appropriate; I have some opinions of mine:&#xA;&#xA;We should make people enjoy computer science! A lot of students get their grades and they can&#39;t write ten lines of code! But coding is actually the most interesting and amusing part of computer science, where you can actually do things and see how they work. Let&#39;s put more tinkering into CS courses!&#xA;We should put some effort at making CS cool. As long as people regard CS as boring, they don&#39;t take that journey.&#xA;&#xA;Otherwise, we risk an even further split between practitioners, those that learn by doing and that sometimes don&#39;t have a formal education, and the academics, that live in the walled garden of universities and have little to no contact with the industry.&#xA;&#xA;#Ollivander #teach #cs #industry #academy&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p><img src="https://files.franzoni.eu/import/alanslair/2017/03/gareth-harper-128701--1-.jpg" alt=""></p>

<p>There seems to be a common belief about computer science: people either <em>get it</em>, or <em>don&#39;t get it</em>.</p>

<p>A recent paper by the University of Toronto, <a href="http://www.cs.toronto.edu/dcs/documents/p113-patitsas.pdf" rel="nofollow">Evidence That Computer Science Grades Are Not Bimodal</a>, dispels such myth.</p>

<p>The paper even suggests some explanations for this myth: <strong>teaching failure</strong>. We don&#39;t know how to teach computer science; so, it seems that some people succeed at learning <em>despite teachers&#39; inability to convey CS concepts</em>.</p>

<p>I think the idea is interesting and very appropriate; I have some opinions of mine:</p>
<ul><li>We should make people <em>enjoy</em> computer science! A lot of students get their grades and <strong>they can&#39;t write ten lines of code</strong>! But coding is actually the most interesting and amusing part of computer science, where you can actually <strong>do things</strong> and see how they work. <strong>Let&#39;s put more tinkering into CS courses!</strong></li>
<li>We should put some effort at making CS <em>cool</em>. As long as people regard CS as boring, they don&#39;t take that journey.</li></ul>

<p>Otherwise, we risk an even further split between <em>practitioners</em>, those that <em>learn by doing</em> and that sometimes don&#39;t have a formal education, and the <em>academics</em>, that live in the walled garden of universities and have little to no contact with the industry.</p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:teach" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">teach</span></a> <a href="/alan/tag:cs" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">cs</span></a> <a href="/alan/tag:industry" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">industry</span></a> <a href="/alan/tag:academy" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">academy</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/students-that-dont-get-computer-science-bimodality</guid>
      <pubDate>Fri, 17 Mar 2017 08:29:00 +0000</pubDate>
    </item>
  </channel>
</rss>