<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>security &amp;mdash; alan</title>
    <link>https://www.franzoni.eu/alan/tag:security</link>
    <description>Alan&#39;s musings</description>
    <language>en</language>
    <pubDate>Tue, 17 Nov 2026 00:32:55 +0000</pubDate>
    <item>
      <title>Password requirements: myths and madness</title>
      <link>https://www.franzoni.eu/alan/password-requirements-myths-madness</link>
      <description>&lt;![CDATA[More than 11 years have passed since the venerable XKCD Password Strength strip:&#xA;&#xA;img src=&#34;https://imgs.xkcd.com/comics/passwordstrength.png&#34; &#xA;&#xA;And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:&#xA;&#xA;Single\Sign\On-1&#xA;&#xA;The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.&#xA;&#xA;I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.&#xA;&#xA;Please use a mix of uppercase and lowercase letters, digits, special chars in your password&#xA;&#xA;Usual explanation: for security reasons you must choose a strong password!&#xA;&#xA;My verdict: Misunderstanding&#xA;&#xA;The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same entropy as a shorter password from a larger charset.&#xA;&#xA;If you host a web application: Enforce a password&#39;s strength, not strange user-hostile policies. There&#39;re libraries for that (e.g. https://github.com/dropbox/zxcvbn) that work both in backend and frontend.&#xA;&#xA;You cannot use whitespace, accented letters, hyphens, \[...\] in your password&#xA;&#xA;Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like &#34;your password contains unsafe characters&#34;.&#xA;&#xA;My verdict: No real motivation beyond, possibly, incompetence&#xA;&#xA;This is a terrible one. You waste a lot of time at generating a &#34;strong password&#34; that complies with a website&#39;s policy, and then you discover that you could not use &#34;-&#34; in it. Or just whitespace for XKCD-style passwords.&#xA;&#xA;There&#39;s no technical reason for restricting any character from appearing in any password. The usual approach for handling a password is:&#xA;&#xA;take the password as a string, as the user typed it;&#xA;Prepare it (see links at the end of the paragraph);&#xA;encode with a specific, well-defined encoding (e.g. utf-8);&#xA;add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an &#34;hashed&#34; password and save that salt+hash in your storage of choice.&#xA;&#xA;What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.&#xA;&#xA;Caveats:&#xA;In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.&#xA;&#xA;If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.&#xA;&#xA;EDIT:&#xA;Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called &#34;string preparation&#34;. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:&#xA;https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html&#xA;https://www.rfc-editor.org/rfc/rfc8265&#xA;http://unicode.org/reports/tr15/\#Stabilized\Strings&#xA;&#xA;So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there could be good reasons to reject some input chars since there could be no stable encoding under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.&#xA;&#xA;You cannot enter a password with fewer than X chars&#xA;&#xA;EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET&#xA;&#xA;Usual explanation: security! I want your password to be veeeery strong!&#xA;&#xA;My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.&#xA;&#xA;This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. NIST standards even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).&#xA;&#xA;This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.&#xA;&#xA;But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the BMP, my password is probably very safe already, since there are (10\\6)\\4 possible passwords, many more than (127\\8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.&#xA;&#xA;If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.&#xA;&#xA;You cannot enter a password with more than X chars&#xA;&#xA;Usual explanation: whew. It&#39;s too large we cannot handle it.&#xA;&#xA;My verdict: Largely unmotivated unless X is quite large.&#xA;&#xA;Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.&#xA;&#xA;This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the &#34;enter your password&#34; field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.&#xA;&#xA;Caveats:&#xA;An higher length limit for passwords must should exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.&#xA;&#xA;EDIT:&#xA;A reader pointed out that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a a longer password doesn&#39;t mean it&#39;s necessarily more secure.&#xA;&#xA;If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user &#34;password too long&#34; if they&#39;re pasting into it.&#xA;&#xA;You cannot paste into this field&#xA;&#xA;Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!&#xA;&#xA;My verdict: Largely unmotivated right now, and dangerous.&#xA;&#xA;Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?&#xA;&#xA;If you host a web application: please allow pasting into password fields.&#xA;&#xA;You cannot show or copy this field&#xA;&#xA;Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.&#xA;&#xA;My verdict: Security through obscurity bad idea.&#xA;&#xA;Don&#39;t do this. While setting a password, there must be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is especially true if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.&#xA;&#xA;If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!&#xA;&#xA;EDIT:&#xA;This entry was edited after some comments. I know that&#39;s not what &#34;security through obscurity&#34; mean, it was a kind of joke about &#34;obscurating&#34; the password.&#xA;&#xA;Your password has expired&#xA;&#xA;Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.&#xA;&#xA;My verdict: Old idea&#xA;&#xA;This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation&#xA;&#xA;Too many failures - your account is locked.&#xA;&#xA;Usual explanation: we don&#39;t want your account to be bruteforced&#xA;&#xA;My verdict: Bad idea especially if &#34;too many&#34; is low, like 3 or 5, and my password is complex.&#xA;&#xA;The idea is that you &#34;protect&#34; a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.&#xA;&#xA;Account locking could be used in 2FA contexts (requires the other factor to provoke a DoS - that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts - they&#39;re too few to guess a password.&#xA;&#xA;Caveats:&#xA;Some compliance policies require locking user accounts.&#xA;&#xA;If you host a web application: consider using temporary locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).&#xA;&#xA;Final thoughts&#xA;&#xA;I really hope that passwordless login will make some steps forward in the next years (e.g. https://passkeys.dev/, https://www.yubico.com/authentication-standards/fido2/). In the meantime, I&#39;d love a &#34;password complexity api&#34; so that my browser (or password manager) can already generate the right password for a website.&#xA;&#xA;References&#xA;&#xA;https://www.auditboard.com/blog/nist-password-guidelines/&#xA;https://pages.nist.gov/800-63-3/sp800-63-3.html&#xA;&#xA;#Ollivander #security #passwords&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>More than 11 years have passed since the venerable <a href="https://xkcd.com/936/" rel="nofollow">XKCD Password Strength strip</a>:</p>

<p><img src="https://imgs.xkcd.com/comics/password_strength.png"></p>

<p>And yet this still happens nowadays on a variety of websites. Maybe it&#39;s even more frequent than 10 years ago:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/12/Single_Sign_On-1.jpg" alt="Single\_Sign\_On-1"></p>

<p>The hilarious part for this website is that the embedded strength checker properly recognizes a good password; but then an additional policy is bolted on, just in case users were too happy. In this case, whitespace is not accepted, and only some non-alphanumeric chars are considered special.</p>

<p>I tried taking a look at those idiosyncratic password requirements, and for most policies I couldn&#39;t find a direct relation to some recommendation or compliance. So, here I&#39;m trying to dig in what the underlying reasons are for this folly.</p>

<h3 id="please-use-a-mix-of-uppercase-and-lowercase-letters-digits-special-chars-in-your-password">Please use a mix of uppercase and lowercase letters, digits, special chars in your password</h3>

<p>Usual explanation: for security reasons you must choose a strong password!</p>

<p>My verdict: <strong>Misunderstanding</strong></p>

<p>The usual explanation is correct, but does not imply that a password should contain a lot of strange, hard-to-type and hard-to-remember characters. That&#39;s exact what xkcd passwords are about: a longer password with plain ascii lowercase letters can have the same <strong><a href="https://en.wikipedia.org/wiki/Entropy_%28information_theory%29" rel="nofollow">entropy</a></strong> as a shorter password from a larger charset.</p>

<p>If you host a web application: Enforce a password&#39;s <strong>strength</strong>, not strange user-hostile policies. There&#39;re libraries for that (e.g. <a href="https://github.com/dropbox/zxcvbn" rel="nofollow">https://github.com/dropbox/zxcvbn</a>) that work both in backend and frontend.</p>

<h3 id="you-cannot-use-whitespace-accented-letters-hyphens-in-your-password">You cannot use whitespace, accented letters, hyphens, [...] in your password</h3>

<p>Usual explanation: Very often there&#39;s no explanation for this, or it&#39;s something like “your password contains unsafe characters”.</p>

<p>My verdict: <strong>No real motivation beyond, possibly, incompetence</strong></p>

<p>This is a terrible one. You waste a lot of time at generating a “strong password” that complies with a website&#39;s policy, and then you discover that you could not use “–” in it. Or just whitespace for XKCD-style passwords.</p>

<p><strong>There&#39;s no technical reason for restricting any character from appearing in any password.</strong> The usual approach for handling a password is:</p>
<ul><li>take the password as a string, as the user typed it;</li>
<li>Prepare it (see links at the end of the paragraph);</li>
<li>encode with a specific, well-defined encoding (e.g. utf-8);</li>
<li>add a salt (probably your favourite library will require it in its input) and use a password-derivation algorithm (pbkdf2, Argon2, whatever) to get an “hashed” password and save that salt+hash in your storage of choice.</li></ul>

<p>What I suspect is that improper escaping and/or encoding techniques are being used in a website imposing that restriction. It&#39;s a red flag.</p>

<p>Caveats:
In some case there can be UX reasons for limiting the input charset, especially regarding special chars. Maybe you don&#39;t want people to use chars that cannot be found on some keyboards, because then, if people move across the world and use different computers, they&#39;re unable to enter their password. Or maybe the keyboard layout isn&#39;t yet set when the password is entered (example: full disk encryption password at boot) and you risk that a user is entering a password at install time, but what they write at boot is actually different. Maybe newline or other control chars are an hassle.</p>

<p>If you host a web application: make sure you&#39;re correctly processing the string coming from the user, rather than adding arbitrary requirements that force the users back-and-forth from a password manager to your application.</p>

<p>EDIT:
<em>Commenters on HN correctly noticed that I forgot a passage when handling passwords, the so called “string preparation”. It doesn&#39;t really matter too much with what we&#39;re dealing here, but here are some links:</em>
<a href="https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html" rel="nofollow">https://withblue.ink/2019/03/11/why-you-need-to-normalize-unicode-strings.html</a>
<a href="https://www.rfc-editor.org/rfc/rfc8265" rel="nofollow">https://www.rfc-editor.org/rfc/rfc8265</a>
<a href="http://unicode.org/reports/tr15/#Stabilized_Strings" rel="nofollow">http://unicode.org/reports/tr15/#Stabilized_Strings</a></p>

<p>So, it appears (Unicode is a terribly large and complicated standard, I may have misunderstood something) there <em>could</em> be good reasons to reject some input chars since there could be no <em>stable encoding</em> under any normalization process. But it doesn&#39;t seem the case in the scenarios I&#39;ve seen.</p>

<h3 id="you-cannot-enter-a-password-with-fewer-than-x-chars">You cannot enter a password with fewer than X chars</h3>

<p><em>EDIT: it seems that the message in this paragraph was widely misunderstood; I&#39;ve changed the wording. Dec 23 2022, 17:15 CET</em></p>

<p>Usual explanation: security! I want your password to be veeeery strong!</p>

<p>My verdict: Makes sense, but could be better, and if X is too large, it&#39;s just annoying.</p>

<p>This is the exact opposite of the first point. Sometimes you get a 16 or 20 minimum length for the password. Why? If I&#39;m autogenerating a password with all-possible-and-incredible-chars, eight should be enough. <a href="https://pages.nist.gov/800-63-3/sp800-63b.html" rel="nofollow">NIST standards</a> even allows just 6 chars in some situations (check 5.1.1.1 Memorized Secret Authenticators).</p>

<p>This is connected with password strength. A short alphanumeric password is usually easy to bruteforce with adequate money and gpu resources.</p>

<p>But, still, a website should aim at password strength, not at password length. If I use four unicode code points out of the more-than-one-million available, by picking those outside the <a href="https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane" rel="nofollow">BMP</a>, my password is probably very safe already, since there are (10**6)**4 possible passwords, many more than (127**8), which is more-or-less the number of possible ascii-printable passwords with 8 chars.</p>

<p>If you host a web application: probably your password strength checking library is not so good at understanding strange Unicode code points, so, you&#39;ll better go for a minimum length, but don&#39;t go wild: 8 or 10 with a password strength checker is the way to go. And: always clearly state the minimum length which is required for a password.</p>

<h3 id="you-cannot-enter-a-password-with-more-than-x-chars">You cannot enter a password with more than X chars</h3>

<p>Usual explanation: whew. It&#39;s too large we cannot handle it.</p>

<p>My verdict: <strong>Largely unmotivated</strong> unless X is quite large.</p>

<p>Once hashed, passwords take up the same amount of space. Restricting the maximum number of chars to a low number makes no sense and then forces people to use strange chars to get complexity right.</p>

<p>This is especially troublesome when the browser field is short, you can&#39;t see the password that you&#39;re pasting because it&#39;s masked, and you finally set a password that&#39;s not the one you expect. Then, when trying to login, the “enter your password” field isn&#39;t got the same limitations, and you cannot login because you&#39;re using the wrong (too-long) password.</p>

<p>Caveats:
An higher length limit for passwords <del>must</del> <strong>should</strong> exist to prevent DoS to an app, but should be rather high, like, not lower than 64 chars. Also, some algorithms that were (and maybe still are) quite common for password verification, like blowfish, don&#39;t properly work beyond 45 chars or so. Still, that&#39;s not the far-too-common 12 or 16 chars that can be found on plenty of websites.</p>

<p>EDIT:
A reader <a href="https://www.sjoerdlangkemper.nl/2021/07/02/long-password-denial-of-service/" rel="nofollow">pointed out</a> that properly-implemented password hashing algorithms aren&#39;t really subject to DoS. But, you may not be 100% sure about your implementation quality, and a <a href="https://crypto.stackexchange.com/questions/98505/argon2-allows-a-huge-key-length-in-input-but-does-it-really-provide-the-securit" rel="nofollow">a longer password doesn&#39;t mean it&#39;s necessarily more secure</a>.</p>

<p>If you host a web application: please allow long passwords at least 64 chars long; even 256 should pose no DoS risk whatsoever. Also, make sure that the HTML field itself is larger than the allowed password size, so you can tell the user “password too long” if they&#39;re pasting into it.</p>

<h3 id="you-cannot-paste-into-this-field">You cannot paste into this field</h3>

<p>Usual explanation: Security! I want you to actually write this and know what you&#39;re writing and write it twice in two boxes! Don&#39;t copy-paste from somewhere else!</p>

<p>My verdict: <strong>Largely unmotivated right now, and dangerous</strong>.</p>

<p>Somebody thinks that disallowing paste is a good idea so people actually need to write a password twice and they can&#39;t make typos. But this prevents using password managers, which are a Good Idea. This is a very old thing that some people thought being a good idea in the past, but I think it&#39;s not considered ideal since.... 2005?</p>

<p>If you host a web application: please allow pasting into password fields.</p>

<h3 id="you-cannot-show-or-copy-this-field">You cannot show or copy this field</h3>

<p>Usual explanation: security! your password is For Your Hands Only. We cannot even trust your eyes.</p>

<p>My verdict: <del>Security through obscurity</del> bad idea.</p>

<p>Don&#39;t do this. While setting a password, there <strong>must</strong> be an icon to show it and make sure I&#39;m not doing mistakes. It won&#39;t be saved in plaintext anywhere else! This is <strong>especially true</strong> if you&#39;re not allowing pasting a password. Then there&#39;s no way to make sure I&#39;ve written what I wanted to write! The default masking of the password is just designed to prevent shoulder surfing.</p>

<p>If you host a web application: please allow showing what I entered when setting or entering a password anyway. Otherwise I can keep doing mistakes!</p>

<p>EDIT:
<em>This entry was edited after some comments. I know that&#39;s not what “security through obscurity” mean, it was a kind of joke about “obscurating” the password.</em></p>

<h3 id="your-password-has-expired">Your password has expired</h3>

<p>Usual explanation: used to be widespread, and even required and recommended by widespread operating systems and applications.</p>

<p>My verdict: <strong>Old idea</strong></p>

<p>This once happened to be recommended, because a leak of password hashes was considered possible, and the password was in plaintext or was using weak hashes, and reuse was frequent. Nowadays, it&#39;s not recommended any more: unless there has been a known leak, if the password is properly salted and hashed, has a reasonable complexity, and hasn&#39;t been reused, there&#39;s no need fo rotation</p>

<h3 id="too-many-failures-your-account-is-locked">Too many failures – your account is locked.</h3>

<p>Usual explanation: we don&#39;t want your account to be bruteforced</p>

<p>My verdict: <strong>Bad idea</strong> especially if “too many” is low, like 3 or 5, and my password is complex.</p>

<p>The idea is that you “protect” a user preventing their account being bruteforced. But the CIA triad includes Availablity, and you&#39;re basically opening a giant door for a DoS this way; just know somebody&#39;s username and you can lock their account. Who knows what it takes to unlock it.</p>

<p>Account locking <em>could</em> be used in 2FA contexts (requires the other factor to provoke a DoS – that&#39;s why in some cases you enter a wrong password and you&#39;re still asked for your second factor, and only at the end you fail the authentication if either was wrong), but it&#39;s still usually pointless to lock an account after a few attempts – they&#39;re too few to guess a password.</p>

<p>Caveats:
Some compliance policies require locking user accounts.</p>

<p>If you host a web application: consider using <em>temporary</em> locking with exponential, kind-of-stochastic backoff (like 10s wait the first time before being able to enter the password again, 22s the second...) after 10-20 attempts instead. If you can, it&#39;s probably useful to limit this measure to specific IPs that are attempting the bruteforce (but can be hard if it&#39;s a distributed attack).</p>

<h2 id="final-thoughts">Final thoughts</h2>

<p>I really hope that passwordless login will make some steps forward in the next years (e.g. <a href="https://passkeys.dev/" rel="nofollow">https://passkeys.dev/</a>, <a href="https://www.yubico.com/authentication-standards/fido2/" rel="nofollow">https://www.yubico.com/authentication-standards/fido2/</a>). In the meantime, I&#39;d love a “password complexity api” so that my browser (or password manager) can already generate the right password for a website.</p>

<h2 id="references">References</h2>

<p><a href="https://www.auditboard.com/blog/nist-password-guidelines/" rel="nofollow">https://www.auditboard.com/blog/nist-password-guidelines/</a>
<a href="https://pages.nist.gov/800-63-3/sp800-63-3.html" rel="nofollow">https://pages.nist.gov/800-63-3/sp800-63-3.html</a></p>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a> <a href="/alan/tag:passwords" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">passwords</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/password-requirements-myths-madness</guid>
      <pubDate>Thu, 22 Dec 2022 20:41:14 +0000</pubDate>
    </item>
    <item>
      <title>Ransomware-resistant backups with duplicity and AWS S3</title>
      <link>https://www.franzoni.eu/alan/ransomware-resistant-backups</link>
      <description>&lt;![CDATA[Ransomware and backups&#xA;&#xA;Article updated: February 5th, 2022&#xA;&#xA;Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.&#xA;&#xA;Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, cryptolocker-like malwares are spreading. You&#39;re not safe just because why should anybody hack into my little server: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.&#xA;&#xA;So: people should revise their own threat models. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a failure scenario (hardware failure, accidental deletion), not for an attack scenario. But even half-serious cybercriminals, once they get access to your server, will delete all backup sets they can access, so that their encrypt-your-data-and-ask-for-money threat actually works.&#xA;&#xA;Once upon a time&#xA;&#xA;You had backups on a tape library. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated - getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.&#xA;&#xA;Then approaches like bacula were common - the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it &#34;wipe everything&#34; so again, barring any bacula server exploit, bacula server data held backups safe.&#xA;&#xA;More recently, cloud approaches like rsync.net leverage ZFS snapshots to make remote data immutable - they can be removed only according to your retention schedule (details). BorgBase works in a similar way and has a good writeup about backup strategies. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider restore costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data - at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.&#xA;&#xA;But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: restic offers rest-server, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.&#xA;&#xA;The cloud to the rescue&#xA;&#xA;So, here we&#39;ll see how to use the good old duplicity backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.&#xA;&#xA;For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your workstation and your server, and you&#39;d like to have your server backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.&#xA;&#xA;So, make yourself a favour and use two separate machines.&#xA;&#xA;Create your S3 bucket and credentials&#xA;&#xA;Those steps must be taken from your workstation&#xA;&#xA;These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.&#xA;&#xA;Create the bucket&#xA;&#xA;Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it sample-duplicity-backup for the sake of this article) in your preferred region (I will use eu-central-1) ; and use these settings:&#xA;&#xA;Disable ACLs&#xA;&#xA;S3\bucket-1-1&#xA;&#xA;Block public access&#xA;&#xA;S3\bucket-2-1&#xA;&#xA;Enable bucket versioning&#xA;&#xA;S3\bucket-3-1&#xA;&#xA;Enable object lock&#xA;&#xA;S3\bucket-4-1&#xA;&#xA;Configure object lock&#xA;&#xA;Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.&#xA;&#xA;sample-duplicity-backup\objectlock&#xA;&#xA;Create a IAM user&#xA;&#xA;Now you need to create a suitable IAM user to be used with duplicity. Let&#39;s call this sample-duplicity-user; it should only have programmatic access, and you should attach only this policy to such user:&#xA;&#xA;{&#xA;    &#34;Version&#34;: &#34;2012-10-17&#34;,&#xA;    &#34;Statement&#34;: [&#xA;        {&#xA;            &#34;Effect&#34;: &#34;Allow&#34;,&#xA;            &#34;Action&#34;: [&#xA;                &#34;s3:PutObject&#34;,&#xA;                &#34;s3:GetObject&#34;,&#xA;                &#34;s3:ListBucketMultipartUploads&#34;,&#xA;                &#34;s3:AbortMultipartUpload&#34;,&#xA;                &#34;s3:ListBucket&#34;,&#xA;                &#34;s3:ListMultipartUploadParts&#34;&#xA;            ],&#xA;            &#34;Resource&#34;: [&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,&#xA;                &#34;arn:aws:s3:::sample-duplicity-backup/&#34;&#xA;            ]&#xA;        }&#xA;    ]&#xA;}&#xA;&#xA;Download the access key ID and the secret access key for such user.&#xA;&#xA;Install duplicity&#xA;&#xA;The following steps must be performed on the server&#xA;&#xA;Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported - check the homepage for further info. Also make sure the boto3 library and gnupg are available on your system.&#xA;&#xA;As an example, for recent Ubuntu versions (tested on 20.04):&#xA;&#xA;Add the stable duplicity PPA&#xA;apt -y install duplicity python3-boto3 gnupg&#xA;&#xA;Create a backup script&#xA;&#xA;Put some data in /root/data for the purpose of this test.&#xA;&#xA;Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography - support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:&#xA;&#xA;!/bin/bash -e&#xA;export PASSPHRASE=&#34;XYZXYZXYZ&#34;&#xA;export AWSACCESSKEYID=&#34;XXXXXXX&#34;&#xA;export AWSSECRETACCESSKEY=&#34;YYYYYYYYY&#34;&#xA;/usr/bin/duplicity \&#xA;--s3-european-buckets \&#xA;  --s3-use-new-style  --asynchronous-upload -v 4 \&#xA;   incr --full-if-older-than 30D \&#xA;  /root/data \&#xA;  &#34;boto3+s3://sample-duplicity-backup/data&#34;&#xA;&#xA;This command will backup the contents of /root/data to your s3 bucket, using a&#xA;data prefix.&#xA;&#xA;One detail to note: the --full-if-older 30D means: create an incremental backup&#xA;since the previous one, or create a new full backup if more than 30 days passed since our last full backup. It is essential that the number of days that we set here is smaller than the object lock days.&#xA;&#xA;So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the data directory:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket&#xA;&#xA;If you like, you can run the script again. You&#39;ll notice that duplicity only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.&#xA;&#xA;Attack scenario&#xA;&#xA;Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for sample-duplicity-user, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.&#xA;&#xA;For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same sample-duplicity-user credentials and pretend you&#39;re the badguy:&#xA;&#xA;aws configure --profile badguy&#xA;&#xA;Now, try acting as the Bad Guy:&#xA;&#xA;$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/&#xA;2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg&#xA;2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg&#xA;2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg&#xA;&#xA;He can read the backup. This is expected. But can he delete the backup?&#xA;&#xA;$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied&#xA;&#xA;The attacker cannot delete files; our object lock policy prevents that!&#xA;&#xA;But, there&#39;s still one thing that the attacker could do: overwrite existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between &#34;add an object&#34; or &#34;overwrite an existing object&#34; (and it would be quite difficult and slow to implement in a distributed storage system):&#xA;&#xA;$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Ouch! This (apparently) succeeded:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-update&#xA;&#xA;But the reality is that the original data is still there: the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the &#34;Versions&#34; tab for our modified file:&#xA;&#xA;sample-duplicity-backup\-\S3\bucket-versions&#xA;&#xA;Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the s3:ListBucketVersions permission (check the API call as well) - we don&#39;t even have it for our backup IAM user.&#xA;&#xA;Retrieving the original files&#xA;&#xA;A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:&#xA;&#xA;$ aws s3api list-object-versions --bucket sample-duplicity-backup&#xA;{&#xA;    &#34;Versions&#34;: [&#xA;        ...&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,&#xA;            &#34;Size&#34;: 50,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;wZOOamRomLYpZO5Gc.Vynrl0cP6bWq&#34;,&#xA;            &#34;IsLatest&#34;: true,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        {&#xA;            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;            &#34;Size&#34;: 267,&#xA;            &#34;StorageClass&#34;: &#34;STANDARD&#34;,&#xA;            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,&#xA;            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;            &#34;IsLatest&#34;: false,&#xA;            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;            &#34;Owner&#34;: {&#xA;                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;&#xA;            }&#xA;        },&#xA;        ...&#xA;    ]&#xA;}&#xA;&#xA;Then retrieve this file in its older version:&#xA;&#xA;$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;&#xA;{&#xA;    &#34;AcceptRanges&#34;: &#34;bytes&#34;,&#xA;    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,&#xA;    &#34;ContentLength&#34;: 267,&#xA;    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,&#xA;    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,&#xA;    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,&#xA;    &#34;Metadata&#34;: {}&#xA;}&#xA;$ ls .gpg&#xA;duplicity-full.20220125T204132Z.manifest.gpg&#xA;&#xA;Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.&#xA;&#xA;Wrapping up&#xA;&#xA;I hope this article can be useful from an implementation standpoint, but the most important takeaway is: always think about your threat model every time you take a decision. Reason in the terms of what an attacker could do when entering your system; security is never binary.&#xA;&#xA;And, please: don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set, like -rwx------ 1 root root 365 Dec  9 21:52 duplicity_run. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!&#xA;&#xA;An exercise&#xA;&#xA;An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.&#xA;&#xA;To my readers: since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.&#xA;&#xA;Updates &amp; footnotes&#xA;&#xA;I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the HN crowd as usual for commenting.&#xA;&#xA;Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.&#xA;&#xA;So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):&#xA;&#xA;You should remember to periodically verify your backups, usually by restoring them - this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, make sure you verify backups before deleting old data.&#xA;Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even replicate buckets between accounts&#xA;Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend setting a proper budget alert for your AWS account and maybe an alert for excessive S3 data usage&#xA;Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job - so you get errors if it fails - and to periodically verify them.&#xA;If you&#39;ve got a lot of data and not a lot of bandwidth - hence the initial backup may take you a lot time - make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.&#xA;&#xA;#Ollivander #backups #security&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<h2 id="ransomware-and-backups">Ransomware and backups</h2>

<p><em>Article updated: February 5th, 2022</em></p>

<p>Ransomware is changing the security scenario. Once upon a time, attackers who entered your systems could pull some data to sell it somewhere; they could deface your website for kudos; but, unless they had some compelling reason (disgruntled former employees), they probably wouldn&#39;t destroy your data and your backups.</p>

<p>Now they would, though. Cryptocurrencies make it very easy to ask for a ransom and never get caught. Hence, <a href="https://en.wikipedia.org/wiki/CryptoLocker" rel="nofollow">cryptolocker-like</a> malwares are spreading. You&#39;re not safe just because <em>why should anybody hack into my little server</em>: if a widespread exploit is found, cybercriminals will perform a mass scan on the whole internet, exploit and implant a malware on every single vulnerable system, then ask some amount of cryptocurrency to get your data back.</p>

<p>So: <strong>people should revise their own threat models</strong>. And your backup strategy is likely one of the first things you should revise. It&#39;s quite likely it was designed for a <strong>failure scenario</strong> (hardware failure, accidental deletion), not for an <strong>attack scenario</strong>. But even half-serious cybercriminals, once they get access to your server, <strong>will delete all backup sets they can access</strong>, so that their encrypt-your-data-and-ask-for-money threat actually works.</p>

<h2 id="once-upon-a-time">Once upon a time</h2>

<p>You had backups on a <a href="https://en.wikipedia.org/wiki/Tape_drive" rel="nofollow">tape library</a>. A tape drive is usually append-only, and the backup is handled through a dedicated system so it was hard (if possible at all) and clunky to wipe an older backup from a compromised system. And tapes would get physically rotated – getting hold of an old tape was not feasible for a remote attacker, so at least you got an older backup which was safe.</p>

<p>Then approaches like <a href="https://www.bacula.org/" rel="nofollow">bacula</a> were common – the bacula server (where the backups are saved) would contact the bacula clients (systems to be backed up) and asked for data to be saved. The bacula server deleted old data using its own schedule, there was no way for the bacula client to tell it “wipe everything” so again, barring any bacula server exploit, bacula server data held backups safe.</p>

<p>More recently, cloud approaches like <a href="https://www.rsync.net/" rel="nofollow">rsync.net</a> leverage ZFS snapshots to make remote data immutable – they can be removed only according to your retention schedule (<a href="https://www.rsync.net/resources/howto/snapshots.html" rel="nofollow">details</a>). <a href="https://www.borgbase.com/" rel="nofollow">BorgBase</a> works in a similar way and has a <a href="https://docs.borgbase.com/strategy/" rel="nofollow">good writeup about backup strategies</a>. Such providers usually work well since it&#39;s their very job, and should be considered when choosing a backup strategy. I won&#39;t recommend any of them since it&#39;s outside the scope of this post (and I haven&#39;t tested all of them!); just rememeber to consider <em>restore</em> costs and speed as well as backup costs for those providers since, just like raw cold-storage providers (e.g. AWS Glacier, Backblaze B2, et cetera) they sometimes tend to ask for a plus whenever you need to download your data – at the moment of this writing BorgBase doesn&#39;t seem to charge for download, though, but they reserve the right to terminate your account if they see excessive usage, so the topic is definitely of concern for backup providers.</p>

<p>But, many modern backup systems don&#39;t take the ransomware scenario into account, since the backup software needs full access to a local or remote filesystem. Some actually try: <a href="https://restic.net/" rel="nofollow">restic</a> offers <a href="https://github.com/restic/rest-server" rel="nofollow">rest-server</a>, which has an append-only option to prevent malicious deletion of the server&#39;s content. But you need an additional system just to host the server.</p>

<h2 id="the-cloud-to-the-rescue">The cloud to the rescue</h2>

<p>So, here we&#39;ll see how to use the good old <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">duplicity</a> backup software to perform backups to AWS S3 in a ransomware-resistant way. You can use any cloud storage that supports properly fine-grained permissions (see later) and is supported by duplicity.</p>

<p>For the sake of this article, I suppose you&#39;ve got access to two distinct machines: your <strong>workstation</strong> and your <strong>server</strong>, and you&#39;d like to have your <strong>server</strong> backed up. You could actually employ a single machine to perform all the steps, but you&#39;ll need to make sure that your master access to AWS S3 is never compromised, otherwise say goodbye to your ransomware resistance.</p>

<p>So, make yourself a favour and use two separate machines.</p>

<h2 id="create-your-s3-bucket-and-credentials">Create your S3 bucket and credentials</h2>

<p><em>Those steps must be taken from your workstation</em></p>

<p>These examples don&#39;t leverage the aws cli to prevent accidents with API key leakeges; if you use the AWS S3 Console, and you&#39;ve properly configured 2FA for your account, it&#39;s less likely you can be attacked that way.</p>

<h3 id="create-the-bucket">Create the bucket</h3>

<p>Enter the AWS Console, choose the S3 service, create a bucket (we&#39;ll call it <em>sample-duplicity-backup</em> for the sake of this article) in your preferred region (I will use <em>eu-central-1</em>) ; and use these settings:</p>

<h4 id="disable-acls">Disable ACLs</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-1-1.jpg" alt="S3\_bucket-1-1"></p>

<h4 id="block-public-access">Block public access</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-2-1.jpg" alt="S3\_bucket-2-1"></p>

<h4 id="enable-bucket-versioning">Enable bucket versioning</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-3-1.jpg" alt="S3\_bucket-3-1"></p>

<h4 id="enable-object-lock">Enable object lock</h4>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/S3_bucket-4-1.jpg" alt="S3\_bucket-4-1"></p>

<h4 id="configure-object-lock">Configure object lock</h4>

<p>Then, select the bucket you just created, go to Properties, choose Object Lock, and enable Default Retention. For the purpose we have, Governance mode is OK; choose a 40 days default retention period, and save your changes.</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_objectlock.jpg" alt="sample-duplicity-backup\_objectlock"></p>

<h3 id="create-a-iam-user">Create a IAM user</h3>

<p>Now you need to <a href="https://console.aws.amazon.com/iamv2/home#/users" rel="nofollow">create a suitable IAM user</a> to be used with duplicity. Let&#39;s call this <code>sample-duplicity-user</code>; it should only have programmatic access, and you should attach <strong>only</strong> this policy to such user:</p>

<pre><code>{
    &#34;Version&#34;: &#34;2012-10-17&#34;,
    &#34;Statement&#34;: [
        {
            &#34;Effect&#34;: &#34;Allow&#34;,
            &#34;Action&#34;: [
                &#34;s3:PutObject&#34;,
                &#34;s3:GetObject&#34;,
                &#34;s3:ListBucketMultipartUploads&#34;,
                &#34;s3:AbortMultipartUpload&#34;,
                &#34;s3:ListBucket&#34;,
                &#34;s3:ListMultipartUploadParts&#34;
            ],
            &#34;Resource&#34;: [
                &#34;arn:aws:s3:::sample-duplicity-backup&#34;,
                &#34;arn:aws:s3:::sample-duplicity-backup/*&#34;
            ]
        }
    ]
}
</code></pre>

<p>Download the access key ID and the secret access key for such user.</p>

<h2 id="install-duplicity">Install duplicity</h2>

<p><em>The following steps must be performed on the server</em></p>

<p>Duplicity may be available in your distribution, but I suggest you pick a recent version (0.8.x) in order to make sure the s3 remote backend is properly supported – check <a href="https://duplicity.gitlab.io/duplicity-web/" rel="nofollow">the homepage</a> for further info. Also make sure the <code>boto3</code> library and <code>gnupg</code> are available on your system.</p>

<p>As an example, for recent Ubuntu versions (tested on 20.04):</p>
<ul><li>Add the <a href="https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git" rel="nofollow">stable duplicity PPA</a></li>
<li><code>apt -y install duplicity python3-boto3 gnupg</code></li></ul>

<h2 id="create-a-backup-script">Create a backup script</h2>

<p>Put some data in <code>/root/data</code> for the purpose of this test.</p>

<p>Then, add the following snippet to a script; set the passphrase as you wish (duplicity encrypts the backups using symmetric cryptography – support for public key crypto is available but I haven&#39;t had great success with it so far), and fill in the AWS credentials you got from the previous step:</p>

<pre><code>#!/bin/bash -e
export PASSPHRASE=&#34;XYZXYZXYZ&#34;
export AWS_ACCESS_KEY_ID=&#34;XXXXXXX&#34;
export AWS_SECRET_ACCESS_KEY=&#34;YYYYYYYYY&#34;
/usr/bin/duplicity \
--s3-european-buckets \
  --s3-use-new-style  --asynchronous-upload -v 4 \
   incr --full-if-older-than 30D \
  /root/data \
  &#34;boto3+s3://sample-duplicity-backup/data&#34;
</code></pre>

<p>This command will backup the contents of <code>/root/data</code> to your s3 bucket, using a
<code>data</code> prefix.</p>

<p>One detail to note: the <code>--full-if-older 30D</code> means: <em>create an incremental backup
since the previous one, or create a new full backup if more than 30 days passed since our last full backup</em>. It is <strong>essential</strong> that the number of days that we set here is smaller than the object lock days.</p>

<p>So, great! Now, from your workstation, you can check that your S3 bucket contains some data; check the Objects tab of your bucket, and enter the <em>data</em> directory:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket"></p>

<p>If you like, you can run the script again. You&#39;ll notice that duplicity <em>only adds new data to the s3 bucket, it never removes or changes existing files once they have been uploaded.</em></p>

<h2 id="attack-scenario">Attack scenario</h2>

<p>Now a Bad Guy exploits a vulnerability and takes total, root control of your server. He gets all your data; you can&#39;t help that, it&#39;s compromised. He even gets access to the credentials for <code>sample-duplicity-user</code>, and, being Bad, after deleting your data directory he tries to delete the content of your backup bucket.</p>

<p>For the sake of simplicity, we&#39;ll configure an aws cli account (do it either on your workstation or on your server, it doesn&#39;t really make a difference. Remember to delete such credentials afterwards) using the same <code>sample-duplicity-user</code> credentials and pretend you&#39;re the badguy:</p>

<p><code>aws configure --profile badguy</code></p>

<p>Now, try acting as the Bad Guy:</p>

<pre><code>$ aws --profile badguy s3 ls s3://sample-duplicity-backup/data/
2022-01-25 21:41:36       3228 duplicity-full-signatures.20220125T204132Z.sigtar.gpg
2022-01-25 21:41:37        267 duplicity-full.20220125T204132Z.manifest.gpg
2022-01-25 21:41:36      27298 duplicity-full.20220125T204132Z.vol1.difftar.gpg
</code></pre>

<p>He can read the backup. This is expected. But can he <strong>delete</strong> the backup?</p>

<pre><code>$ aws --profile badguy s3 rm s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
delete failed: s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied
</code></pre>

<p>The attacker cannot delete files; our object lock policy prevents that!</p>

<p>But, there&#39;s still one thing that the attacker could do: <strong>overwrite</strong> existing files. In fact, AWS S3 Put actions (and permissions) don&#39;t make a distinction between “add an object” or “overwrite an existing object” (and it would be quite difficult and slow to implement in a distributed storage system):</p>

<pre><code>$ aws --profile badguy s3 cp temp.txt s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
upload: ./temp.txt to s3://sample-duplicity-backup/data/duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Ouch! This (apparently) succeeded:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-update.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-update"></p>

<p>But the reality is that <strong>the original data is still there:</strong> the attacker has uploaded a new file with same name as ours, but we can still retrieve the first version; Object Lock and Bucket Versioning work together. Check the “Versions” tab for our modified file:</p>

<p><img src="https://files.franzoni.eu/import/alanslair/2022/01/sample-duplicity-backup_-_S3_bucket-versions.jpg" alt="sample-duplicity-backup\_-\_S3\_bucket-versions"></p>

<p>Choose the older version and download it: it&#39;s the original file! So, if you get hit by a ransomware, you can still retrieve all your original files. Accessing those in a programmatic way requires the <em>s3:ListBucketVersions</em> permission (check the <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html" rel="nofollow">API call as well</a>) – we don&#39;t even have it for our backup IAM user.</p>

<h2 id="retrieving-the-original-files">Retrieving the original files</h2>

<p>A full demonstration of how to retrieve the original files is trivial and beyond the scope of this article; but I&#39;ll leave some breadcrumbs here. With an user with the ListBucketVersions permissions, call this command:</p>

<pre><code>$ aws s3api list-object-versions --bucket sample-duplicity-backup
{
    &#34;Versions&#34;: [
        ...
        {
            &#34;ETag&#34;: &#34;\&#34;500e2a10137f805dba21f4bb7bf3678a\&#34;&#34;,
            &#34;Size&#34;: 50,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;w_ZOOamRomLYpZO5Gc.Vy_nrl0cP6bWq&#34;,
            &#34;IsLatest&#34;: true,
            &#34;LastModified&#34;: &#34;2022-01-25T20:50:15+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        {
            &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
            &#34;Size&#34;: 267,
            &#34;StorageClass&#34;: &#34;STANDARD&#34;,
            &#34;Key&#34;: &#34;data/duplicity-full.20220125T204132Z.manifest.gpg&#34;,
            &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
            &#34;IsLatest&#34;: false,
            &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
            &#34;Owner&#34;: {
                &#34;ID&#34;: &#34;132149dd72a1af36909b73ae719ccba0096cd23aa62158308ef4b9619f3b63ed&#34;
            }
        },
        ...
    ]
}
</code></pre>

<p>Then retrieve this file in its older version:</p>

<pre><code>$ aws s3api get-object --bucket sample-duplicity-backup --key data/duplicity-full.20220125T204132Z.manifest.gpg duplicity-full.20220125T204132Z.manifest.gpg  --version-id &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;
{
    &#34;AcceptRanges&#34;: &#34;bytes&#34;,
    &#34;LastModified&#34;: &#34;2022-01-25T20:41:37+00:00&#34;,
    &#34;ContentLength&#34;: 267,
    &#34;ETag&#34;: &#34;\&#34;2dd325e03ac11eb6edaf0e9a7b177064\&#34;&#34;,
    &#34;VersionId&#34;: &#34;ORkLIhukxVZKTeQruI9hPDWXUzAJoYgb&#34;,
    &#34;ContentType&#34;: &#34;binary/octet-stream&#34;,
    &#34;Metadata&#34;: {}
}
$ ls *.gpg
duplicity-full.20220125T204132Z.manifest.gpg
</code></pre>

<p>Apply for all files where you actually need it. Actually, it&#39;s quite likely that the attacker will have noticed the bucket is versioned and object locked, so he won&#39;t waste time at overwriting all files.</p>

<h2 id="wrapping-up">Wrapping up</h2>

<p>I hope this article can be useful from an implementation standpoint, but the most important takeaway is: <strong>always think about your threat model every time you take a decision</strong>. Reason in the terms of what an attacker could do when entering your system; <strong>security is never binary</strong>.</p>

<p>And, please: <strong>don&#39;t run your applications as root, but run your backup script as root, and make sure its ownership and permissions are properly set</strong>, like <code>-rwx------ 1 root root 365 Dec  9 21:52 duplicity_run</code>. It&#39;s great to have a ransomware resistant backup, but you shouldn&#39;t make things too easy for an attacker!</p>

<h3 id="an-exercise">An exercise</h3>

<p>An attacker enters your server, but he realizes you&#39;re using object lock and bucket versioning. But he really wants to find a way to get some money from you, since he knows you&#39;re full of XMR in your wallets. Hypothesize how could he proceed, then go on and think about what you could do to prevent damages to your system and to your backups.</p>

<p><em>To my readers:</em> since this is sort of security-related and a topic with a potentially high impact, please let me know if got anything wrong. Contact me by commenting or directly via e-mail.</p>

<h3 id="updates-footnotes">Updates &amp; footnotes</h3>

<p>I received a decent amount of feedback about this article. I think this means that the topic is interesting at least interesting. Thanks to the <a href="https://news.ycombinator.com/item?id=30105538" rel="nofollow">HN crowd</a> as usual for commenting.</p>

<p>Even though this should be a basic functionality, it seems incredibly hard to get done right unless you have a dedicated SRE team.</p>

<p>So, some additional, interesting points (I meant for some of those were covered in my &#39;exercise&#39; paragraph, but I&#39;m now jotting them down):</p>
<ul><li>You should remember to periodically verify your backups, usually by restoring them – this is true in general for backups, not just for ransomware resistant ones. For the ransomware situation, you must make sure to verify your backups more often than their expiration period on S3, otherwise an attacker could a) tamper with your backup script making it ineffective and b) wait for governance lock time to expire, and hope old data gets deleted by some schedule (they can&#39;t actually delete data because they don&#39;t have such right). So, <strong>make sure you verify backups before deleting old data</strong>.</li>
<li>Use a proper account for backups; the risk, sometimes, is that backups get deleted/attacked because of accidental exposure from the backup account. I recommend using a separate AWS account with highly restricted IAM users for backup scenario; you can even <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html" rel="nofollow">replicate buckets between accounts</a></li>
<li>Somebody stealing your S3 credentials could just use them for their own purpose and save arbitrary data there, with you incurring the costs. But you don&#39;t want an hard limit before s3 stops working (I&#39;m not even sure it&#39;s possible), otherwise you&#39;ll enable the attack to escalate into a ransomware. So I recommend <a href="https://aws.amazon.com/aws-cost-management/aws-budgets/" rel="nofollow">setting a proper budget alert for your AWS account</a> and maybe an <a href="https://stackoverflow.com/questions/51541751/alarm-when-object-size-in-s3-bucket-exceeds-threshold" rel="nofollow">alert for excessive S3 data usage</a></li>
<li>Some people report that duplicity, when used with a great amount of data, can start using an excessive amount of memory and ultimately gets killed because the server runs out of ram. I can&#39;t confirm or deny this either, just make sure, again, to monitor your backup job – so you get errors if it fails – and to periodically verify them.</li>
<li>If you&#39;ve got a lot of data and not a lot of bandwidth – hence the initial backup may take you a lot time – make sure you pick a good, reliable provider that won&#39;t be gone in one or two years. That&#39;s the reason I used S3: I suppose it will be still there in 10 or 20 years.</li></ul>

<p><a href="/alan/tag:Ollivander" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">Ollivander</span></a> <a href="/alan/tag:backups" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">backups</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/ransomware-resistant-backups</guid>
      <pubDate>Thu, 27 Jan 2022 19:50:24 +0000</pubDate>
    </item>
    <item>
      <title>Don&#39;t give a shell to system users - use su</title>
      <link>https://www.franzoni.eu/alan/dont-give-a-shell-to-system-users-use-su</link>
      <description>&lt;![CDATA[I seem very ancientwisdom-oriented these days.&#xA;&#xA;Sometimes you&#39;ve got a system user on your linux server that&#39;s used to run a specific service; it&#39;s a good practice to employ those users to achieve privilege separation, since it limits the impact of many security breaches. It&#39;s security 101, really.&#xA;&#xA;If you&#39;ve got a system user, most probably its shell would be /bin/false or /bin/nologin. Again, this is a good practice, since it would be harder for an attacker to gain shell access to your machine through that user.&#xA;&#xA;But I find that many people just change such shell, because sometimes they need to customize the environment for a user, and if you try doing something like&#xA;&#xA;su - systemuser&#xA;&#xA;you&#39;ll find yourself just back to your original shell, because su tries to invoke the shell which is set for the target user, and sudo doesn&#39;t generally help here, because its --shell option doesn&#39;t start a proper login shell for that user (most of the environment is that of the calling user).&#xA;&#xA;Very well, gentlemen: this is a wrongdoing and a potential security hole. You should never enable a shell for your system users. Instead, you should leverage an option of su:&#xA;&#xA;su - systemuser --shell /bin/bash&#xA;&#xA;That lets you choose your favourite shell on a per-launch basis.&#xA;&#xA;#MostlyUnixish #linux #ancientwisdom #su #security&#xA;]]&gt;</description>
      <content:encoded><![CDATA[<p>I seem very ancientwisdom-oriented these days.</p>

<p>Sometimes you&#39;ve got a system user on your linux server that&#39;s used to run a specific service; it&#39;s a good practice to employ those users to achieve <a href="http://en.wikipedia.org/wiki/Privilege_separation" rel="nofollow">privilege separation</a>, since it limits the impact of many security breaches. It&#39;s security 101, really.</p>

<p>If you&#39;ve got a system user, most probably its shell would be <code>/bin/false</code> or <code>/bin/nologin</code>. Again, this is a good practice, since it would be harder for an attacker to gain shell access to your machine through that user.</p>

<p>But I find that many people just <strong>change</strong> such shell, because sometimes they need to customize the environment for a user, and if you try doing something like</p>

<pre><code>su - systemuser
</code></pre>

<p>you&#39;ll find yourself just back to your original shell, because <strong>su</strong> tries to invoke the shell which is set for the target user, and <code>sudo</code> doesn&#39;t generally help here, because its <code>--shell</code> option doesn&#39;t start a proper login shell for that user (most of the environment is that of the calling user).</p>

<p>Very well, gentlemen: <strong>this is a wrongdoing and a potential security hole</strong>. You should <strong>never</strong> enable a shell for your system users. Instead, you should leverage an option of <strong>su</strong>:</p>

<pre><code>su - systemuser --shell /bin/bash
</code></pre>

<p>That lets you choose your favourite shell on a per-launch basis.</p>

<p><a href="/alan/tag:MostlyUnixish" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">MostlyUnixish</span></a> <a href="/alan/tag:linux" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">linux</span></a> <a href="/alan/tag:ancientwisdom" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">ancientwisdom</span></a> <a href="/alan/tag:su" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">su</span></a> <a href="/alan/tag:security" class="hashtag" rel="nofollow"><span>#</span><span class="p-category">security</span></a></p>
]]></content:encoded>
      <guid>https://www.franzoni.eu/alan/dont-give-a-shell-to-system-users-use-su</guid>
      <pubDate>Thu, 11 Dec 2014 22:25:00 +0000</pubDate>
    </item>
  </channel>
</rss>