alan

mostlyunixish

I was tired of repeating the same old pattern. Send a kill signal. Check with pgrep. Did the process exit? Send a kill -9.

Now all my knowledge has been permanently dumped here:

Kill Me Softly on GitHub

If you want to kill a process by letting it to exit gracefully, and yet kill it without mercy if it's too slow to exit, that's your new favourite shell script.

You can discuss this on Hacker News as well

#MostlyUnixish #linux #processmanagement #kill

A desktop freeze is something that's not so rare nowadays – that's not good, but we shouldn't get desperate about that.

I like Compiz with Unity Desktop on my Ubuntu Trusty very much; I feel it's a step forward compared to traditional desktops. The only major nuisance I had on the previous versions was the inability to disable the global menu (which I like by itself, but I don't like the global menu disappearing when the mouse is not over it), but you can now disable it and have the menu on the window – even though you can't still enable an always-visible behaviour.

The problem is: compiz sometimes gets buggy. Running a different window manager, like metacity and/or xcompmgr just doesn't seem good, you can experience other issues and you're using the stack in a way it wasn't designed for. I don't like that approach.

So, what can you do when your desktop freezes? The symptoms are usually very clear: you can move your mouse, but you can't click nor type anything.

You don't need to reboot nor to restart X. You can fix this without losing your work.

Get a raw tty

Press Ctrl+Alt+F1 and get to a raw linux console, and login with your credentials. Make sure you use the same user you're logged into X with.

Find and kill your stuck process

pgrep compiz

This will print the PID of our stuck window manager.

Let's kill it

pkill compiz

This won't usually be enough because the process is really stuck: if you repeat the pgrep command above, you'll find it still running – i.e. pgrep will return the very same PID as above.

So let's kill it, really this time!

pkill -9 compiz

Then wait some 15-20 seconds.

Now, if you repeat the pgrep , you'll either see a) no result or b) a different PID.

If you see a different PID, it's Ubuntu that noticed the crashed process and relaunched it; press Ctrl+Alt+F7, and you should just get back a working desktop!

Relaunching

If you see nothing with pgrep, you'll need to relaunch Compiz manually.

You cannot simply relaunch compiz on the console, because it must connect to X: the proper way to do it is:

DISPLAY=:0 compiz

Now switch back to your X instance with Ctrl+Alt+F7 and enjoy!

#MostlyUnixish #linux #ancientwisdom #community #ubuntu #compiz

I seem very ancientwisdom-oriented these days.

Sometimes you've got a system user on your linux server that's used to run a specific service; it's a good practice to employ those users to achieve privilege separation, since it limits the impact of many security breaches. It's security 101, really.

If you've got a system user, most probably its shell would be /bin/false or /bin/nologin. Again, this is a good practice, since it would be harder for an attacker to gain shell access to your machine through that user.

But I find that many people just change such shell, because sometimes they need to customize the environment for a user, and if you try doing something like

su - systemuser

you'll find yourself just back to your original shell, because su tries to invoke the shell which is set for the target user, and sudo doesn't generally help here, because its --shell option doesn't start a proper login shell for that user (most of the environment is that of the calling user).

Very well, gentlemen: this is a wrongdoing and a potential security hole. You should never enable a shell for your system users. Instead, you should leverage an option of su:

su - systemuser --shell /bin/bash

That lets you choose your favourite shell on a per-launch basis.

#MostlyUnixish #linux #ancientwisdom #su #security

This is another old trick, but I like to tell the world about such amenities.

I often find people writing command lines such as:

ssh arcane-system-user@someextremelylong.andimpossibiletotype.anderrorprone.hostname.com

or, worse

ssh idliketorememberwhichuser@10.20.30.40

or maybe it was

ssh someuserimashamedof@1.2.3.4

yes, sometimes ssh is simply tedious.

But you can do better. In fact, you can do much better.

your ~/.ssh/config or /etc/ssh/ssh_config can contain Host configuration sections; I have already blogged about such directives when I told the world about stopping key verification compliants – they're very flexible, indeed.

Consider adding Host directives in your configuration for your most used services:

Host myniceserver
    Hostname myniceserver.mylove.com
    Port 12345
    User teddybear

And then you can just

ssh myniceserver

You can override most options on a per-server basis and/or select a certain private key for connections.

#MostlyUnixish #linux #ssh #ancientwisdom

This is a very old trick, but I find many people doesn't know about it yet.

If you're using sudo (and you definitely should), one minor complaint is that you need to re-enter your password quite often.

One common pseudo-solution is to completely disable the password for executing sudo with such user; while it may be a good idea sometimes, it does decrease the overall security – if you happen to forget an open shell or screen/tmux lingering somewhere, you give total power to whoever gets in touch with it.

If you happen to let people into your sudo-enabled account with your ssh key, anybody that can get hold of your ssh key will get root access to your servers. It's not the best possible scenario.

There's a very simply workaround: just edit your sudoers file!

sudo visudo

and add a line like that (or modify it if it's already there)

Defaults    timestamp_timeout=120

This means that sudo will ask for a password every two hours instead of the default (15 minutes on most installations). You can use -1 in order to never let it expire (I don't suggest that – 8-10 hours should be enough for everybody.)

The only drawback is that such timeout is per shell, not system-wide – i.e. if you open multiple terminals, you'll have to enter the password again for each new terminal instance.

#MostlyUnixish #linux #sudo

It might be high time to update to the latest Ubuntu 14.04.

I think that the upgrade process that is provided by Canonical is slightly underoptimal: if you're using a local mirror which can provide a very fast download of all packages, they'll usually be disabled during the upgrade process, and an official mirror (possibile one very far away) will be employed.

This can lead to a very slow and painful upgrade process.

The solution is easy.

The following instructions regard moving from 12.04 (precise) to 14.04 (trusty)

Switch your sources

Assuming that your official ubuntu sources are configured from /etc/apt/sources.list just do something like:

sudo sed -i.old -e 's/precise/trusty/g' /etc/apt/sources.list

I'm not considering other sources in /etc/apt/sources.list.d, since they'll be disabled whatsoever; but if you happen to have ubuntu sources there as well, you should repeat the SEDding process for each file.

Update your index

sudo apt-get update

Download updated packages without installing

sudo apt-get dist-upgrade --download-only

Revert to old sources

sudo mv -f /etc/apt/sources.list.old /etc/apt/sources.list

Update index again

sudo apt-get update

Rejoice!

Now you can just follow the upgrade instructions and almost nothing will be downloaded during the upgrade, because everything is already on your hard drive.

#MostlyUnixish #ubuntu #linux #upgrade

UPDATED: 26th January 2015, with Tomcat 8 support.

Today I'm gonna share something that I figured out some times ago: how do I keep my tomcat installation up to date on my servers?

Of course I'm not willing to automatically switch majors, but I'd like to pull bugfix releases in as soon as possible; most of my servers should work unattended, and I prefer to handle it if something crashes rather than being hacked because of some public exploit.

Apache Tomcat makes it incredibly hard to do so for different reasons: first, there's no latest link for the latest release, you must first check the homepage to see what's the latest version; second, the archive you can download mingles the application itself with the configuration, log and webapp dir.

What should we do, then?

Tomcat allows us to configure almost everything. Let's use its power!

For this setup, I use Tomcat7 and I chose to employ /etc/tomcat7/conf as a configuration directory, /opt/tomcat7/latest as tomcat7 own directory, /opt/tomcat7/webapps for our webapps, /opt/tomcat7/logs, /opt/tomcat7/temp, /opt/tomcat7/work as logging, temp and work directory, respectively.

The same applies for tomcat8, it's just the prefix that will change.

To begin with, create a user (maybe a service user, I'll leave the details to you) tomcat with a main group tomcat as well, which we'll use to run the servlet container – you can pick a different user if you like, the class allows you to choose.

Now make sure your Puppet manifest includes the tomcat_versioned class with the proper parameters. In order to do so, you'll need this custom fact that helps the system determine the very latest tomcat7 version, place it in your custom facts directory:

#!/usr/bin/env python
# requires lxml
from lxml.etree import HTML
import re
import sys
import urllib

for version in (7,8):
    pattern = re.compile("^{0}\.0\.\d\d\d?$".format(version))
    root = HTML(urllib.urlopen("http://tomcat.apache.org/download-{0}0.cgi".format(version)).read())
    for e in root.iterdescendants():
        if isinstance(e.text, basestring) and pattern.match(e.text.strip()):
        print "tomcat{1}_latest_version={0}".format(e.text.strip(), version)
        break

and the tomcat_versioned class, of course; just pass the major of the tomcat you'd like to use, and it will find the latest tomcat update for such major, and install it in /opt/tomcatX:

# this will install latest tomcat from apache website, and yet retain your
# config from /etc/tomcat7/conf
# tested with puppet 3.7 and tomcat 7 and 8.

class tomcat_versioned($tomcat_major=8, $tomcat_user="tomcat") {
    # assumption: the user should exist, it won't be created
    # you should define a tomcatX service outside this class
    $tf = getvar("tomcat${tomcat_major}_latest_version")
    $tm = "tomcat${tomcat_major}"

    exec { "/bin/tar xvf apache-${tm}-archive.tar.gz":
    creates =>"/opt/${tm}/apache-${tm}-${tf}",
    cwd => "/opt/${tm}",
    refreshonly => true,
    alias => "${tm}_unpack",
    require => File["/opt/${tm}"]
}

file { "/opt/${tm}":
ensure => "directory",
mode => 0755,
owner => "root",
group => "root"
}

file { "/opt/${tm}/apache-${tm}-archive.tar.gz":
ensure => "present",
source => "/tmp/apache-tomcat-${tf}.tar.gz",
require => Exec["${tm}_download_latest"],
notify => Exec["${tm}_unpack"]
}

exec { "/usr/bin/wget --timestamping http://www.eu.apache.org/dist/tomcat/tomcat-${tomcat_major}/v${tf}/bin/apache-tomcat-${tf}.tar.gz":
alias => "${tm}_download_latest",
cwd => "/tmp"
}

exec { "/bin/ln -sf --no-target-directory apache-tomcat-${tf} latest":
refreshonly => true,
subscribe => Exec["${tm}_unpack"],
cwd => "/opt/${tm}",
alias => "${tm}_symlink"
}

exec { "/bin/rm -rf conf.orig && /bin/mv -f conf conf.orig && /bin/ln -sf --no-target-directory /etc/${tm}/conf conf":
refreshonly => true,
cwd => "/opt/${tm}/latest",
subscribe => Exec["${tm}_symlink"],
notify => Service["${tm}"],
alias => "${tm}_config_move"
}

# first-time only executions. I might like to abstract some logic if I were a bit less lazy than I am.
# in order to stay on the safe side, we never let the normal user to access our files; this may be relaxed,
# depending on your context.

# this contains our config. our servlet container should be able to read it, but never write it.
exec { "/bin/mkdir -p /etc/${tm} && /bin/cp -r /opt/${tm}/latest/conf.orig /etc/${tm}/conf && /bin/chmod 0750 /etc/${tm}/conf && /bin/chown -R root:${tomcat_user} /etc/${tm}/conf && /bin/chmod 0640 /etc/${tm}/conf/*":
creates => "/etc/${tm}/conf",
subscribe => Exec["${tm}_config_move"]
}

# this will contain the actual code of our webapps. Again, the container must be able to read them,
# never write to them.
exec { "/bin/mkdir -p -m 0750 /opt/${tm}/webapps && /bin/chown root:${tomcat_user} /opt/${tm}/webapps":
creates => "/opt/${tm}/webapps",
}

# those are working directories where the container must be able to write.
exec { "/bin/mkdir -p -m 0770 /opt/${tm}/work && /bin/chown root:${tomcat_user} /opt/${tm}/work":
creates => "/opt/${tm}/work",
}

exec { "/bin/mkdir -p -m 0770 /opt/${tm}/temp && /bin/chown root:${tomcat_user} /opt/${tm}/temp":
creates => "/opt/${tm}/temp",
}

exec { "/bin/mkdir -p -m 0770 /opt/${tm}/logs && /bin/chown root:${tomcat_user} /opt/${tm}/logs":
creates => "/opt/${tm}/logs",
}
}

The tomcat7 service is trivial and omitted, fill it in by yourself (or delete it).

Once this is done, you'll find a base configuration in /etc/tomcat7/conf. We must now edit server.xml there and tell Tomcat where we'd like to have our webapps, logs and work directories:

<!-- this should go inside <Server><Service><Engine> -->

<Host name="localhost"  appBase="/opt/tomcat7/webapps"
            unpackWARs="false" autoDeploy="true" workDir="/opt/tomcat7/work">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/opt/tomcat7/logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>

Last but not least, we should tell tomcat what is CATALINA_HOME and CATALINA_TMPDIR. This is an example script that can be used with upstart on Ubuntu 12.04, change it according to your OS:

description "tomcat7"

  start on runlevel [2345]
  stop on runlevel [!2345]
  respawn
  respawn limit 10 5

  # run as non privileged user
  # add user with this command:
  ## adduser --system --ingroup www-data --home /opt/apache-tomcat apache-tomcat
  # Ubuntu 12.04: (use 'exec sudo -u apache-tomcat' when using 10.04)
  setuid tomcat
  setgid tomcat

  # adapt paths:
  env JAVA_HOME=/usr/lib/jvm/java-7-oracle
  env CATALINA_HOME=/opt/tomcat7/latest
  env CATALINA_TMPDIR=/opt/tomcat7/temp
  env HOME=/home/tomcat

  # adapt java options to suit your needs:
  env JAVA_OPTS="-Djava.awt.headless=true"
  env CATALINA_OPTS="-Xmx1536M -server"

  exec $CATALINA_HOME/bin/catalina.sh run

  # cleanup temp directory after stop
  post-stop script
    rm -rf /opt/tomcat7/temp/*
  end script

In this situation I had a “normal home” for tomcat because I needed it, but your requirements may work differently.

I hope that's useful to somebody.

#MostlyUnixish #puppet #ConfigurationManagement #tomcat

Sometimes you would like to use an http URL as the source file for puppet; while it's being discussed since long, no implementation has been created so far.

Of course it would be better to periodically download the file from its http source to the puppet master, and then use puppet file server to send such file to our client; but that's impractical in some contexts, especially if you're using your puppet in headless mode, i.e. without a master. In such situations you may really want to source files from http, without redownloading them if they haven't changed.

I found a good way to do it is to use the excellent [wget] http://www.gnu.org/software/wget/) command line tool.

This is an example of what I use to keep my installation of Jenkins up to date:


class jenkins {
    exec { "/usr/bin/wget --timestamping http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war":
        alias => "jenkinslatest",
        cwd => "/tmp",
    }

    file { "/opt/tomcat7/latest/webapps/jenkins.war":
        ensure => present,
        source => "/tmp/jenkins.war",
        alias => "jenkinswar",
        require => Exec["jenkinslatest"] }

}

The timestamping flag will let wget check the remote server for file modification time; if file hasn't changed, it won't be redownloaded.

Just choose a proper directory for temporary downloads (tmp is volatile and gets sometimes cleaned up at runtime as well), make sure the server properly sets the Last-Modified header and remember to refresh your other resources when files change, and you're done.

#MostlyUnixish #puppet #ConfigurationManagement #http

Sometimes you need to write a quick shell script; many a time, you need to put it on a server, and launch it every now and then, or run it in a cron job. That script may end up doing something important and/or altering the state of the server machine.

If that script starts being using often enough, you'll want logging as well, and that can be a complex beast to tame, for logs require rotation and deletion, otherwise the beast will eat up all your disk space. Setting up logrotate for a single shell script looks like an overkill to me most of the times.

So what? Many a times logging will be neglected, and you'll lose the ability to track what happened and why.

This is my quick recipe for a basic bash logging solution with line-based truncation: the logfile will never exceed RETAIN_NUM_LINES lines (actually, it will never exceed such size when the script is launched, so the number of lines when the script has run is RETAIN_NUM_LINES + number of lines logged by the script).

Whenever you use the log function you'll get output on stdout as well, complete with a timestamp.

#!/bin/bash

# set LOGFILE to the full path of your desired logfile; make sure
# you have write permissions there. set RETAIN_NUM_LINES to the
# maximum number of lines that should be retained at the beginning
# of your program execution.
# execute 'logsetup' once at the beginning of your script, then
# use 'log' how many times you like.

LOGFILE=quicklog.sh.log
RETAIN_NUM_LINES=10

function logsetup {
    TMP=$(tail -n $RETAIN_NUM_LINES $LOGFILE 2>/dev/null) && echo "${TMP}" > $LOGFILE
    exec > >(tee -a $LOGFILE)
    exec 2>&1
}

function log {
    echo "[$(date --rfc-3339=seconds)]: $*"
}

logsetup

log hello this is a log

Launch this a lot of times and you'll end up with this in quicklog.sh.log:

[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:09+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:10+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:11+02:00]: hello this is a log
[2016-09-26 11:44:14+02:00]: hello this is a log

Enjoy!

UPDATED on September 26th, 2016 to employ the rfc-3339 date option, which makes the logline quite more readable and greppable.

#MostlyUnixish #bash #scripting #shell

Sometimes when performing your daily apt-get update, you'll experience messages like

E: The package lists or status file could not be parsed or opened.

or

The package index files are corrupted

Such messages usually don't go away for quite a long time, and may prevent legitimate updates from installing.

This usually happens even more frequently if using a proxy to access apt archives.

While I don't know what's the root problem, I've got a quick workaround that seems to fix the immediate error:

sudo -s -H bash -c 'apt-get clean && rm -f /var/lib/apt/lists/* ; rm -f /var/lib/apt/lists/partial/* && apt-get clean && apt-get update'

#MostlyUnixish #linux #ubuntu #debian #deb #apt