How I Learned to Stop Worrying and Backup the Server

Server system administration is not my primary interest. For most of our services, we prefer to offload the headache of system administration to the fine folks at WP Engine. We utilize web services outside of WordPress, including our Help Desk and Knowledge Base running on Jira and Confluence hosted on an EC2 instance provided by Amazon Web Services.

We gain control of what we can run on the server, but the price is lost sleep because we need to manage our own backups. EC2 makes it fairly easy to do backups manually. Every EC2 instance uses a virtual disk volume with their Elastic Block Store service. The EBS volume can be backed up at a block level (like a full hard disk clone) using the Snapshots feature. Manual backups are not what we want because they still rely on one of us remembering to do it, which is error-prone and annoying.

Instead, we set up the EC2 instance to snapshot itself automatically, every day using cron, and email us a summary of the task.

Who's behind the camera?

The server may be snapshotting itself, but it requires an AWS user account with access to the server and snapshotting. In the identity management section of AWS called IAM, I added a group called "Snapshotters":

Groups in AWS IAM Console

To this group, I added an inline policy called CreateDeleteSnapshots where I define what permissions are available to users who will belong to this group:

IAM Group Policy - CreateDeleteSnapshots

The actual content of the policy is:

    "Version": "2012-10-17",
    "Statement": [
            "Sid": "Stmt1426541828000",
            "Effect": "Allow",
            "Action": [
            "Resource": [

The Action and Resource are the most important bits.  The Actions are the smallest number of permissions needed to perform snapshots automatically. The Resource is set to * enabling it to work on all servers.

I created a user called "snapshots" and added it to the Snapshotters group, so the "snapshots" user has all of those privileges I set up for the group.

AWS IAM User list of groups

Robots Taking Snapshots

It's all well and good to have a user to do the snapshotting, but we need a snapshotting mechanism too. Doc Brown isn't time traveling without the Delorean. We can create a script to take care of the snapshots for us, and then set it to run periodically using cron.

The AWS CLI tools are a great set of utilities to administer all kinds of AWS services from the command line, including snapshotting. I installed them easily on our Ubuntu server using sudo pip install awscli which places the single executable in /usr/local/bin/aws.

Breaking down the steps, we need to:

  1. Identify the current EC2 instance ID. One strategy is to use  the ec2metadata command that's built into all Ubuntu EC2 servers since Ubuntu 12.04. Here's another way to do it.
  2. Identify the volume ID of the EBS volume attached to this instance and acting as the main drive (i.e. /dev/sda1). We can use the ec2 describe-volumes command here.
  3. Create the snapshot, using the predictably-named ec2 create-snapshot command.
  4. Identify the snapshot IDs for all snapshots associated with this EBS volume and sort them by date. We can use the command ec2 describe-snapshots.
  5. We want to delete all but the 3 most recent snapshots. We can do this using ec2 delete-snapshot.

The meat of the script is:

export DATE_STR=`date +%y.%m.%d.%I`;
export INSTANCE_ID=`ec2metadata --instance-id`;
# Get the ID of the volume mounted as the root device on this instance
export VOLUME_ID=`/usr/local/bin/aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=$INSTANCE_ID Name=attachment.device,Values=/dev/sda1 --query 'Volumes[*].{ID:VolumeId}' | grep ID | awk '{print $2}' | tr -d '"'`
echo "Initiating EBS volume snapshot of volume $VOLUME_ID attached to instance ID $INSTANCE_ID...";
    /usr/local/bin/aws ec2 create-snapshot --volume-id $VOLUME_ID --description $VOLUME_ID;
echo "Done.";
echo "Deleting old snapshots...";
# Get any snapshots older than the last $NUMBER_OF_SNAPSHOTS_TO_KEEP
for SNAPSHOT_ID in `/usr/local/bin/aws ec2 describe-snapshots --filters Name=volume-id,Values=$VOLUME_ID --query 'Snapshots[*].{ID:SnapshotId}' | grep ID | head -n -$NUMBER_OF_SNAPSHOTS_TO_KEEP | awk '{print $2}' | tr -d '"'` ; do
    echo "Deleting snapshot $SNAPSHOT_ID...";
    /usr/local/bin/aws ec2 delete-snapshot --snapshot-id $SNAPSHOT_ID;
echo "Done.";

In order for this to work in the context of a cron job, we need to set the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION (this last one because our EC2 region, us-west-2, is different from the default of us-east-1) according to this AWS CLI guide. I also need to set the PATH to include /usr/local/bin, the location of the aws command.

Scheduling for Peace of Mind

This script I saved in a folder full of cron scripts in our home folder, /home/newsapps/cron/ and made it executable with chmod +x ~/cron/ I scheduled this with cron to run every day at midnight server time using crontab -e and by adding the following lines (this crontab generator helped greatly):

# EC2 EBS Snapshot -- run once a day
0 0 * * * /home/newsapps/cron/

The extra MAILTO= is what emails us the output of the script. The only trick with getting that to work is that I had to install a mail server. I innocently installed the mail program by doing sudo apt-get install mail and in the process installed the mail server postfix and configured for our Fully-Qualified Domain Name

iPhone Screenshot of email from cron job

Et voilà! I can check from my phone that the server backed itself up, and I can go run carefree through a field full of daisies in my dreams.


OS X Setup for News Apps Development

I have the good fortune to be working with a team that values productivity by providing me with an Apple laptop. OS X works really well for what we do and matches the way my brain works. I like to have the power of Unix under the hood, along with the inspiring design of the Apple operating system signature look and feel.

Starting with a fresh Apple MacBook Pro, delivered Monday morning while I was introducing myself in our daily video scrum, here's what I immediately installed to get to 95% of what I need to contribute to  the INN Nerds projects.


It's a really good idea to encrypt the hard drive using the FileVault feature, and it's offered by default on a new OS X setup. By default, this uses your iCloud password for encryption. Set your password to something challenging, which you should be doing anyway.


After you encrypt your drive, it's imperative that you have a regular backup strategy. I worked for 11 years in tech support, helping people recover data from their crashed laptop hard drives after accidentally running over their laptops in the car, and it was only possible if the drive wasn't encrypted. The odds of data recovery were still pretty bad, but the people who were already backing up didn't skip a beat in getting back to work. Your laptop is not your work, it's just a handy set of tools.

If nothing else, get an external hard drive that you leave at home and set up Time Machine to back up to it. Set a calendar reminder to do this regularly (aim for doing this daily). You could also back up to a Network Attached Storage on your home network, which would work over the wireless connection. Apple sells one called an AirPort Time Capsule.

Any files you're using that are shared interest to the company, work on them in Dropbox. For your coding projects, be sure to be regularly pushing your commits or branches to Github/Bitbucket.

System Updates

Install all OS X system software updates, all of them until you go blue in the face. At no time other than right now, with a computer that has nothing interesting or fun running on it yet, is it going to be less annoying to do a series of reboots. Just get it over with; install other software while this is going on, but reboot as often as you need in order to get up to date. Putting this off just leads to pain later.

System Preferences

Look through the OS X System Preferences and make a few choices. Perhaps you have personal preferences about how notifications do/don't appear, what the screensaver looks like, what hot corners do (what happens when you put your mouse cursor in the corner of a screen), or that the "Quack" sound should be used for all alerts. You can always make adjustments later, but you might as well explore what the computer can do. You chose to make news apps because you are curious and want to change things, remember?

The System Preferences window in OS X 10.10 Yosemite.

Web Browsers

Install additional web browsers. Safari is a fine browser, but when writing and testing web applications, it makes no sense to have only one browser installed. I install Google Chrome and Mozilla Firefox. Choose your preferred default browser and make sure you never get nagged again by the others.

FTP Client

You may not need the FTP client immediately, but when you do need to access an FTP server you'll be glad you already have a client. You can use FTP from the command line, but I find that experience to be akin to doing a road trip in a horse and buggy cart. Do yourself a favor, install Cyberduck, and ride in style with air conditioning and power steering.

Text Editor

If you already have a favorite text editor, feel free to skip this section. If you are open to trying new things, or are looking for a recommendation, then I highly recommend you install Sublime Text. It's free, is beginner-friendly, and using it makes me feel like I'm driving a space ship. I originally installed it for the color schemes which abound.

I also recommend installing the Package Control for Sublime Text, which then gives you access to a bunch of nifty tools that plug into the editor.

Terminal Emulator

The default terminal emulator that comes with OS X is okay, but I like pretty color schemes and it seems easier to do this in iTerm2. Anyway, there are more options and it's pretty popular, and it's free.

With either the default terminal emulator or iTerm2, create a new terminal and install some things you'll use there.

Command Line Utilities

Start with the command-line tools you'll need to use for further terminal goodness. You can install that by running xcode-select --install and following the instructions that appear to install the offered command-line tools.

Homebrew is so useful, I wouldn't be surprised if it was included in OS X in the future. It's a command-line package manager with a simple interface that lazy people like me can use. When I want to install a new CLI tool, and I don't want to deal with the tomfoolery of finding all the dependencies and the latest download link, I can usually do it with Homebrew. Get Homebrew with ruby -e "$(curl -fsSL".

Let's install MySQL, to be able to work with MySQL databases in the future. brew install mysql. Yup, that's it. There are some instructions it supplies you after completing the install about getting the MySQL server to start automatically or manually.

Later, we'll be using Python for a few projects, and the way to keep Python library versions and environments organized with different projects is to use virtualenv and virtualenvwrapper. To install those, we need to first install the pip Python package manager, and then we'll install the virtualenv packages with pip. Run sudo easy_install pip && sudo pip install virtualenv virtualenvwrapper in your terminal.

If you installed Sublime Text, it's nice to be able to invoke it from the command line, like subl We can create a subl alias by using these instructions.

We might as well generate an SSH key now, which we'll eventually use with Bitbucket and Github so we don't have to log in from the command line when pushing to those repositories. Generate an SSH key using the command ssh-keygen. The contents of ~/.ssh/ is what you can use for your Github and Bitbucket account settings.

Since we're on the topic of git, you should configure your name and email globally that you'll be using with Github/Bitbucket. Following these instructions from the official Git documentation book, you can do it like this:

git config --global "Nick Bennett"
git config --global

Virtual Machines

We use virtual machines to set up simulated environments of the public web servers we ultimately deploy to, mirroring in many ways the setup of that final environment but with greater speed and control. We use the free and open source project VirtualBox as a host for our virtual machines, in conjunction with Vagrant which gives us a scripted way to efficiently create those virtual machines.

For a real example of using VirtualBox and Vagrant, check out our deploy-tools project on Github; we use this for every one of our WordPress site projects. If you are developing a WordPress site, I highly recommend checking this out to smooth your development and deployment process.


Working remotely full-time only works if we're all in constant communication. We use a host of tools for this, most of them are browser-based. You can use HipChat in the browser too.  I strongly recommend installing that as a native client. Among other great things HipChat enables is the ability to share animated GIFs like this one:

Captain Kirk in a rain of Tribbles.

Dropbox is another tool we use that really begs for a native client to be installed. Sharing files through email or instant messaging kinda stinks. Dropbox is like the shared network drive so common to Windows-based networks, only the files actually go on your computer instead of accessing them through some tenuous ethereal connection. This is why FileVault hard drive encryption is so important.

Along with being able to share files and words, we need to be able to share secrets, aka passwords and logins. We use 1Password vault, available for free as a 30-day trial followed by a $50 purchase. For keeping the keys safe for our organization's fleet of virtual facilities, that's a minor expense.

I will cover personal preference in a future post or series of posts, when the whole INN team are able to weigh in on their personal software recommendations, what assists them in doing what they each do, and work-in-progress configurations.

Other Tools

I mentioned this briefly in the article but I want to point it out again: the other nerds at INN have documented the tools we use for collaboration, starting with HipChat and including several others, that we recommend to others to enable highly distributed and productive teams. This is all a work in progress, as we frequently re-evaluate what best fits our changing needs.

Beyond the collaboration tools, check out this list we maintain of free or discounted tech tools available to non-profit news organizations.

I would also recommend you take a look at How to Setup Your Mac to Develop News Applications Like We Do on NPR Apps' blog, which helped me get started.

Please comment and share your tips for getting started, links to your own guide or others, and share your recommended software and configs.