Tales from a base64 WordPress Hack, part 3: prevention

(See parts one and two for background information.).

My sites are almost fully recovered. I’ve ditched the old fancy theme in favor of a more austere default theme that turns out to have some pretty nice features. I like minimalism anyways.

It turns out that the culprit was a vulnerability in timthumb.php. If ANY of your themes, even one you don’t currently have active, happen to use this script, please ensure that you download the latest version of timthumb.php and replace all instances on your server with that new version. It was quite common, particularly among so-called “premium” themes, and many themes that used image rotators on the homepage. Also, backup all themes on your site to your local computer, and then delete all themes you are not actively using on your blog. No reason to introduce any additional vulnerabilities.

In the aftermath, I have been able to implement a few solutions to help prevent this from happening again. I actually already had one relapse, but recovered in minutes. Here they are:

Cleaning it More Easily

My service provider helpfully provided this shell / perl script to clean out future infections. I have tested it and it does indeed work. Run it from the shell prompt in whatever directory you want to clean; it will clean out all PHP files in that directory and any child directories. (CAVEAT: This will only work so long as the attack mode remains the same; if they change the encoding so that it does not perfectly match, this will not work. Always verify your files before and after to ensure they have changed appropriately).

find . -name "*.php" -exec perl -pi -e 's/<\?php.*eval\(base64_decode\(.*\)\);\?>//' {} \; -exec perl -pi -e 's/<\?php.*aWYoZnVuY3Rpb25fZXhpc3RzKCdvYl9zdGFydCcpJiYhaXNzZXQoJF9TRVJWRVJbJ21yX25vJ10pKXsgICRfU0VSVkVSWydtcl9ubyddPTE7ICAgIGlmKCFmdW5jdGlvbl9leGlzdHMoJ21yb2JoJykpeyAgICBmdW5jdGlvbiBnZXRfdGRzXzc3NygkdXJsKXskY29udGVudD0iIjskY29udGVudD1AdHJ5Y3VybF83NzcoJHVybCk7aWYoJGNvbnRlbnQhPT1mYWxzZSlyZXR1cm4gJGNvbnRlbnQ7JGNvbnRlbnQ9QHRyeWZpbGVfNzc3KCR1cmwpO2lmKCRjb250ZW50IT09ZmFsc2UpcmV0dXJuICRjb250ZW50OyRjb250ZW50PUB0cnlmb3Blbl83NzcoJHVybCk7aWYoJGNvbnRlbnQhPT1mYWx.*<\?php/<\?php/' {} \;

Use Git Repositories

Most Linux-based servers should support git, or at least allow you to install it. Git is a really fantastic tool, primarily intended for development, but I have figured out a clever hack to use it to easily monitor your files and act as a de facto backup tool; making it easy to take a snapshot of your site.

First, determine if you have it. In your shell, simply type git. You should see something other than "command not found". If you don't have it installed, contact your service provider's helpdesk and ask them about it.

Create the bare repo

I like to use a bare repo for this because it prevents you from accidentally monkeying with the files and also makes it clear about the purpose. If you have multiple sites, you will want to create a repo for each individual one. Maybe create a folder called "backups"?

/home/USERNAME/$ mkdir ~/backups
/home/USERNAME/$ cd ~/backups

Now inside of this folder, you'll initialize your bare repo. We'll just use a generic "blog" name here.

/home/USERNAME/$ git init --bare blog
Initialized empty Git repository in /home/USERNAME/backups/blog/

Also, if this is your first time using git, you'll want to set up some global config options. You can type these from any directory.

$ git config --global user.name "Firstname Lastname"
$ git config --global user.email "your_email@youremail.com"
Git-i-fy your site

Currently the bare repo is, well, bare. We need to fill it with good stuff. Navigate to your blog location (I will assume it is in /home/USERNAME/blog/, for the sake of this example). We need to do a normal "init" of git there as well.

WARNING: Before you do this, ensure that your site has been properly cleaned. You don't want to commit infected files to the repo, now, do you? If it was an infected wordpress installation, I would suggest cleaning and backing up the config files and the uploads folder. Here's a cheat sheat:

/home/USERNAME/blog/$ tar -cf blog_backup.tar *
/home/USERNAME/blog/$ gzip blog_backup.tar
/home/USERNAME/blog/$ mv blog_backup.tar.gz ~/
/home/USERNAME/blog/$ cp wp-config.php ~/
/home/USERNAME/blog/$ cd wp-content/uploads
/home/USERNAME/blog/wp-content/uploads/$ rm -rf *.php
/home/USERNAME/blog/wp-content/uploads/$ cd ~/blog/
/home/USERNAME/blog/$ mv wp-content/uploads/ ~/uploads.backup
/home/USERNAME/blog/$ rm -rf *
/home/USERNAME/blog/$ wget http://wordpress.org/latest.zip
/home/USERNAME/blog/$ unzip latest.zip
/home/USERNAME/blog/$ mv wordpress/* .
/home/USERNAME/blog/$ mv ~/wp-config.php .
/home/USERNAME/blog/$ mv ~/uploads.backup wp-content/uploads

With this done, navigate to your blog on the live site, it should look like a blank screen. OH NO!

Just kidding. Go to the admin page: http://yourblogurl.com/wp-admin/ and login as your admin user. Go to the Appearance page and it should inform you it can't find your old theme so it's switching to the default. Go back to your blog URL again and it should now look plain and boring, BUT INTACT. You may need to flush your permalinks by going to Settings -> Permalinks and clicking "update permalinks" or "save" or whatever it's called. The big blue button at the bottom of that page.

If stuff looks crazy or you realized "OMG I HAD MY DOCTORAL DISSERTATION UPLOADED TO THE ROOT FOLDER!!!!" don't worry, you've got a backup created and compressed in ~/blog_backup.tar.gz.

Back in the shell...

So first we're going to initialize the local git repository for the blog. This is important as it tracks all file changes.

/home/USERNAME/blog/$ git init
/home/USERNAME/blog/$ git add .
/home/USERNAME/blog/$ git commit -m "Initial commit of clean WP install."

If you have a crap ton of stuff in your uploads folder this may take a while.

Now we have to link it to the bare repo.

/home/USERNAME/blog/$ git remote add origin /home/USERNAME/backups/blog/

It's important that you type out the full path there. Now push that puppy on up the river, into the bare repo.

/home/USERNAME/blog/$ git push origin master

If you have any really large files in your uploads folder this will take a while. It may even tank, if those files are > 100M or so. If it does, you will need to start over. I had to do this a few times since a couple of the blogs had video files uploaded. If it happens to you, comment on this blog and I'll reply with how to do it (unless you want to figure it out yourself!).

Otherwise -- congrats, you now have a backup of your site!

Maintaining Your Blog

Now that you have a git repository of your blog, you will need to maintain it occasionally. There is one really crucial command you need to learn:

/home/USERNAME/blog/$ git status

That command will inform you of every file that has been changed, compared to the repo. If you get hacked again, a LOT OF FILES WILL APPEAR ON THIS (see below for instructions on how to recover). If you just installed a new plugin, or uploaded some images for a blog post, those files will appear, though it may just reference the folder they are in, rather than each individual file.

As an example, here's what my git status shows since I recently updated my theme on this blog:

~/blog$ git status
# On branch master
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   wp-content/themes/twentyeleven/style.css
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-1000x288.jpg
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-1024x393.jpg
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-150x150.jpg
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-300x115.jpg
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-500x192.jpg
#	wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o.jpg
#	wp-content/uploads/2012/03/background-1000x288.jpg
#	wp-content/uploads/2012/03/background-1024x768.jpg
#	wp-content/uploads/2012/03/background-150x150.jpg
#	wp-content/uploads/2012/03/background-300x225.jpg
#	wp-content/uploads/2012/03/background-400x300.jpg
#	wp-content/uploads/2012/03/background.jpg
#	wp-content/uploads/2012/03/rockstand-150x150.jpg
#	wp-content/uploads/2012/03/rockstand-300x86.jpg
#	wp-content/uploads/2012/03/rockstand-500x144.jpg
#	wp-content/uploads/2012/03/rockstand.jpg
no changes added to commit (use "git add" and/or "git commit -a")
~/blog$ 

Note the two sections: In the first section "modified", it lists all files that it knows about that have been changed. If your site got hacked again, a bunch of files would show up here.

The second group is "untracked files". These are files that git sees that it has no effing idea what they are. Be sure to carefully look through the list of files that shows up (if any) when you do this. Images and CSS are typically ok. If you see any files with the "php" or javascript files (extension "js") you may want to view them in your favorite text editor first, before committing. This is really important The integrity of this backup system is built upon the foundation that you are vetting all files that go into it, so that it stays clean.

Once you've verified your stuff is kosher and ready for Passover dinner, you can commit the changes:

~/blog$ git add .
~/blog$ git commit -m "Updated theme."
[master a454066] Updated theme.
 17 files changed, 2698 insertions(+), 2679 deletions(-)
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-1000x288.jpg
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-1024x393.jpg
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-150x150.jpg
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-300x115.jpg
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o-500x192.jpg
 create mode 100644 wp-content/uploads/2011/05/218258_10150148884453639_551193638_6654329_4309630_o.jpg
 create mode 100644 wp-content/uploads/2012/03/background-1000x288.jpg
 create mode 100644 wp-content/uploads/2012/03/background-1024x768.jpg
 create mode 100644 wp-content/uploads/2012/03/background-150x150.jpg
 create mode 100644 wp-content/uploads/2012/03/background-300x225.jpg
 create mode 100644 wp-content/uploads/2012/03/background-400x300.jpg
 create mode 100644 wp-content/uploads/2012/03/background.jpg
 create mode 100644 wp-content/uploads/2012/03/rockstand-150x150.jpg
 create mode 100644 wp-content/uploads/2012/03/rockstand-300x86.jpg
 create mode 100644 wp-content/uploads/2012/03/rockstand-500x144.jpg
 create mode 100644 wp-content/uploads/2012/03/rockstand.jpg
elecmahm@delphinus:~/blog$ 

Verify that the commit took with another call to git status:

~/blog$ git status
# On branch master
nothing to commit (working directory clean)
~/blog$ 

Now we're ready to push up to the bare repo.

~/blog$ git push origin master
Counting objects: 37, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (27/27), done.
Writing objects: 100% (27/27), 1.04 MiB, done.
Total 27 (delta 7), reused 0 (delta 0)
Unpacking objects: 100% (27/27), done.
To /home/USERNAME/backups/blog/
   a1c5cc5..a454066  master -> master
~/blog$ 

All done! Congrats.

You will want to check in on your blog on a regular basis. If you upload stuff or update it frequently, check in weekly and do a commit (vetting your files!) and push to the master repository. If you're very clever, you'll make a shell script that runs a git status and emails you the output, then assign it to a weekly cron task.

Recovering From Disaster

Oh no! So some sort of disaster has befallen your site -- maybe it got hacked. Maybe you BROKE ALL THE THINGS. Maybe you deleted it by mistake. Or maybe you're just paranoid.

Before doing anything destructive, do a quick git status and see how much is different. In particular, you want to check the uploads folder (if it still exists) to see if anything has been uploaded since your last snapshot.

~/blog$ git status | grep uploads
#	wp-content/uploads/testing.txt
~/blog$ 

In this example, I created a file called "testing.txt" for demo purposes. If you have no new / changed files in uploads, you will see nothing outputted from that command.

Let's assume, for the sake of relevant example, that your site got hacked again and the shell script I mentioned in the beginning didn't work. It's bad. Real bad. Like sex tape bad. Weird fetish sex tape uploaded to porn site in the amateurs section bad. First off, backup the site because hey, why not.

/home/USERNAME/blog/$ tar -cf blog_backup-TODAYSDATE.tar *
/home/USERNAME/blog/$ gzip blog_backup-TODAYSDATE.tar
/home/USERNAME/blog/$ mv blog_backup-TODAYSDATE.tar.gz ~/

Now we're going to blow it all away.

/home/USERNAME/blog/$ rm -rf *
/home/USERNAME/blog/$ rm -rf .git/*

Wait, what? Deleting git??? Why???

Just a precaution. We don't know how badly they corrupted your data, so it's best to just raze it all and start fresh.

Re-create a new git node here:

/home/USERNAME/blog/$ git init
/home/USERNAME/blog/$ git remote add origin /home/USERNAME/backups/blog

And now we pull down the backup:

/home/USERNAME/blog/$ git pull origin master

You should see a lot of stuff fly by. When you're done, do a directory listing, or just check your blog in your browser. It should look normal again! (Any files that you changed or uploaded since the last snapshot will not be reflected, but any posts you've written should still be intact.)

This method of restoration will NOT correct database corruption, so you should still make routine backups of your wordpress database.

By using git to create regular snapshots, you will benefit in two ways:

  1. git status will quickly and easily tell you if any files were changed.
  2. Should you ever have to restore your data, you can always pull down the last snapshot you uploaded.

Cheers to that!