Notes on cleaning up the MediaTemple hack (JohnnyA)

NOTE: Be sure to read through the comments, there’s a lot of additional and very useful information in there. I’ll compile the notes as soon as I’m able and add them to the main post.

First, the hack described here is not MediaTemple’s fault, nor an exploit of their security system. It’s a reminder to all of us that checking permissions and updating software are the bare minimum we can do to avoid a lot of headache and potential security issues. MediaTemple was apparently just targeted because of their size, the responsibility for the hacked sites lies with the site’s owner/webmaster.¹

Second, it doesn’t just affect WordPress. I found bits of the exploit in MODx files, static files, and even some sites that weren’t accessible through the internet.

Lastly, to the perpetrators: thanks for the security reminder, but curses upon you and your families. May you spend eternity grepping for obscure lines of code while every client you have calls you to tell you their business is going to fail because their site is throwing security warnings.

What follows is not a full-fledged guide to cleaning up a hacked server, just some notes from my experience with this round.

Have you been hacked?

The exploit is a little difficult to notice. It inserts percent-escaped lines of code into JavaScript files you’re already including on your page, which then use document.write to add redirect code to the rendered page. This means that viewing source won’t show it, you have to see the rendered source, ala Firebug, Safari Web Inspector, or one of various plugins that show the actual source after the DOM has loaded.

What you’ll find in the rendered source is a little function that uses an array of colors and urls to generate somewhat random redirects. A url called ‘edisonsnightclub’ is always in the mix, so that’s what I search for. The redirect has the side-effect of triggering Firefox’s security warnings, which is a fair sight better than just sending clients off to malicious sites. However, it appears to only execute the code for search engine bots, so it could potentially go unnoticed on your site for quite some time.

Cleanup

I did my cleanup job hastily, and didn’t track everything as well as I should have. There were lines inserted into main template PHP files which I just quickly deleted. These always began with $o=, so that became my server-wide search pattern (more on that in a second). This didn’t only affect WordPress, it also hit my MODx sites and the exploit would probably hit static sites or any other CMS just fine, so check everything.

Once the PHP files were cleaned up, I realized that the most insidious part of the hack was that it picked random script files from your page to insert itself into. Tracking it down got interesting. One thing to note: you can tell what file it’s in when you view the rendered source by looking at the script tag right before the exploit code. The included script is where you’ll find the document.write line.

The point of entry on my GridServer turned out to be a single directory in a single WordPress install, from which a rootkit (identifiable by the existence of an is_writable.php file) was able to gain write access to 20+ sites that I host there. The rootkit was found in the img directory of a TinyMCE theme within a WordPress install.

The exploit doesn’t even change the mod dates on the files, so it’s hard to filter out the infected scripts. What I did was SSH in and cd to the domains directory, where I ran a grep for the patterns I’d found so far. The line within the javascript file looks like this (truncated):

var st1 = 0;document.write(unescape('%3C%73%63...%74%3E'));var gr0=0;

So I set up a grep command like this:

find ./ -name "*.js" -exec grep -l "gr0=0" {} \;

That gives me just the name and path of any file which contains a match to the string “gr0=0,” which is unique enough to narrow things down. I ran a similar grep for the PHP files:

find ./ -name "*.php" -exec grep -l '$o=' {} \;

That one yields some false positives, I need to find a more specific search criteria. It gets the job done, though. Then it’s just a matter of manually cleaning out the inserted lines. Be careful not to delete anything in your JavaScript files after the gr0=0;, such as the opening of a comment block or the first line of a jQuery plugin. Just delete up to the semicolon and you’ll be fine.

You’ll also want to check all of your databases. First look for extra admin users (JohnnyA is what they were always called in mine), then check for lines in your content tables as mentioned on the MediaTemple wiki.

Check that your permissions are secure² and make sure you find the rootkit and remove it. That will be several PHP files in a random directory, and they’ll have recent modification timestamps. As I mentioned, one of them should be called is_writeable.php, which you can use in a find statement as well.

Best of luck, and please add any additional notes or corrections in the comments!