The Warning Signs

First, I started receiving messages from followers on Twitter about virus alerts on my site.  I try to personally respond to every message I receive, but at that time I had nearly 15,000 Twitter followers and was getting a ton of spam!  Most of it is about some video they found of me on Facebook or how someone is writing terrible things about me.  (It may have worked to trick me to click through if I didn’t have 20 different people all sending me the exact word-for-word message.)  After checking my site and not seeing anything wrong or getting any virus alerts when I visited it, I assumed these were more virus spam.

I have thousands of great followers, some of whom have alerted me when there were problems with my site which I have taken very seriously, but I could’t find anything wrong this time.

Finally, someone sent me a screenshot of a warning message from Chrome.  I checked it once again, and bam!  There it was in my face. Google had blacklisted me as a potentially malicious website.

Flailing About Helplessly

Frantically, I searched the code on my site, but I couldn’t find anything wrong.  Google Webmaster Tools will even tell you exactly what code on your website is getting you blacklisted, but I couldn’t find that code, even when doing a full search of my site.

I immediately changed all of my passwords including FTP, WordPress, and MySQL.

Pinpointing the Problem

After much cursing of Google for telling me what the code was but not where to find it, I found how hackers often use base64 code to obfuscate what they are doing, and then I shifted tactics.  I had been searching the source code in the browser, falsely assuming that if Google saw the code then I should be able to see it in the source code.  I was even using the Web Developer tool to view the generated source code, so that I would be sure to see any generated javascript.

Now I decided to search the source files on the server.  This was going to be a pain because I am using WordPress and that pulls in files from all over the place.  Thank God for Linux.

My First Clue

I used Putty to SSH into my server.  If you don’t have SSH access to your site and you are serious about web development, then you need a new web host.  From the command line I used the “grep” command, one of my all time favorite Linux tools, to quickly and easily search every file in my site.  I searched for the PHP command to eval base64 code, and a lot of things came up that were just white noise that I had to sift through, but eventually I found something very suspicious.

Inside my index.php file before the doctype was even declared, before the header is called in, before anything else there was obfuscated code that I didn’t think should be there.  The first mistake I made was quickly deleting the code and saving the file.  When you have been searching all day, and you finally find the problem, your first instinct is to remove it ASAP.  But that code has a lot of information that could prove helpful in finding anything else, because I guarantee that this is NOT the only thing they did.

Not only can the code be useful, but the time stamp on the file can tell you when they edited that particular file.  Fortunately, my mistake was not fatal because I found the same problem in the index.php file of every WordPress theme I had installed in my site.  Like many, I had played around with many different themes when first setting up my site and also, like many, I didn’t remove them when I finalized the look of my site.  Don’t ever do that again!  I will talk about it more later, but this was a major security hole in my website.

Jackpot

Like Sherlock Holmes checking under the fingernails of the dead victim, I found an online base64 decoder, and pasted the code into it.  What I discovered was more base64 code.  Again, I copied that in and voila!  There was the javascript code that Google Webmaster tools had been telling me all day was my problem.

This time I made copies of these files and added the file extension .hacked.txt to render them useless, downloaded them to my computer, and deleted both the copy and the original, then uploaded a fresh version of any file that needed it from my computer.

All done and good now, right?  Not quite.  How did my site get hacked in the first place?  All I did was clean up the mess, but the door was still open somewhere, and likely, I knew, they had added their own backdoor.

Closing the Front Door

One of the first things I did when trying to figure this mess out was to shut off all of the plugins in my WordPress site, knowing that they could have been how the hacking occurred.  I didn’t see any evidence of that, so I slowly re-activated some of the more important plugins over the course of a few days.

One item I noted was that all of the unused themes that were installed had their index.php file hacked.  At this point I researched the vulnerability that unused themes and plugins could cause, and it turned out that is a very real problem.  I had several old themes with old versions of Timthumb installed and these older versions of Timthumb have a serious security hole in them.

So I deleted all of the themes from the server except the one I am using and I made sure it was using the newest version of Timthumb.  I also deleted any plugins I wasn’t using and evaluated the security of each plugin by how recently it was updated and if it is actively being maintained.

I also found several folders in my account that I had carelessly left with insecure directory permissions which I promptly changed.

Closing the Back Door

Now I made another mistake.  Having appeased the Google gods that my site was clean, and having removed what I thought had given them access to my website in the first place, I rested.  Like most people I don’t have endless amounts of time to spend hardening my website against hackers.  I had already had one full day and several partial days wasted, and I needed to get back to real work.  Even though I knew I hadn’t likely gotten it all.

Sure enough, within a week I was starting to get the same warnings again.  This time I knew where to look, found the code and removed it.  Now I knew I had a deeper problem, so I delved into error logs and searching for any files that had been changed in the last few months.

Eventually I found two files buried in the WordPress directories of my site.  One I believe was being called by the javascript to do whatever nastiness the hacker intended, and the other was the back door.

Since removing those files, my site has been clean now for almost a month (at the time of this writing).

I hope that in sharing my story it will help others to secure their site or to clean it up in the unfortunate case that it has been hacked.  I also hope to spur some conversation, because I am sure that there are better ways I could have gone about this.  So I welcome your comments and questions below.

In my next article I will go into the technical details of what I did to clean up my site and make it more secure for the future.