WordPress: Freaky WordPress Catastrophes… And How To Prevent Them

WordPress: Freaky WordPress Catastrophes… And How To Prevent Them

Sep 08

Freaky WordPress Catastrophes And Site Crashes, And How To Prevent Them
By Johnny B. Truant
September 3, 2010

Original Link

[The comments this post generated also produced some helpful suggestions. Visit the link above to read them. –DS]

You should pay attention to this post. It may be a little drier than I sometimes am, but I’m going to tell you how to cover your ass in the event of a website mishap. I learned through experience and figured I’d share.

So here’s what happened.

Last Wednesday, I decided to sleep in (which is rare), and Robin came into the bedroom after I’d laid around enough and said, “Apparently your site is down. You’ve got all these emails like, ‘Dude, what’s wrong with your site?’”

Now, I’m not prone to freakouts, so I took my time getting in to the office, turned on my screen, and went to my inbox. And it was indeed filled with “Dude, what’s wrong with your site?” emails.

So I went to my site. Which looked great, except that I got that “404 – Page not found” error on the front page, and didn’t see any posts, or any of my sidebar stuff. I logged in and saw that I currently had zero posts, zero pages, and zero comments.

As of the time I’m writing this, I actually have 131 posts, 90 pages, and 2145 comments. Those stats have changed some since last week, but even back then, I was able to notice that zero posts/pages/comments left a discrepancy.

(I should pause here to describe the sinking, panicked feeling I had at this point, but I honestly felt nothing. I’d wager that some of you reading this right now feel more sinking and more panicked that it could happen to your blog than I did. And it totally could. I don’t know, maybe I’m just confident or something, but it was strangely bland.)

It was cool, though, because I’d been doing the right things. I was prepared for something like this.

What are the first and second rules of computing?

The first rule of computing is to back up your files.

The second rule of computing is to back up your files.

Eventually, if you stay in this world long enough, someone will make a Fight Club reference and mention the First and Second Rules, and nerds will start pulling out flash drives.

So the good news was that I had been doing the backup thing… which was fortunate, because I found out later that thanks to some technicality, my webhost had not been making the backups they promised. So if you aren’t backing stuff up on some level right now, there’s a chance that your webhost is doing (or not doing) the same thing, and you’ll have a crash and be totally and completely fucked.

The good news is that I’d been backing up.

But the bad news is that you never know if a backup plan works until something bad happens and you need to restore. And in my case, what I found out was that the backup itself was completely fucked.

When I went in to the backup console and clicked the little “restore” button for my last backup, it just kind of kept working and kept working but never finished. I tried a manual restore with the same result — totally fucked.

A handful of posts and pages showed up. I think around 10 of each. It was nice to have some of my content back, but the other 121 posts? Gone. And that wasn’t the worst part, because for most people, old posts are old news anyway. Technically, I could repost (out of order, randomly) and reformat them based on the original text files, which I had.

But all of the stuff missing from the sidebars? And the other 80 pages? I didn’t even know what those pages were. I couldn’t recreate most of them if I had to. Few of them were linked from the main page; they were pages I referred to for pre-sale or post-sale — things like my 10-step process for clients to get hosting the correct way. Sales pages I had painstakingly written. Ugh. The thought of recreating all of that or losing it was not cool.

To attempt to keep a long story shorter, the problem was that my backup file was somehow corrupt. Andrew Norcross diagnosed it as some fuckery that occurred when I moved from LearnToBeYourOwnVA.com to JohnnyBTruant.com wherein I was kind of somehow pulling content from two different databases. (Don’t ask; I’m not a database guy and don’t understand it myself.) He said that the database and the calls it made for content just got so muddled up that the whole thing kind of died a little, and then died a lot.

Andrew meticulously rebuilt my database from the corrupt backup files, and then I reposted the recent post I had made and my assistant Amy re-added the comments people had made in the time between that most recent backup and the crash. I made a few more tweaks and in under 36 hours, I was back to 100%.

Back to 100%, but also really paranoid.

So here’s the bulletproof stuff I’ve put into place, with Andrew’s advice. It’s a bit of overkill, but I feel super-secure now that I’m protected going forward.

The Bei Fen backup plugin: This is what I had before — the backup measure that allowed my site to rise from the dead, but which also gave me a screwed backup. I’ve decided that the screwed backup wasn’t the fault of this plugin, but it was that “two databases, overwrought” thing that caused that mess.

When I was originally looking for backup plugins, Bei Fen was the only one I found that would make a COMPLETE backup of the site (database PLUS all of your static pages, media, images, etc.) and not insist on emailing the giant resulting monstrosity to you. It puts the backup in a directory on your site. Upside? Quick and easy and convenient. Downside? It’s ON YOUR SITE. If the whole site gets corrupted by malware or something, then there goes your backup.

But I use it anyway still, as a catch-all. I have it make a new compete backup every Wednesday and replace the old backup (otherwise the space consumed gets HUGE). I moved my weekly backup to Wednesday because I found out that my webhost makes backups on Sundays. So I verified that they would keep making them, and between the two, my most recent complete backup will always only be a few days old.

(The files on my computer also function as a second but imperfect backup of files, media, etc… everything but the database.)

The WordPress Database Backup plugin: As it sounds, this one makes a copy of the database only — but that’s arguably the most important part of a WordPress site. It will email you the database, so I set up a new email account using Google Apps (which is what runs my main email address) and have the databases mailed to that account daily. This means that the database backup is being stored off-site, and it’ll archive automatically… the databases will just accumulate in that email account until I go in and clear out the old ones.

Two considerations here: If the database is all fucked up and overburdened like mine was, it may not backup quite right and/or be corrupt. Also, if the database gets too big, your server won’t email it out. It’ll say, “No way, dude… attachment too big. Fuck off.” And it won’t tell you that it’s refusing to send them, and you’ll find out when you need one and your most recent one is from six months ago.

Both of those problems can be solved by this next piece:

The WP-Optimize plugin: This one adds an area on your dashboard where you can go in and clear out the junk cluttering up your database, thus making it function more cleanly and keeping its size down.

Two things that will clutter you up like a bastard are spam comments and post revisions. I knew about spam, but the revisions thing took me by surprise. Apparently, by default, WordPress saves every single revision you make to a post, and those revisions NEVER GO AWAY. Go in and change “a” to “an” in a certain sentence? You’ve just created a post revision, and it takes up as much space in the database as the actual published page. Thanks to post revisions, I believe my database was 10-15 times as large as it should have been.

You have to actually remember to use WP-Optimize to clear the crap out, though, so don’t forget. Andrew suggested doing it weekly.

The three plugins above would have addressed my issues if the same thing happened in the future, but then there’s hacking and malware to worry about too. I hadn’t gotten hit by those, but didn’t really want to wait and find out what it was like. So I added this other stuff too:

The Login Lockdown plugin: You can eventually break a password if you just keep trying until you get it. To combat this kind of “brute force” attack, what this plugin does is to allow someone to try only a specified number of login attempts before locking them out of the system for a specified period of time.

The Secure WordPress plugin: There area bunch of geeky ways that hackers can get at your site and that I don’t totally understand. By making some changes to the way WordPress presents itself (doesn’t show the version number which would indicate version-specific weaknesses, removes certain error messages that give hackers tips on what they’re doing right and wrong), you can lock your site down even further. This plugin lets you control and change those weaknesses.

Sucuri malware protection service: I did this one purely on the recommendation of Tony Clark, who’s the Copyblogger Clark that doesn’t come out in public as often as that Brian guy. (And no, they’re not related.) Earlier this year, Copyblogger had a huge malware infestation and subsequently got blacklisted by Google. Sucuri got them out of it and got them back on Google’s good side. What’s good enough for Copyblogger is good enough for me, so I signed up. It’s like $90 a year.

Basically, Sucuri monitors your site and looks for unexpected or nutty software, activity, database calls, and other stuff that I don’t understand. And it lets you know when something’s fishy, and gets you back up and running if bad things go down.

Based on what Tony told me, hackers can get in through your webhost sometimes, which means that the stuff that you can do to protect your site can end up being pretty irrelevant — you’re barricading the front door while they sneak in through the back. Sucuri covers your ass by watching all of your doors.

So now, I feel a lot safer. And given that all but Sucuri above are free, you really should do this stuff yourself. Because having a website problem sucks giant balls.

Be safe, y’all.


For more information on keeping your WordPress website/blog secure, go here.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.