6 Hours and 60 Minutes. That’s how long my blogs were down flat this morning/afternoon. Perhaps my math is a little rusty, but I still thought that an hour was made up of 60 minutes. So rather than 6 Hours and 60 Minutes, maybe Pindgom should be telling me that my blogs were flat for 7 hours? That’s just my opinion.
So last night I was relatively active on my blogs. I replied to a bunch of comments, and published 3 posts on WhatAboutStephen, with a 4th scheduled for later today. I worked on TheUnfriendlyGiant.com, changing the layout, configuring Thesis, adding a few plugins, and starting to add some content. All of this was taking place while having a Twitter-meltdown of epic proportions. It was not a fun night. But I was being more productive than I’ve been in a few days.
Then around 5:30am I tried to save some work on TheUnfriendlyGiant.com and I received an error saying that it could not connect to the database. I tried again, with the same result. Then I tried this blog, and was unable to connect. Same thing with the others. Not good. It didn’t feel like another account problem. It felt more like a technical problem. I confirmed that by successfully connecting to Write-ClickMedia.com which does not run on WordPress. Obviously my problem was related to my WordPress installations, which run on a database on a shared hosting server, courtesy of PowWeb. I checked my hosting provider’s website to see if they had posted any planned work maintenance. 5:30am here means that it’s even earlier throughout the rest of North America, making it the perfect time for a planned maintenance schedule. There was indication of that on their website.
By this time I had received an email from a service called Pingdom, which I only recently subscribed to. They automatically “ping” your website at predetermined intervals, and alert you if they are unable to reach it successfully. The internet isn’t perfect. Things happen. A certain amount of downtime is understandable, but Pingdom is there to alert you in the event of something major. It’s free to install on a single website, so I chose my most active one, WaS. Because this isn’t a revenue-generating blog (damn), it’s not exactly mission critical to my online presence. I have the Pingdom interval set to 30 mins before emailing or texting me an alert. If I check the logs for the month, I can see that there have been instances of missed pings, but this is the first time that they have been required to contact me.
As I finished my shift, all I could care about was getting out of there and getting home. It had been a miserable night, and I wasn’t in a mood to submit a support ticket to PowWeb. Sleep was my only priority, and if the blogs were still down when I woke, then I would deal with it. Unfortunately, after a few hours of restless sleep I woke up briefly to check my iPhone, and discovered that nothing had changed. I decided that I’d better call PowWeb’s support line and find out what was going on. As soon as I heard “We are currently experiencing higher than normal call volumes. Please wait for the next available representative.” I knew that it was something much larger than just me. To be honest, I took some comfort in that. In my mind, it meant that they were probably already aware of the outage and working on it. If the problem had been localized to me solely, chances are that it would have been pushed down their list of priorities, as they worked on other trouble tickets in their queue. That’s normally how these things work, and it’s generally a pretty good system.
Considering the fact that they were warning me that I might be on hold for awhile, a customer service representative was there in under 2 minutes! I wasn’t quite ready for that. I explained the problem to the gentleman, who informed me that the support engineers were working on a database failure, and that it should be resolved soon. I didn’t even bother to ask what “soon” meant because I know that he either didn’t know, or he wouldn’t tell me if he did. It’s best to keep the customers in the dark about these things. I was fine with that. I just rolled over and went back to sleep.
The first thing I did when I woke again was check this blog. It worked! Whohooo! I checked the others. Everything looked good. There was even an email from Pingdom alerting me that this website was back in service, and had it been down for a total of 6 hours and 60 minutes. Then it occurred to me… the last post showing as published was Good Ole CAA. Where were the posts that I had published after that? There were 3 of them… plus the one that was scheduled and should have been published through the day. Obviously PowWeb had restored my blogs with a recent backup copy of their database. That sucked, but was not unexpected. The main thing was that the sites were restored… even if TheUnfriendlyGiant.com had lost all the work that I’d put into it.
I hate when I’m writing something, and it gets lost. If I have to re-write it, it’s never as good the second time. It happens from time to time with emails, or the occasional blog post that gets deleted as I’m writing it. There’s something about taking the time to write something again.. it’s just never as good. I wasn’t enjoying the prospect of having to re-type 4 blog posts. In fact, I really considered just letting it go. There’s never really anything important here anyway, and in the case of 2 of those posts, I knew that they had already been read by the person they were intended for.
Then I remembered something. Google RSS Reader! I’ve been adding friends’ blogs to my Google Reader page, and syncing them with my iMac and iPhone. I had also added all my Write-Click Media blogs. Why? I have no idea.. just to see how they looked when presented through RSS as opposed to directly on the website. I wondered if they would still be there in Google Reader… WHOHOOO! 3 of the 4 were there. The last was never published, so Google Reader never had a chance to capture it. At the moment I can’t even remember what it was about, so it might be lost forever.
The point is, I don’t have to re-write them from scratch. I am going to go back to Google Reader and simply copy and paste them in, and re-publish them. It doesn’t help me with TheUnfriendlyGiant.com because those were style changes, not content. And I’ll have to go back and reply again to those people who had commented on this blog. But that’s a pretty quick process.
Let this be a reminder to everyone… whether you’re concerned about your own personal files, or running a company responsible for tons of information… BACKUP BACKUP BACKUP!