Oops, I did not do it [Site down time update]

Posted on May 30th, 2012 in blogging - 8 comments

Site down time - updateSo that was stressful. As many of you know, our main site http://chandoo.org/wp/ was down for good part of last 2 days due to a mysterious PHP error. After trying to fix it for 2 days and drawing a blank every time, I gave up and used my trump card – delete wordpress and manually re-install it. That seemed to do the trick. I can load the site alright, write this post and share the news with you.

I will post something awesome & useful on Excel tomorrow. Wish you a beautiful day ahead.

If you want to what happened & how it is fixed:

  1. 28 may: On May 28th evening, Hui sent me an email saying site is down.
  2. Since I was roaming in Melbourne I could not check the email until I got back to hotel.
  3. I started frantically searching for the “fatal error…” and possible fixes.
  4. Everyone seems to suggest a different solution.
  5. So I called up Pothi, my website admin. Unfortunately, Pothi is in a remote place with no internet access.
  6. So he gave me a few instructions, but nothing useful.
  7. Meanwhile, the error message kept changing. So I thought may be the core wordpress files are corrupt.
  8. Then, went to vaultpress  my automated website backup system to restore the site to an older version.
  9. For some reason that did not work. (although I have hourly backups since middle of last year)
  10. Then I replaced some of the files in wordpress core folders
  11. Suddenly the site started loading again, despite some errors.
  12. So I go to sleep as I had a training program next day (29 may)
  13. 29 may: The site is still working (although with some error messages at top) when I woke up. So I rush to my training program.
  14. But the site stopped loading midway thru the day and I had to time to look at it.
  15. Later in the evening, Once I reach home, I call another friend in India who is good with this stuff and ask his help.
  16. No use.
  17. I replace few more files hoping that would fix it again. No use.
  18. Now I am really sleepy and I have no clue how to solve it. So I shoot some SOS emails to Pothi hoping he can work on it.
  19. 30 may: When I wake up I get a brilliant idea. “why not delete everything and re-install?” Since I have up to date backups of database and all the files I got nothing to loose.
  20. So I quickly pack my bags, check out from my hotel in Melbourne & reach airport by 11 AM.
  21. My flight to Brisbane is at 2PM and I have 2 hours to fix.
  22. I delete both wp-includes & wp-admin folders.
  23. Update them with latest version from wordpress.org
  24. Replace the files in wordpress main folder & theme & plugin folders.
  25. Then run the upgrade script.
  26. And that worked. My site is up. Although some errors are showing.
  27. So I go and upgrade all the plugins. And that fixed the errors.
  28. And I still have 40 mins for the flight. So I thought let me write a 29 point list explaining what happened.
  29. So there you go :D

Thanks for all the lovely emails & concern you have showered while the site is down. I am glad it did not take more time to fix. Wait for something juicy, fun & awesome on Excel tomorrow :)

Yours,
Chandoo

Your email address is safe with us. Our policies

Written by Chandoo

Home: Chandoo.org Main Page
? Doubt: Ask an Excel Question

8 Responses to “Oops, I did not do it [Site down time update]”

  1. kidakaka says:

    Man! That was quite fast!! Good to see the scripts running as required. 

  2. oldchippy says:

    Congratulations Chandoo, one of your many skills besides Excel

  3. Kiev says:

    That’s great to hear, and amazed by your passion & great efforts.
    Really a gene person… thanks.

  4. JP says:

    I thought about getting Vaultpress, looks like I’ll be scrapping that idea.

    You should consider changing your passwords, in case your site was hacked. 

    • Chandoo says:

      I think you should go for it. I will still be paying them monthly fees. The reason why automatic restore failed has something to do with database password & permissions.

      I am not suspecting a hack yet. I think this was due to an old plugin and some other PHP error. But I have changed all the passwords anyways.

      • JP says:

        I get regular XML backups of all pages and posts sent to my email and also to a S3 bucket. The site is customized only through plugins, of which I have backups. So if my whole website disappears I should be able to restore it quickly.

  5. bill says:

    thanks for the blow by blow. i got a bit of unexpected schooling reading this. thanks!

  6. Narinder Kumar says:

    Hi .. .. .  Cong_ _ late chandoo u r gr8 amazing i s….te you.

Leave a Reply