Oops, I did not do it [Site down time update]

So that was stressful. As many of you know, our main site http://chandoo.org/wp/ was down for good part of last 2 days due to a mysterious PHP error. After trying to fix it for 2 days and drawing a blank every time, I gave up and used my trump card – delete wordpress and manually re-install it. That seemed to do the trick. I can load the site alright, write this post and share the news with you.

I will post something awesome & useful on Excel tomorrow. Wish you a beautiful day ahead.

If you want to what happened & how it is fixed:

1. 28 may: On May 28th evening, Hui sent me an email saying site is down.
2. Since I was roaming in Melbourne I could not check the email until I got back to hotel.
3. I started frantically searching for the “fatal error…” and possible fixes.
4. Everyone seems to suggest a different solution.
5. So I called up Pothi, my website admin. Unfortunately, Pothi is in a remote place with no internet access.
6. So he gave me a few instructions, but nothing useful.
7. Meanwhile, the error message kept changing. So I thought may be the core wordpress files are corrupt.
8. Then, went to vaultpress  my automated website backup system to restore the site to an older version.
9. For some reason that did not work. (although I have hourly backups since middle of last year)
10. Then I replaced some of the files in wordpress core folders
11. Suddenly the site started loading again, despite some errors.
12. So I go to sleep as I had a training program next day (29 may)
13. 29 may: The site is still working (although with some error messages at top) when I woke up. So I rush to my training program.
14. But the site stopped loading midway thru the day and I had to time to look at it.
15. Later in the evening, Once I reach home, I call another friend in India who is good with this stuff and ask his help.
16. No use.
17. I replace few more files hoping that would fix it again. No use.
18. Now I am really sleepy and I have no clue how to solve it. So I shoot some SOS emails to Pothi hoping he can work on it.
19. 30 may: When I wake up I get a brilliant idea. “why not delete everything and re-install?” Since I have up to date backups of database and all the files I got nothing to loose.
20. So I quickly pack my bags, check out from my hotel in Melbourne & reach airport by 11 AM.
21. My flight to Brisbane is at 2PM and I have 2 hours to fix.
22. I delete both wp-includes & wp-admin folders.
23. Update them with latest version from wordpress.org
24. Replace the files in wordpress main folder & theme & plugin folders.
25. Then run the upgrade script.
26. And that worked. My site is up. Although some errors are showing.
27. So I go and upgrade all the plugins. And that fixed the errors.
28. And I still have 40 mins for the flight. So I thought let me write a 29 point list explaining what happened.
29. So there you go 😀

Thanks for all the lovely emails & concern you have showered while the site is down. I am glad it did not take more time to fix. Wait for something juicy, fun & awesome on Excel tomorrow 🙂

Yours,
Chandoo