Manc Posted July 8, 2009 Primary hard disk failure on MancuNET. They're going to replace it within 24 hours, but things may be unavailable for the next few days as I will have to go through and set everything up again. Thankfully there's no charge for this (except for my time and frustration) and ServerBeach are quick and professional (as opposed to dreamhost who are a joke and terrible). Will keep this thread updated with any new information. Yes data are backed up. [UPDATE 07/08/09 3:11PM] The drive is being replaced during a 4 hour window of 8pm-11:59pm CDT (so starting within about 5 hours). Starting tomorrow I will be going through the process of setting back up all the crap. Thankfully I got this thing with a spare drive in it I was not yet using, so I partitioned, formatted and now I have a spare drive to copy things to. Makes backing up much faster! [UPDATE 07/08/09 5:35PM] Attempting to do mysqldump does not fully dump before throwing itself into read-only mode (from warm boot). I've backed up the actual database files and dirs themselves, which should restore fine. I used the packaged version (lenny) of mysql 5 and I will be reinstalling the same version most likely. Nothing too critical will be lost with these dbs but I am doing my best to ensure they're still alive. ServerBeach continues to be fantastic in their support. I will not be near a computer until tomorrow morning. When I return to a computer I should have waiting for me a freshly provisioned server. Thankfully lessons were learned from the errors I was having before in my initial setup so things should go more smoothly this time. The biggest issue tends to be permissions related and besides lessons learned, everything has been backed up with that information intact. [UPDATE 07/09/09 10:08AM] The server is back online, but it only has the new drive, not my second backup drive. They were testing to see if I could log in before putting in my other drive. However, their language seemed to indicate that they'd be putting in my original disk back in so I could copy off of it. EDIT: Spoke with them on the phone, they did indeed mean putting back in my other healthy drive. It's happening as I type this. [UPDATE 07/09/09 10:55AM] All drives are in, fresh /dev/sda, backup /dev/sdb and fartknocker brokendrive on /dev/sdc. Will be doing software rebuild over the course of the afternoon/evening starting in a few hours. Most if not all services will be restored in the next 12-24 hours. Thank you everyone for your continued patience and support. [UPDATE 07/10/09 8:26AM] Most everything is back to normal. Email is not set up yet but that is a separate process. All accounts should be restored, passwords are the same, all subversion repositories are restored and websites are restored. Please note: things may still be a little intermittent as I stop/restart different services during the remainder of the setup process. If you are finding that something is missing, I will still be checking these threads regularly. You can also send an email to my offsite email address (fuzztooth@gmail.com). 0 Share this post Link to post
Khorus Posted July 8, 2009 Bugger. :( But good to hear that data wasn't lost. 0 Share this post Link to post
exp(x) Posted July 9, 2009 Don't most large scale servers have hot-swappable drive arrays with redundancy? Or are you just using them as a colocation facility for your own server? 0 Share this post Link to post
Enjay Posted July 9, 2009 Aha! That would explain why I've not been able to access it. Thanks for letting people know. 0 Share this post Link to post
Scuba Steve Posted July 9, 2009 DOWN TIME IS LOST URBAN BRAWL PROFITS! 0 Share this post Link to post
DuckReconMajor Posted July 9, 2009 I was gonna buy 1,000 copies of Action Doom 2 but the site was down. Oh well. 0 Share this post Link to post
HackNeyed Posted July 9, 2009 Wait, what? I just bought AD2:UB last night... I hope this doesn't impact the arrival of my order of pure awesome in a can of a box sewn and stitched by blind elderly orphans who also love awesome! 0 Share this post Link to post
Creaphis Posted July 9, 2009 As soon as I saw Manc's avatar wouldn't load I knew some shit was going down. 0 Share this post Link to post
GreyGhost Posted July 9, 2009 I hope that drive failure wasn't the result of having too many fish avatars on the one page. 0 Share this post Link to post
leileilol Posted July 9, 2009 I saw it start happening around 6:35AM CST when I tried to approve members of my board, then KAPOW sessions table corrupt, and BAM can't go into webadmin, and DOOSH 404 forever. 0 Share this post Link to post
Manc Posted July 9, 2009 exp(x) said:Don't most large scale servers have hot-swappable drive arrays with redundancy? Or are you just using them as a colocation facility for your own server? Yes and if I had a large scale server setup I might have that. But that costs money, and my setup is one machine just for me. I have two drives but they're not set up for any redundancy at this time. 0 Share this post Link to post
Manc Posted July 9, 2009 All drives are in, fresh /dev/sda, backup /dev/sdb and fartknocker brokendrive on /dev/sdc. Will be doing software rebuild over the course of the afternoon/evening starting in a few hours. Most if not all services will be restored in the next 12-24 hours. Thank you everyone for your continued patience and support. 0 Share this post Link to post
Manc Posted July 10, 2009 Most everything is back to normal. Email is not set up yet but that is a separate process. All accounts should be restored, passwords are the same, all subversion repositories are restored and websites are restored. Please note: things may still be a little intermittent as I stop/restart different services during the remainder of the setup process. If you are finding that something is missing, I will still be checking these threads regularly. You can also send an email to my offsite email address (fuzztooth@gmail.com). GreyGhost said:I hope that drive failure wasn't the result of having too many fish avatars on the one page. More like there's not enough! Now that it's back up I can finally update the gallery! Scuba Steve said:DOWN TIME IS LOST URBAN BRAWL PROFITS! There's profits?? 0 Share this post Link to post
Scuba Steve Posted July 10, 2009 OK, so I pretty much just break even... but nonetheless... 0 Share this post Link to post
kristus Posted July 11, 2009 Don't worry Scuba. I'll make you rich. *Orders a copy* 0 Share this post Link to post