RockyMountainPirate's posts wiped out

Posted by sdesapio 10 years, 1 month ago to The Gulch: General
27 comments | Share | Best of... | Flag

I'll try and make this as clear and concise as possible...

THE SHORT VERSION
This morning, RockyMountainPirate, one of our most revered and active Gulch Producers, had her account and all of her posts deleted. The error was mine alone. There is nothing wrong with the system. We were not hacked. I deleted all of the posts accidentally.

THE LONG VERSION
I went to LA this Wednesday to watch the Director's cut of the film. After watching the film, I was in meetings the rest of Wednesday and Thursday until I went to the airport to catch a late flight back to Texas. I was effectively out of pocket for two days - no time to check emails, etc.

Waking up Friday morning I had a TON of stuff to catch up on; made my coffee, went to work; checked my email and the first thing I saw, still with one eye open, was a flagged comment. I went to check out the comment and sure enough, it was a spammer.

Here's where it gets interesting...

First, you need to know that admins of GGO have special links attached to each comment that no one else has - "Hide", "Edit", "Delete", and the new "Expunge." Expunge is a quick way to wipe out an account and all of its related posts. This is what we use for spammers.

Back to the story... right underneath the spammer's comment, was a comment from our beloved RockyMountainPirate. Both comments of course carried an "Expunge" link. Guess which one I clicked on. Right. Pirate's. The system actually asked "Do you really want to expunge RockyMountainPirate." With one eye open, I clicked "yes"... and immediately proceeded to call Jay.

Me: Jay, I just expunged RockyMountainPirate.
Jay: Oh boy.
Me: Long story. Please tell me we can get everything back.
Jay: Give me a minute.

Jay took a minute and unfortunately, there was no way to recover the account from expungement. All of Rocky's posts were lost for good.

I immediately called and emailed Pirate to tell her the news, give her free lifetime membership to the Gulch and invite her to the Atlas Shrugged 3 World Premiere in Vegas (it doesn't make up for the mistake, but hopefully it takes the edge off of the sting a bit).

So, to be clear, we were not hacked. The mistake was mine. The system worked as designed. We will be adding some further safeguards to ensure nothing like this happens in the future.

My apologies to everyone, ESPECIALLY PIRATE, for the negligence and the resulting loss of great content.


All Comments

  • Posted by SRS66East 10 years ago
    I respect the way this was handled. I am glad to see RMpirate is okay with it as well. So glad to see this was handled openly and the person responsible got out in front of it and took responsibility. Kudos to you Scott.
    Reply | Permalink  
  • Posted by tpatter4 10 years, 1 month ago in reply to this comment.
    Right, we don't see much of it in the current administration either. But that's what taking responsibility looks like. You have to come here to see examples.
    Reply | Permalink  
  • Posted by j_IR1776wg 10 years, 1 month ago
    Wow someone standing up and taking responsibility for their mistakes. Can we get you to run for Congress?
    Reply | Permalink  
  • Posted by Zero 10 years, 1 month ago in reply to this comment.
    Hear, hear.
    I'd expect no less from an Obj, but in this day and age it's downright refreshing.

    RMP - sorry for your (our) loss. It was good stuff. More to come, I'm sure.
    Reply | Permalink  
  • Posted by 10 years, 1 month ago in reply to this comment.
    Because Galt's Gulch Online runs on top of Google App Engine, traditional backups are, in a sense, unnecessary. The data is constantly replicated across multiple servers automatically - so it's essentially impossible for this type of data loss to occur without you going into the system and explicitly deleting it on purpose. Which is basically what happened.

    As a fail safe however, we are looking into implementing an interval based historical backup.
    Reply | Permalink  
  • Posted by 10 years, 1 month ago in reply to this comment.
    I don't want to create any false hope but yes, there is a chance. We're looking into it.
    Reply | Permalink  
  • Posted by Eudaimonia 10 years, 1 month ago
    Hey, Scott, if you need a nightly backup system installed, you have my number.
    Reply | Permalink  
  • Posted by $ stargeezer 10 years, 1 month ago
    For free tickets to the AS3 premier, feel free to wipe me out ANYTIME!

    On a serious note, it's nice to be onboard with first rate folks like these here. Accidents happen and not trying to speak for Pirate, but knowing that the fist move you made after discovering your error was fessing up to the mistake and moving to makeup for the error. Very well done.
    Reply | Permalink  
  • Posted by $ AJAshinoff 10 years, 1 month ago
    Class act plainly stating and owning up for what happened. Its much appreciated. In the future please feel free to accidentally wipe all of my post if it gets me an invite the World Premiere in Vegas :) j/k
    Reply | Permalink  
  • Posted by conscious1978 10 years, 1 month ago
    Any chance you could pull the old posts from Google cache? Some of them are visible there that are up to a year old.
    Reply | Permalink  
  • Posted by $ rockymountainpirate 10 years, 1 month ago
    Thank you for the explanation Scott. Apology accepted.

    Stuff happens and I thank you for quickly contacting me and letting me know. I'll get busy finding more things to stimulate discussions.
    Reply | Permalink  
  • Posted by khalling 10 years, 1 month ago
    I know you have been dealing with those spammers alot. it's a real problem. sorry this happened. but it could happen and thanks for being vigilent. but poor pirate. ;( that's one hell of a super signed poster and DVD and J.A. handshake...
    Reply | Permalink  
  • Posted by LetsShrug 10 years, 1 month ago
    From now on your name is Scott Expunge Desapio. Expunge for short. Thanks for the explanation. I will mourn my lost comments now. Sniff.
    Reply | Permalink  

  • Comment hidden. Undo