Drunkard's Walk

And I come back 90 minutes later, try to refresh the recent changes page, and get an artsy-looking error page from Cloudflare:

Website is offline No cached version of this page is available.
Error 522 Ray ID: 224ee3c9d46c0f21 • 2015-09-12 22:01:38 UTC
Connection timed out

Along with graphics that say Orain's to blame, basically.

Still throwing the same error two and half hours later.

Bob Schroeck wrote:And I come back 90 minutes later, try to refresh the recent changes page, and get an artsy-looking error page from Cloudflare:

Website is offline No cached version of this page is available.
Error 522 Ray ID: 224ee3c9d46c0f21 • 2015-09-12 22:01:38 UTC
Connection timed out

Along with graphics that say Orain's to blame, basically.

And at exactly 10:15 PM EDT it went back down again, with the same error.

Bob Schroeck wrote:And at exactly 10:15 PM EDT it went back down again, with the same error.

That isn't the first time this month one of the wikis has has that sort of problem. Does Orain need more disk?

Actually, we discovered the issue sometime last night: Turns out some douche canoe tried DDoSing Orain again. Thankfully, DigitalOcean wasn't asleep at the wheel half as long and nullrouted the retard not long into their reign of terror, and we found and plugged the leak.

Do note that even though we now have security measures against this sort of crap, doesn't make it impossible if some idiot with a grudge and way too much free time is really determined, but they didn't do even a fraction of the damage they were intending now that we have safeguards.

If it happens again, I'll let you guys know.

Update: It happened again, we're currently identifying the source of the attack so it can be shunted off. In the meantime, bookmark this link:

http://ganglia.orain.org/ganglia/

You can see the memory usage for servers via this. If the values are frightfully high for bandwidth (as in, spiking to the ceiling for no real reason) on multiple servers but fine on other and Orain is down, you'll know we had another attack. FYI, prod 8/9 are currently our MW servers.

Just how bad is the most recent attack? Orain appears to be completely offline now, except for ganglia (which shows zero http activity in the entire cluster in the last hour).

robkelk wrote:Just how bad is the most recent attack? Orain appears to be completely offline now, except for ganglia (which shows zero http activity in the entire cluster in the last hour).

Some idiot seems to be randomly blitzkrieging every port of entry they can find, probably the same douche who had a grudge awhile back.

We're working on making it much harder for them to pull that off, and we're a tad reluctant to get everything back up just yet since we know this idiot is still hiding in the weeds to do it again at the moment.

Is anyone else getting "Certificate Expired" errors or getting redirected to porn sites when trying to access Orain?

robkelk wrote:Is anyone else getting "Certificate Expired" errors or getting redirected to porn sites when trying to access Orain?

Butthurt DDoSer trick #2: When DoS proves not a effective, try to poison the well by spoofing/redirecting the DNS certificate to another site.

As I said, whoever this retard is, they are certainly persistent.

 

I have some horrible news to report: Orain has been epically compromised, as in, raped to an absurd degree.

We were compromised at the server level, and all databases have been wiped. As in  sudo rm -rfed level wiped.

That includes ATT.

The GOOD news is that we have database backups from at least June 15th 2015, but all of Orain is utterly hosed.

I plan to ask for backups of our content as recently as can be obtained for importation to another wiki host, should we want to move elsewhere.

That aside, were still trying to assess the damage, but we definitely will need to pitch tents elsewhere for now.

I am definitely open to ideas guys, and as an Orain Staff member, I'm sorry I was unable to stop this at all.

FYI, JohnLewis is planning throwing us a rope here:

https://meta.miraheze.org/wiki/Miraheze

I'm already in the process of working with John on getting us back up, but you guys, do me a favor:

Scrape every page source you can find off of the Wayback Machine for ATT Orain between June 2015-present as well as images. We have over 89% of the images already, but we'll have to do what ED did after Sherrod DeGrippo killed the original, just not to as severe a degree.

Also, checked, our new host is more stable and has more features for the same (free) price.

HOLY SHIT!!! But who did that, and why?

Another Troper wrote:HOLY SHIT!!! But who did that, and why?

John is already working on getting us set back up:

https://github.com/miraheze/mw-config/issues/108

Sign up for Miraheze here:

https://allthetropes.miraheze.org/wiki/Main_Page

Note; All database info for Orain was wiped, so you'll have to make new accounts here.

GethN7 wrote:I have some horrible news to report: Orain has been epically compromised, as in, raped to an absurd degree.

We were compromised at the server level, and all databases have been wiped. As in  sudo rm -rfed level wiped.

That includes ATT.

The GOOD news is that we have database backups from at least June 15th 2015, but all of Orain is utterly hosed.
...

I really, really hope three's something of the freebie wiki in there - because the Wayback Machine has only one page from my wiki.

If not, then over 2000 pages of content have disappeared.

Fuck.

robkelk wrote:

GethN7 wrote:I have some horrible news to report: Orain has been epically compromised, as in, raped to an absurd degree.

We were compromised at the server level, and all databases have been wiped. As in  sudo rm -rfed level wiped.

That includes ATT.

The GOOD news is that we have database backups from at least June 15th 2015, but all of Orain is utterly hosed.
...

I really, really hope three's something of the freebie wiki in there - because the Wayback Machine has only one page from my wiki.

If not, then over 2000 pages of content have disappeared.

Fuck.

Our backups are three months old? Why the hell weren't they more recent?

And Miraheze gives me a timeout/pageload error. Methinks our asshole is reading this thread.

Bob Schroeck wrote:Our backups are three months old? Why the hell weren't they more recent?

And Miraheze gives me a timeout/pageload error. Methinks our asshole is reading this thread.

It's good, John was just rebooting the servers to add HHVM support when I posted that.

Anyway, if you want to scrape page content, start here:

http://wayback.archive.org/web/20150805 ... /Main_Page

And yeah, I'm pissed too.We can even have some of the features we couldn't have on Orain. I've even asked for periodic backups, told we could have them.

FYI, John has confirmed someone (likely the same guy who murdered Orain) is going after him, but he's got better security and their attempts are getting slapped down hard so far, and he's willing to pour on the money to keep things secure.

Um. Asking the Wayback Machine for all archived pages under http://allthetropes.orain.org/wiki/ gets a page that reports "3,825 URLs have been captured for this domain." We had a whole hell of a lot more pages than that. I filtered the results on various capitalizations of "RWBY" and got nothing at all. Filtering on "RW" gets only three pages:

http://allthetropes.orain.org/wiki/My_N ... ot_Durwood
http://allthetropes.orain.org:80/wiki/Darwinia
https://allthetropes.orain.org/wiki/Air ... scratchers

none of which are newer than November 2014.

The Wayback Machine is going to be mostly useless. Better to raid the Google cache while we can.
Edit:  I so did not need to learn about this today.  I've had a lousy week, a worse day, my lower back, left shoulder and head are all aching, and I am in the foulest mood.  Reading about this makes me want to just throw in the towel, especially if we lost three months' work.  And Miraheze is still throwing a pageload error on me.