Anime Central Forums: What The Heck Happened To Reg? - Anime Central Forums

Jump to content

  • 2 Pages +
  • 1
  • 2
  • You cannot start a new topic
  • This topic is locked

What The Heck Happened To Reg? Rosa's "brief" explanation.

#1 User is offline   sisterdiscord 

  • Ace
  • PipPip
  • Group: Members
  • Member No.: 1559
  • Posts: 380
  • Joined: 03-February 04

Posted 20 May 2008 - 10:14 AM

First, guys, I just want to extend my apologies to all of you that experienced the lines problem in registration this year. I know you're unhappy with it (and I realize that unhappy is an understatement) We didn't intend or want this to happen, and we did take a lot of measures to try and prevent it, and to try and correct it. I'm not going to offer excuses--it was wholly unacceptable, and nothing I can say will change that fact. I wish it could.

What I do feel you all deserve is an explanation of what was happening, and what we were and are still doing to try and counteract it and correct for next year.
To clear up a rumor I've heard from more than one person-- no, our registration database was not hacked. Here's what I know as of right now: beginning about an hour after the reg line opened on Friday, our servers were subjected to a distributed denial-of-service (DDOS) attack. Basically, in an attack like this, there are a number of computers in multiple locations spamming the servers with so many requests that they couldn't respond to everything, and began to 'drop' traffic. Dropping traffic first caused us to slow down, and backed up our badge print queue, and then ultimately completely squashed our ability to get to the registration system, which was the same system we used for online reg.

When the web registration tool at con had issues, we moved to our backup plan. Offline data was supposed to be used, and we had a network connection through the convention center for processing credit cards. Unfortunately, thousands of Otaku brought thousands of laptops, and my theory is that the convention center's whole network was under a much heavier-than-normal load because of it. In any case, the network connection to and through RES was stable on Thursday night, but for Friday and a portion of Saturday we experienced a lot of network instability.

Our third fall-back plan, manual badges and manual payments, was put in place, but not as quickly as any of us would have liked on Friday. At that point, we had the lines moving, though still not quickly enough, because the manual process was slow and somewhat cumbersome. We also discovered Thursday that the convention center had not understood the floorset we provided to them, and the lines were not oriented as we had planned--this created bottlenecks at badge claim.

Friday night the coding team worked the overnight hours (Thanks Riker, thanks Eli!) and IT switched off the ACEN server and made some processing changes to speed up and streamline things so that we didn't have to hit anything at our server site to process registrations. We also purchased wireless modems from our cellular provider for several of the computers as a 'Plan Four', so that even if the network wasn't stable we could get on to run cards and manage payments. Those became very useful on Saturday as it became clear the network wasn't being our friend.

There was also vandalism late Friday night to some of our registration computers and network equipment, and because of this Saturday we were more limited in the number of stations we had available. I don't excuse this kind of behavior, but I actually can understand where it came from, because there was an immense amount of frustration and anger at having to sit in that line interminably. That said, if any of you who were in the line near the end of the night saw anything and want to talk privately about it, I'll be happy to listen, because the damage sure didn't help.

All summarized, there is a massive amount fail in the paragraphs above, some internal, some external. We're working things through right now to figure out where we can fix what was broken, and how we can correct and contingency plan for next year. Any and all suggestions you guys have are welcome--we have quite a few of our own, and if any of you are nerd-trained like myself, we're always looking for extra folks for our tech team if you're interested in helping us make it better. We know it can be much better, and quite frankly, it has to be better. We don't want to ever put you guys through anything like that again.


We still love you guys, and we know some of you are very upset. If anyone wants to talk, feel free to reach out to me. Right now, I'm going to go get some more sleep.

Rosa.
This is my signature. It is only one line.

#2 User is offline   animefan1977 

  • Bigfoot
  • Pip
  • Group: Bigfoot
  • Member No.: 4518
  • Posts: 63
  • Joined: 09-May 05
  • Location:IL, Southern 'Burbs of Cook County

Posted 20 May 2008 - 10:53 AM

View Postsisterdiscord, on May 20 2008, 11:14 AM, said:

There was also vandalism late Friday night to some of our registration computers and network equipment, and because of this Saturday we were more limited in the number of stations we had available.



What the hell? WHAT THE FREAKIN' HELL! :mad:

I am about to sound like my mother when I say this but I have to say it. What the hell is wrong with kids these days? What are you thinking? I AM ANGRY GRRR OTAKU SMAAASH!!! ???????

YEAH, that is going to make registration run smoother.

Thanks for the explanation Sister Discord I hope that things go better next year. I have attended Acen since 1998 and it is a fun con.

#3 User is offline   this_chick25 

  • Registration Staff
  • PipPipPipPip
  • Group: Moderator
  • Member No.: 7489
  • Posts: 5,831
  • Joined: 25-June 06
  • Location:Wherever I am, I'm working on my cosplay...

Posted 20 May 2008 - 11:02 AM

Sisterdiscord, I loooooove you.

So we did get DDoS'ed. Grrr... jerks.
Acen Attendee since 2001
Wondering how much to save for ACen? Click here!
FREE! Cosplay construction and materials advice--get yours now!
Never cause more mischief than that which is necessary for success.
Sig invade'd'd'd'd'd'd'd'd'd by Alk <3

#4 User is offline   Caerulea Windseeker 

  • Lurker
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 18564
  • Posts: 45
  • Joined: 19-May 08

Posted 20 May 2008 - 11:44 AM

Rosa, I am sure that a lot of people appreciate the explanation. I wish ACen and M.A.P.S. the best of luck in what I've heard will be an F.B.I. investigation of the attack.

I was wondering if you'd gotten a chance to peruse the thread in the Suggestions/Gripes forum, linked here: http://www.acen.org/...showtopic=23262

To sum it up, lines are always going to be a fact of life at conventions. Even if Registration goes off without a hitch, as others have pointed out, there are just a lot of people who want to get into this con. Wedge, the ADH of I.R.T. has stated that it is not ACen's policy to let people back in line if they get out of it. However, that policy is completely impractical and poses a great health risk to anyone who is standing in line for long stretches of time, as they were for Registration and the Soap Bubble. There are many guests with medical needs, whether they appear to have those needs or not. There is also the hard and fast reality that everyone, regardless of their health status, needs to stay hydrated and use the washrooms, or they will be sick. I don't want to see people hurt, nor do I want to see the con canceled due to someone getting hurt and the con getting into trouble for it.

Some have suggested reporting individual I.R.T. folks who didn't let people back into the lines or who were unduly rude about it. However, I do not want to report I.R.T. folks who are caught between a rock (the policy stated by Wedge) and a hard place (the needs of the con-goers). It is not I.R.T.'s fault that the policies are completely out of step with reality.

So what I'm looking for is an apology to those who were sick in lines because they were faced with the decision to either get out of line and have to spend yet more hours at the back of the line, or stop eating/drinking/using the restroom and got sick as a result. I'm also looking for a commitment from the highest levels of ACen leadership to come up with a sensible, practical solution to this problem that will help reduce the stress and danger to both con-goers and I.R.T. Finally, I am looking for the new policy to be published so that con-goers have something in writing to defend themselves when staff implements the policies in an inconsistent manner.

Thanks for any answers you can provide.

#5 User is offline   sisterdiscord 

  • Ace
  • PipPip
  • Group: Members
  • Member No.: 1559
  • Posts: 380
  • Joined: 03-February 04

Posted 20 May 2008 - 12:30 PM

View PostCaerulea Windseeker, on May 20 2008, 05:44 PM, said:

I was wondering if you'd gotten a chance to peruse the thread in the Suggestions/Gripes forum, linked here: http://www.acen.org/...showtopic=23262


I'm looking at it right now, and will reply over there in a bit. Still recovering, and have come down with the ever popular Con Funk since I got home, so I'm not yet operating at 100 percent.


View PostCaerulea Windseeker, on May 20 2008, 05:44 PM, said:

So what I'm looking for is an apology to those who were sick in lines because they were faced with the decision to either get out of line and have to spend yet more hours at the back of the line, or stop eating/drinking/using the restroom and got sick as a result. I'm also looking for a commitment from the highest levels of ACen leadership to come up with a sensible, practical solution to this problem that will help reduce the stress and danger to both con-goers and I.R.T. Finally, I am looking for the new policy to be published so that con-goers have something in writing to defend themselves when staff implements the policies in an inconsistent manner. Thanks for any answers you can provide.


Actually, when we recognized that the lines were out of bounds that policy was gone. We were allowing folks to step out of line and back into line, to get food, to run to the bathroom, and so on. I think that the notice of that went out on channels a few dozen times, but not every staff member has a radio, and it's possible folks were applying old rules out of habit. For the inconsistency, I do apologize, and it can, will, and frankly MUST be addressed. You have my personal commitment that this will happen, and since Momma Rosa has no plan on retiring after this year, you can bet your Keyblade that it'll get the attention it deserves.

We surely didn't want anyone to be sick or ill, we stationed EMRT staff nearby just in case, and we were actively pulling and seeing to people where we could if they didn't look well. Part of the problem is that in that mass of people, it was hard to spot someone not feeling well, unless they or someone near them told us about it. Especially in cosplay/masks/facepaint/dressed like a zombie.

We did have problems with people trying to cut into line, and IRT was handling those pretty seriously, so if no one communicated "Hey, gotta potty and grab a dog, be back in a few.." then it's possible that's where some of the issues came from. If you or any of yours experienced treatment that wasn't appropriate, I will certainly extend an apology, and do my best to correct things.

We do everything we can to make sure ACen is a safe and fun experience every year. Like lots of folks have said on this thread, we're all volunteers, and sometimes communication is our big stumbling point. I will add some additional basic IRT/line control and courtesy education to the menu for all staff who come on board or stay for next year, and we'll make a run at coming up with a way to identify people who're stepping out of line--maybe a wrist band, or a hubcap on a string or something.

We love you guys. We want to get this right, and we know it's not going to be perfect, but it can certainly be much less imperfect.
This is my signature. It is only one line.

#6 User is offline   Sexeh Cloud 

  • Newbie
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 6984
  • Posts: 6
  • Joined: 01-May 06

Posted 20 May 2008 - 12:34 PM

wow, that's actually a pretty sound and reasonable explanation. Yes, it was ridiculously frustrating, and as I stated elsewhere, I didn't even get the worst of it. I posted a pretty long gripe about this in the gripes section before I had been able to find this particular thread, and it helps a lot.

Here's to hoping you guys find a suitable resolution to the registration problems NEXT year.

#7 User is offline   Caerulea Windseeker 

  • Lurker
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 18564
  • Posts: 45
  • Joined: 19-May 08

Posted 20 May 2008 - 05:04 PM

I'd like to thank you, Rosa, and also AngelMercury for giving responses that recognize the infuriating way the lines were run without throwing either staff members or con-goers under the bus. Your response here is very helpful and gives me hope that next year will not have the health problems that many folks saw this year.

I'd just like to say that whatever is done to help ensure peoples' basic needs are met while in the Registration line ought to be implemented in all lines. I like how you phrased it--line control education. Lines happen at cons; they are a fact of life and I think people expect some queuing when they come to a convention. How the lines are handled makes a big difference in how acceptable they are to people. So long as we get new line policies that are easier, more realistic, fairer to both the volunteers and the con attendees, and implemented for all events, I am very happy.

I feel much, much, much better after reading Rosa's and AngelMercury's posts. ^_^

#8 User is offline   angrypata 

  • Bigfoot
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 10308
  • Posts: 89
  • Joined: 11-April 07
  • Location:port huron

Posted 20 May 2008 - 08:59 PM

Yes. I've known that since Friday. Pretty sure 420ch/4ch (there's a lot of overlap, especially among the anon that are capable of that kind of thing) had something to do with it, and YOU KNEW THAT. It had ALREADY BEEN ALLUDED TO ON THE FORUMS--YES, THESE FORUMS.

You had time to deal with this, and you didn't. You had time to make backup preparations for something like a denial of service attack (which should have been in place ANYWAYS for ANY large-scale realtime operation that involves online anything) and you didn't. You kept the entire line in the dark about everything that happened, and then only gave a five-dollar discount (because oh yeah, we missed Friday entirely) to I don't know how many people, and as far as I know, that was only after I grabbed the finance manager and gave him a...friendly talking-to. On the forums you claimed there would be at-con reg on Thursday night. My party (ten people) left a day early to take advantage of this option, and got there only to find out that there WAS no at-con reg, only pre-reg. Thanks, guys. I missed another day of work and paid for another night's stay for no reason whatsoever.

If you people cannot handle what this convention has become, you need to hand over certain tasks to professionals. Two of those tasks are security and registration, both of which became serious issues during the marathon reg wait. Otakon is a convention that eclipses ACen in every way, from attendance to events to guests, and I have never waited for more than three hours in their registration line in the last five years, even with the line wrapped several times around the Baltimore Convention Center.

That was pathetic, ACen. You screwed it up last year, you screwed it up this year, and I'm certain you'll have some woeful explanation for the newest crisis when you screw it up again next year.

You need to fix this from the ground up, or you're going to collapse under the weight of your own hubris.

IN SHORT:

1. HIRE SOME PROFESSIONALS.

2. HAVE A BACKUP PLAN.

3. PAY ATTENTION. YOU ARE FULLY AWARE THINGS LIKE THE CH EXIST.

PS. If it WAS you, Anon, well played. It was a perfect storm.
Thanks to Time Mage, we did Super-Job! This is the way!

psst, i'm over here

#9 User is offline   Lenne 

  • ACen Staff
  • PipPipPip
  • Group: ACen Staff
  • Member No.: 425
  • Posts: 642
  • Joined: 15-May 03
  • Location:Brookfield

Posted 20 May 2008 - 09:06 PM

The post I saw on the forums about 4chan morons possibly trying something (which I only just saw today) was posted only a couple days before the con. A lot of the senior staff goes offline a few days before the con officially starts because of travel and other preparation and setup that must be done.
♥ Alana ♥
Table Top Gaming ADH ACen 2008 - 2010
Table Top Gaming Staff ACen 2006 - 2007
No one has invaded my sig yet. Mwahahahaa >P

....Then along came SongstressLenne ♥ Alkaren stopped in too ♥

#10 User is offline   angrypata 

  • Bigfoot
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 10308
  • Posts: 89
  • Joined: 11-April 07
  • Location:port huron

Posted 20 May 2008 - 09:09 PM

View PostLenne, on May 20 2008, 11:06 PM, said:

The post I saw on the forums about 4chan morons possibly trying something (which I only just saw today) was posted only a couple days before the con. A lot of the senior staff goes offline a few days before the con officially starts because of travel and other preparation and setup that must be done.


Doesn't matter. If you choose to place yourself in a position of leadership at an event as large as ACen, it is YOUR responsibility to stay informed up to and throughout the event. The idea that the senior staff goes "offline" and basically checks out during the crucial last days before the event is ludicrious. The obvious time to perform a sabotage is at the last minute, and anyone with half a brain knows to be prepared for that.

Excuses, excuses. Is that the official response to everything?
Thanks to Time Mage, we did Super-Job! This is the way!

psst, i'm over here

#11 User is offline   jrbestler 

  • Lurker
  • Pip
  • Group: Lurker
  • Member No.: 10946
  • Posts: 47
  • Joined: 07-May 07
  • Location:Chicago Il

Posted 20 May 2008 - 09:56 PM

Thankyou for the explination. I appreciate that you're taking the time to speak the attendees and to adress the isue.

Better communication at the time of the incident would have been the biggest help. As I stated in a seperate thread, had I realized the computers were likely tot ake several hours to fix I would have left and come back later.

You mentioned that IRT was allowing people to leave and come back for bathroom breaks etc. Would it be possible to formalize a system for people to leave and come back later in the case of another major shutdown like this one? Part of the problem was that many of us felt "trapped" on Friday; it was either stick it out for a "little while longer" (actualy a lot longer) or go through the whole ordeal again. We're all "stuck" in line, bored and causing trouble fopr IRT at that point. That more and more people were constantly being added to the line could not have been helping.

An option to leave the line without loosing 3, 4, 7 hours of waiting would have allowed many people a chance to cool thier heads. You mentioned that long lines breed longer lines. reducing the line size might also help things run more smoothly once things are running again. Also, it might help reduce the load on IRT, who I'm sure were not ahppy about keeping that many people in line.
The hordes of hades all need to ship their dorm rooms home. Right now.
The leviathian has a job that will take 12 hours to run, but they need it in 2.
The four horsemen want to know why my store isn't open 24 hours.

My day's great. How's yours?

#12 User is offline   animefan1977 

  • Bigfoot
  • Pip
  • Group: Bigfoot
  • Member No.: 4518
  • Posts: 63
  • Joined: 09-May 05
  • Location:IL, Southern 'Burbs of Cook County

Posted 21 May 2008 - 08:59 AM

View Postangrypata, on May 20 2008, 09:59 PM, said:

Yes. I've known that since Friday. Pretty sure 420ch/4ch (there's a lot of overlap, especially among the anon that are capable of that kind of thing) had something to do with it, and YOU KNEW THAT. It had ALREADY BEEN ALLUDED TO ON THE FORUMS--YES, THESE FORUMS.

You had time to deal with this, and you didn't. You had time to make backup preparations for something like a denial of service attack (which should have been in place ANYWAYS for ANY large-scale realtime operation that involves online anything) and you didn't. You kept the entire line in the dark about everything that happened, and then only gave a five-dollar discount (because oh yeah, we missed Friday entirely) to I don't know how many people, and as far as I know, that was only after I grabbed the finance manager and gave him a...friendly talking-to. On the forums you claimed there would be at-con reg on Thursday night. My party (ten people) left a day early to take advantage of this option, and got there only to find out that there WAS no at-con reg, only pre-reg. Thanks, guys. I missed another day of work and paid for another night's stay for no reason whatsoever.

If you people cannot handle what this convention has become, you need to hand over certain tasks to professionals. Two of those tasks are security and registration, both of which became serious issues during the marathon reg wait. Otakon is a convention that eclipses ACen in every way, from attendance to events to guests, and I have never waited for more than three hours in their registration line in the last five years, even with the line wrapped several times around the Baltimore Convention Center.

That was pathetic, ACen. You screwed it up last year, you screwed it up this year, and I'm certain you'll have some woeful explanation for the newest crisis when you screw it up again next year.

You need to fix this from the ground up, or you're going to collapse under the weight of your own hubris.

IN SHORT:

1. HIRE SOME PROFESSIONALS.

2. HAVE A BACKUP PLAN.

3. PAY ATTENTION. YOU ARE FULLY AWARE THINGS LIKE THE CH EXIST.

PS. If it WAS you, Anon, well played. It was a perfect storm.



Yes because apparently the "anonymous" are to be taken seriously at all times. :rolleyes: Who on these forums said that at-con registration was happening Thursday night? AFAIK Thursday night has always been pre-reg only.

#13 User is offline   Aj-chan 

  • Addict
  • PipPipPip
  • Group: Newbie
  • Member No.: 6981
  • Posts: 955
  • Joined: 01-May 06
  • Location:Waiting in line for Acen 2012

Posted 21 May 2008 - 09:23 AM

Yep, that was repeated over and over and over again on the forums.

#14 User is offline   arina_shirakawa 

  • Ace
  • PipPip
  • Group: Ace
  • Member No.: 81
  • Posts: 238
  • Joined: 11-April 03
  • Location:Neko-Town

Posted 21 May 2008 - 11:40 AM

View Postangrypata, on May 21 2008, 03:09 AM, said:

Doesn't matter. If you choose to place yourself in a position of leadership at an event as large as ACen, it is YOUR responsibility to stay informed up to and throughout the event. The idea that the senior staff goes "offline" and basically checks out during the crucial last days before the event is ludicrious. The obvious time to perform a sabotage is at the last minute, and anyone with half a brain knows to be prepared for that.

Excuses, excuses. Is that the official response to everything?


I remember a few years back ACen implemented like an instant online update thing where staffers can go online during con and update any last min changes during the con for those who have internet access to their rooms. Don't know if they still do it, but I thought it was awesome.

Arina Shirakawa (and don't take it as my real name, lol :p ) Registration Staff 2003-2004/ Production/ Semi -Registration Staff 2005

Cheesy with my Cutie Since Dec 14, 2005 <3

Gonna try to get a table @ Artist Alley in 09~ Come Visit me <3

http://arina-shirakawa.deviantart.com


#15 User is offline   Alysia 

  • MAPS Director
  • PipPip
  • Group: MAPS Board
  • Member No.: 237
  • Posts: 208
  • Joined: 24-April 03

Posted 21 May 2008 - 12:11 PM

View Postangrypata, on May 20 2008, 10:09 PM, said:

Doesn't matter. If you choose to place yourself in a position of leadership at an event as large as ACen, it is YOUR responsibility to stay informed up to and throughout the event. The idea that the senior staff goes "offline" and basically checks out during the crucial last days before the event is ludicrious. The obvious time to perform a sabotage is at the last minute, and anyone with half a brain knows to be prepared for that.

Excuses, excuses. Is that the official response to everything?


Please link to where an ACen staffer said that there would be at-con reg Thursday. I can find no evidence to back up this claim. This has never been the case.

We do not "check out." We go to the convention site and set up our departments and begin our duties, which generally do not include surfing the forums. Or, we may be in transit. We have not yet managed to find a way to warp time and space and appear at the con instantaneously, alas.

For example, I set up our internal print shop, take delivery of the program books and schedules and generally run around making sure everyone who has print needs is taken care of. By Friday morning I had printed several thousand impressions on our high speed printer alone, not counting the offsite printing my ADH did. I have relatively light duties at the con, most of our print needs are taken care of in advance.

I understand your frustration, but you are being irrational here. Your suggestion makes very little sense-are you serious that you feel senior management should be surfing 4chan for possible threats rather than overseeing convention setup? Isn't that a dereliction of duties in and of itself? It can also be very difficult to measure the seriousness of Internet threats-plenty of people make them, very few carry them out. What solution do you propose for determining what threats to take seriously?

We are perfectly willing to take criticism, but we need suggestions that can actually be implemented.

#16 User is offline   davebb 

  • EMRT
  • PipPip
  • Group: ACen Staff
  • Member No.: 11282
  • Posts: 289
  • Joined: 14-May 07
  • Location:Oak Forest, IL

Posted 21 May 2008 - 01:03 PM

I would like to add a note on the network aspect. There are MANY ways to run a denial of service attack, of which because this is a public forum I don't want to give people ideas. But remember when YAHOO was knocked out from a DDoS? Now if Yahoo goes down, imagine a small internet site like ACEN compared to a big company like Yahoo. There really isnt' much to do other than just disconnect yourself from the network. Now proactive measures in how the system is set up initially things can be done, but without being on the network staff at acen, which i rarely have time for my own life, I really can't say anything of how to set it up.
Medic 6 : ACen 2014 EMRT ConCenter Supervisor (serving since 2008)
Dave Bukowski -- NREMT-B
EMT-B / EMA
..........
"ask not what ACen can do for you--ask what you can do for ACen"

#17 User is offline   magicreaver 

  • Bigfoot
  • Pip
  • Group: Bigfoot
  • Member No.: 3343
  • Posts: 75
  • Joined: 29-November 04
  • Location:NW Indiana

Posted 21 May 2008 - 02:06 PM

Anyone who thinks you should have been fully aware and prepared for attacks posted on some other forums are fools. I'd like to thank the ACen staff that did their job to the best of their ability. Unlike alot of people, I had nothing but pleasant experience with all reg staff or IRT I was around.

Unfortunatly, the dozens of angry people on the forums and beyond are wanting someone to step up and accept responsibility for the failings of the Registration department. I have heard multiple people ask for both the Reg DH and the Con Chair to step down, and that is just foolish. Ultimatly the failing was the fact that they were fully dependant on Web servers, instead of the much more logical (and secure) local server, and anyone who does any IT work will tell you that people in charge, in most cases have no clue what setup any computer systems are using. I highly doubt either of those two, who obviously have far more responsibilities were responsible. However, whoever was responsible should take responsibility. I'm sure it will be unpleasant for you, but I'd much rather see the one or two people who messed up getting flamed here then all the inocent staff that did their best, or even ACen as a whole.
I'd also like to see ACen stop trying to blame others. Yes its unfortunate you got attacked, and those responsible should be tarred-and-feathered and publicly humiliated next acen; but ultimately the failing was yours. I'm not saying you should have been aware of, or ready to stop such an attack, but your network shouldn't have even been conected to the web in the first place. The Reg staff did the same thing last year with badge mailings. You guys foolishly mailed bulk mail very close to the con, then tried blaming the post office for your own mistake. ACen should start to accept responsibility for its own failings, if you can't do that you are never going to improve.

And as far as understaffed issues (which i didn't see here, but have heard others name), that is also ACen's fault. I would gladely give up some of my time to help ACen, if your staffing requirements weren't so rediculious. I have in previous threads been saying 18 hours, however I recently saw a staffer say 24 hours of work. That is $180 at minimum wage in IL, for a $50 badge and maybe a room. Not to mention that means missing alot of events that you are there to see. I'm sorry, I appreciate everything ACen staff does, and would like to give back, but I'm not sacrificeing my vaction for you, and alot of others I know feel the same way.

#18 User is offline   The Archfiend 

  • Regular
  • PipPip
  • Group: Regular
  • Member No.: 1145
  • Posts: 121
  • Joined: 17-October 03
  • Location:Chicago, IL

Posted 21 May 2008 - 02:25 PM

View Postdavebb, on May 21 2008, 07:03 PM, said:

I would like to add a note on the network aspect. There are MANY ways to run a denial of service attack, of which because this is a public forum I don't want to give people ideas. But remember when YAHOO was knocked out from a DDoS? Now if Yahoo goes down, imagine a small internet site like ACEN compared to a big company like Yahoo. There really isnt' much to do other than just disconnect yourself from the network. Now proactive measures in how the system is set up initially things can be done, but without being on the network staff at acen, which i rarely have time for my own life, I really can't say anything of how to set it up.


Okay, bear with me on this since (A), I have no real idea whether this has been touched on and (B), have no real desire to slog through the umpteen threads on this subject.

Everyone knows about the DDoS by now. One possible solution to a repeat is to download preexisting reg info (and onsite reg software) to a CPU or server specifically disconnected from your ISP in order to avoid a domain-wide DDoS from causing a massive SNAFU like this again. After the convention, those updated forms and info can be uploaded back to oldcrows.net (I'm presuming that's still acen.org's backbone provider), which means that a DDoS doesn't actually hit anything except online functions. Why this wasn't done in the first place when the 420chan threat came through is beyond me unless no one took it seriously.

That being said, my irritation with having to be in reg 9-plus hours on Friday is somewhat mitigated by the fact that Acen was caught with their pants down on this, but not much. If this had actually been caused by outright incompetence instead of an external attack, this would've been a much nastier post to read.
Chris Krolczyk
Security/IRT Staff, 2003-5

Further proof of my degeneracy can be found at: http://the-archfiend.livejournal.com/ or http:the-archfiend.blogspot.com/

#19 User is offline   Yoric 

  • Newbie
  • Pip
  • Group: Newbie
  • Member No.: 6489
  • Posts: 19
  • Joined: 10-March 06
  • Location:Zion, IL

Posted 21 May 2008 - 02:46 PM

I've seen a seperate offline server mentioned, and I as well think that is the way to go. As a suggestion, perhaps kill online pre-reg as of a week or so before the con and move a copy of the reg database onto the offline server and then there is no chance of this specific event happening again. I know it might make things difficult for people who don't know until last minute that they are attending, but if forcing the last minute people to have to reg at con can keep something like what happened this year from happening again, I think its worth it. Thats just my two cents.
Co-founder/Host - Gold Coin Panel: Nerd/Geek Podcast
www.facebook.com/goldcoinpanel

#20 User is offline   sisterdiscord 

  • Ace
  • PipPip
  • Group: Members
  • Member No.: 1559
  • Posts: 380
  • Joined: 03-February 04

Posted 21 May 2008 - 02:53 PM

View Postmagicreaver, on May 21 2008, 08:06 PM, said:

However, whoever was responsible should take responsibility. I'm sure it will be unpleasant for you, but I'd much rather see the one or two people who messed up getting flamed here then all the inocent staff that did their best, or even ACen as a whole.


I'd honestly sooner take the beatings myself than pass them on to staff members who volunteered their time and energy in good faith to help make this happen, Reaver. I'm one of the officers responsible for the event, and a piece of the preplanning for that event failed. Lack of oversight is not an excuse, you're right, and I wasn't trying to offer excuses, but I did want to explain and try to dispel some of the crazy myths I'd heard...everything from staffers maliciously damaging stuff to us firing half the reg staff, none of which are true.

As for stepping down, no, I'm not going to do that, and not going to ask Artie to. I want to be around like an old dog to see this fixed. We're drastically increasing our IT team, and taking a page from six sigma as far as our failure and contingency planning strategies. Drastic changes to our technology strategy are required, and those changes will be made.


View Postmagicreaver, on May 21 2008, 08:06 PM, said:

I'd also like to see ACen stop trying to blame others. Yes its unfortunate you got attacked, and those responsible should be tarred-and-feathered and publicly humiliated next acen; but ultimately the failing was yours.


You are absolutely correct. We do our due diligence every year, and we take every threat seriously. There are threats of some sort every year, and when we learn of them we make our staff, the hotel, and Rosemont police aware. I did know about the 420chan thread, as I posted elsewhere on the forums. I personally interpreted it as more of a physical mischief threat, unfortunately, than an electronic one, and I'm not even certain that the registration issue was related to the 420-chan thread. I'm not local to Chicago, so as of this writing I'm out of the loop on those proceedings.

View Postmagicreaver, on May 21 2008, 08:06 PM, said:

And as far as understaffed issues (which i didn't see here, but have heard others name), that is also ACen's fault. I would gladely give up some of my time to help ACen, if your staffing requirements weren't so rediculious. I have in previous threads been saying 18 hours, however I recently saw a staffer say 24 hours of work. That is $180 at minimum wage in IL, for a $50 badge and maybe a room. Not to mention that means missing alot of events that you are there to see. I'm sorry, I appreciate everything ACen staff does, and would like to give back, but I'm not sacrificeing my vaction for you, and alot of others I know feel the same way.


Your point is well taken and one I'll sit down with our chief of staff and review. Yes, there is a high time commitment for full staff, and many of us, myself included, don't see much of the con. The "gofer" or volunteer squad has a lower commitment, badge/badge+crash space, depending on the hours worked.
What seems like a fair middle ground that won't radically increase our staff numbers?
This is my signature. It is only one line.

#21 User is offline   Sakamura 

  • Bigfoot
  • Pip
  • Group: Members
  • Member No.: 4619
  • Posts: 72
  • Joined: 16-May 05
  • Location:Chicago

Post icon  Posted 21 May 2008 - 02:58 PM

Convention Center internet - ??
Hotel Internet - Perfect (1 Gb of glory)

There could have been numerous reasons why ACEN registration was down: DDoS unlikely.

"PS. If it WAS you, Anon, well played. It was a perfect storm."

It wasn't. We don't raid convention websites... oops did I say we?! ;)

animefan1977 I'm only going to agree with you on #2. Yet if you're running on your backup plan for 5 hours straight, you have been running at less than capable performance.
Former Staff

#22 User is offline   gregly 

  • Bigfoot
  • Pip
  • Group: Members
  • Member No.: 478
  • Posts: 89
  • Joined: 19-May 03
  • Location:Aurora

Posted 22 May 2008 - 02:45 PM

View PostSakamura, on May 21 2008, 08:58 PM, said:

Convention Center internet - ??
Hotel Internet - Perfect (1 Gb of glory)

There could have been numerous reasons why ACEN registration was down: DDoS unlikely.

"PS. If it WAS you, Anon, well played. It was a perfect storm."

It wasn't. We don't raid convention websites... oops did I say we?! ;)

animefan1977 I'm only going to agree with you on #2. Yet if you're running on your backup plan for 5 hours straight, you have been running at less than capable performance.


It was actually angrypata who said that, not animefan1977. And I agree: backup plans should have included standard dial-ups and a few modems in case the broadband failed, and the registration process never should have relied on an off-site server, especially on the same IP as the website. That was an unfortunate design, because it stuck a gigantic huge point of failure/avenue of attack in the middle of a critical process. I'm in embedded programming, not IT administration, but that just seems like common sense to me.

Here is what I'd like to see ACen staff do to respond to the cries of "get professional registration and security staff": get a bunch of estimates on what this would cost. Post them publicly on the forums. I would be willing to bet that it would require raising badge costs by a hundred bucks or more. I would also love to be proved wrong.
Former Production Staff Member (3 years, 2003-2005)
Artists' Alley Husband And Table Lackey (2006, 2008-2009)

#23 User is offline   Aj-chan 

  • Addict
  • PipPipPip
  • Group: Newbie
  • Member No.: 6981
  • Posts: 955
  • Joined: 01-May 06
  • Location:Waiting in line for Acen 2012

Posted 22 May 2008 - 02:54 PM

View Postgregly, on May 22 2008, 03:45 PM, said:

Here is what I'd like to see ACen staff do to respond to the cries of "get professional registration and security staff": get a bunch of estimates on what this would cost. Post them publicly on the forums. I would be willing to bet that it would require raising badge costs by a hundred bucks or more. I would also love to be proved wrong.

This is the same thing we are discussing in regards to bringing in professional con planners. Price is a big deal to many people who attend the con, it has to be decided if we want to lose those folks in exchange for a high quality con (although it would be nice to have both :D ).

#24 User is offline   Bloo09 

  • Sage
  • PipPipPipPip
  • Group: Sage
  • Member No.: 17891
  • Posts: 2,255
  • Joined: 14-April 08
  • Location:Chicago ish

Posted 22 May 2008 - 09:03 PM

I'm just curious how professional reg staff which would be 50ish paid staffers (do the math for min pay etc for 4 days of reg work.) would stop system errors and well.. everything that went wrong was computer based.. so hire the entire staff with computer training? and paid security 100+ more security due to the extra space we are using. Think about how much a badge would cost.

Our mailing this year had a 1%-2% fail rate, compared to last year thats amazing so if you're worried about lines do that next year.
And for next year we will not be hooked to the internet other then for credit cards. (some mail problems were the us postal service with a post marked envelope for April 14th showing up early May)

The computers that were ruined.. whoever did that, you're a special sort of person. I didn't see numbers so i don't think people realize how serious it was. 18 keyboards, 17 mice (2 survived because they were unplugged by con goers and thrown behind our curtains) a few monitors were written on in permanent marker and 3 or 4 towers were ruined(there was a pen cap shoved in a cd-rom, things put inside hard drives..). That is a large hit to our reg supplies and definitely slowed things down a lot.

Sure Acen says full time staffers work 24 hours over the weekend. I spent a lot of time in reg (no I am not reg staff ignore the side info thing but Im married to the DH and help a lot) I would say 90% of that staff worked from 7am until closing every day of the convention and barely took breaks. Most of them never saw the dealers hall or their favorite voice actors. These guys and girls worked their butts off to make things run and make up for the half of staff that decided not to show and will not be invited back on staff.

My husband and I are discussing ways to ensure staff shows up and works their hours. If you have -any- ideas other then hire a company which we wont be doing, then please PM me or email acenreg08@gmail.com with the subject title staff suggestion or something along that line. Even after Acen I will be checking email but watch here for when we change to acenreg09 we will post a sticky.


Finally I am sorry I have not been posting on here but I was one of the lucky few who caught "con plague" and I'm just getting over it >.< So if emails from acenreg08@gmail.com aren't answered they will be shortly unless they are not constructive. We do keep record of complaints but there is no need to reply to most. EVERY email is read.
ACen 14 - Panel Programming Department Head and Crystal Ball Manager
ACen 12 - 14 - ACen & Anime Central Facebook Group(s) Administrator
ACen 13 - Panel Programming Department Head and Crystal Ball Manager, ACen 12 - Panel Programming Dept. Head and Crystal Ball Manager,
ACen 11 - Reg Dept. Head, ACen 10 - Reg Dept. Head, ACen 09 - Reg Asst. Dept. Head, ACen 08 - Guest Relations staff, ACen 07 - IRT Guest Escort

Please contact me if you have any questions, comments or concerns regarding Panel Programming for Anime Central. PanelProgramming@ACen.org.
If you have any questions about the Crystal Ball please email CrystalBall@ACen.org

My private messages are now disabled. Between friend invite spam, and forum notices as a moderator I was missing panel related private messages. Please email us to contact panels, thank you!

#25 User is offline   Aj-chan 

  • Addict
  • PipPipPip
  • Group: Newbie
  • Member No.: 6981
  • Posts: 955
  • Joined: 01-May 06
  • Location:Waiting in line for Acen 2012

Posted 22 May 2008 - 09:11 PM

View PostBloo09, on May 22 2008, 10:03 PM, said:

The computers that were ruined.. whoever did that, you're a special sort of person. I didn't see numbers so i don't think people realize how serious it was. 18 keyboards, 17 mice (2 survived because they were unplugged by con goers and thrown behind our curtains) a few monitors were written on in permanent marker and 3 or 4 towers were ruined(there was a pen cap shoved in a cd-rom, things put inside hard drives..). That is a large hit to our reg supplies and definitely slowed things down a lot.

:mellow: Wow. Just... wow.

And yay! Bloo09 is back. Thought maybe you and Tevva had taken off on a long vacation or something. You guys deserve it. :heart:

#26 User is offline   Riker 

  • Ace
  • PipPip
  • Group: Members
  • Member No.: 17316
  • Posts: 261
  • Joined: 25-February 08
  • Location:Evanston, IL

Posted 22 May 2008 - 09:41 PM

View PostCaerulea Windseeker, on May 20 2008, 12:44 PM, said:

I'm also looking for a commitment from the highest levels of ACen leadership to come up with a sensible, practical solution to this problem that will help reduce the stress and danger to both con-goers and I.R.T. Finally, I am looking for the new policy to be published so that con-goers have something in writing to defend themselves when staff implements the policies in an inconsistent manner.


I am not at the highest levels of ACen leadership -- quite the opposite -- but I agree with you that the policies should be published ahead of time. As one of the reg programmers I have some ideas on how to improve the visibility of this information. I need to have them approved and discussed first, but it is something that I'm paying attention to.


View Postangrypata, on May 20 2008, 09:59 PM, said:

You kept the entire line in the dark about everything that happened, and then only gave a five-dollar discount (because oh yeah, we missed Friday entirely) to I don't know how many people, and as far as I know, that was only after I grabbed the finance manager and gave him a...friendly talking-to.


I don't mean to steal anyone's thunder, but the five-dollar discount was my doing. I spent a fair amount of time on the radio with our finance manager (who is a woman, by the way, so I don't know who you were talking to) advocating for a discount. Five dollars was what my superiors authorized me to give, despite me asking for more, so that's what we implemented. (Note that I am NOT trying to pass the buck on this to my superiors. They are responsible for maintaining the financial stability of the con and I stand by their decisions.) We then had to send runners to get five-dollar bills to stock the cash registers so that we would have enough to make change for the discounts. It wasn't something we could implement with a moment's notice. I had to get it cleared and then we had to get the actual cash.


View Postangrypata, on May 20 2008, 10:09 PM, said:

If you choose to place yourself in a position of leadership at an event as large as ACen, it is YOUR responsibility to stay informed up to and throughout the event. The idea that the senior staff goes "offline" and basically checks out during the crucial last days before the event is ludicrous.


I can vouch for what Alysia said earlier -- there is so much to be done in the days leading up to ACen every year that the staff barely has time to check their voice mail, let alone surf the forums randomly looking for anything out of the ordinary. However, while reading this, I did have a thought. It might be feasible to have a staff member whose job it was to monitor the forums throughout the convention and the days leading up to it, and notify senior staff of any concerning threads. I will bring this up at our meetings this year and see if we can keep closer tabs on this. I can't promise anything but this is a good idea that deserves some thought.


View Postjrbestler, on May 20 2008, 10:56 PM, said:

Better communication at the time of the incident would have been the biggest help. As I stated in a seperate thread, had I realized the computers were likely tot ake several hours to fix I would have left and come back later.


JR, I apologize. If we had realized the computers were going to take that long to fix, we would have let you know. At first we suspected a network problem in the convention center, so we wanted to wait a few minutes to see if our network tech could fix it. Then later we suspected the server itself was the issue, so we rebooted it. (Many of you may remember me yelling at everyone trying to explain this.) It wasn't until we had already been down for quite some time that our network tech discovered that the problem was a DDoS attack. Also remember that it began gradually, with the network getting slower and slower before grinding to a halt. A sudden failure would have been easier to comprehend and diagnose. By the time we knew for sure what was going on, we were already into the manual-badge-writing phase and it was felt that pushing through and getting badges to all the people who had been waiting would be better than turning anyone away. Honestly, I apologize. If we'd known it was going to be that long we would have let you know.

As another consideration, I thought a few times about standing up and shouting to everyone that we were being DDoS attacked. But not everyone understands a DDoS attack, nor why it's outside our realm of direct control to stop. I decided it was best not to waste my voice explaining networks to the entire reg line. :rolleyes:


View Postmagicreaver, on May 21 2008, 03:06 PM, said:

Ultimatly the failing was the fact that they were fully dependant on Web servers, instead of the much more logical (and secure) local server.... whoever was responsible should take responsibility.


View PostThe Archfiend, on May 21 2008, 03:25 PM, said:

One possible solution to a repeat is to download preexisting reg info (and onsite reg software) to a CPU or server specifically disconnected from your ISP .... Why this wasn't done in the first place when the 420chan threat came through is beyond me unless no one took it seriously.


View PostYoric, on May 21 2008, 03:46 PM, said:

I've seen a seperate offline server mentioned, and I as well think that is the way to go. ...move a copy of the reg database onto the offline server and then there is no chance of this specific event happening again.


OK. I have multiple levels of response to this. First of all, the decision to keep all the information on the web server was a poor one. I can tell you, as one of the people who contributes to ACen's computer setup, that there was no moment when we said, "sure, we'll just keep it on the web server, everything will be fine!" Instead, there was a general failure of a number of people to think that strategy through and consider possible points of failure. No one person is responsible for that strategic decision; everyone who might have caught it didn't do so, myself included. I know I've been kicking myself about it since Friday afternoon and frankly, we all have been. We're aware that we missed a doozy. We feel bad about it. And we're fixing it.

Secondly, the local server is exactly what we did do, beginning Saturday morning. We brought in a server Friday night at 1 am and myself and Rabbi stayed up all night moving our database and software onto it, and reconfiguring the floor computers to use it instead of the online server. This wasn't a solution that could be implemented immediately (for one thing, we had to find a server that one of our staff had access to that could handle the load, and then get it driven in from an hour away). For another thing, all of the staff who might have been working on that solution were busy trying to get people through the line and out of it as quickly as possible. Installing an operating system, configuring apache, importing the database, loading our software onto it, and testing does not happen in a few minutes.
Riker
Night Shift Dispatcher, IRT
Programmer and Network Tech, IT Dept.
Convention Committee Member, Operations Section


Property of SongstressLenne ♥

Orbis Arangiam Nostram Est

#27 User is offline   Yoric 

  • Newbie
  • Pip
  • Group: Newbie
  • Member No.: 6489
  • Posts: 19
  • Joined: 10-March 06
  • Location:Zion, IL

Posted 22 May 2008 - 10:39 PM

View PostRiker, on May 23 2008, 03:41 AM, said:

OK. I have multiple levels of response to this. First of all, the decision to keep all the information on the web server was a poor one. I can tell you, as one of the people who contributes to ACen's computer setup, that there was no moment when we said, "sure, we'll just keep it on the web server, everything will be fine!" Instead, there was a general failure of a number of people to think that strategy through and consider possible points of failure. No one person is responsible for that strategic decision; everyone who might have caught it didn't do so, myself included. I know I've been kicking myself about it since Friday afternoon and frankly, we all have been. We're aware that we missed a doozy. We feel bad about it. And we're fixing it.

Secondly, the local server is exactly what we did do, beginning Saturday morning. We brought in a server Friday night at 1 am and myself and Rabbi stayed up all night moving our database and software onto it, and reconfiguring the floor computers to use it instead of the online server. This wasn't a solution that could be implemented immediately (for one thing, we had to find a server that one of our staff had access to that could handle the load, and then get it driven in from an hour away). For another thing, all of the staff who might have been working on that solution were busy trying to get people through the line and out of it as quickly as possible. Installing an operating system, configuring apache, importing the database, loading our software onto it, and testing does not happen in a few minutes.


Ah, well then I applaud the work done. Upon hearing just how much physical damage was done to the systems Friday night, I can understand the situation for Saturday. I work in IT, so I know how long it can take to set up a server and get the entire network running off of it, so I again, I applaud the hard work and long hours put in to fixing everything as best as can be expected during a network crisis. Keep up the good work, and I have faith that all this will be fixed for '09.
Co-founder/Host - Gold Coin Panel: Nerd/Geek Podcast
www.facebook.com/goldcoinpanel

#28 User is offline   Riker 

  • Ace
  • PipPip
  • Group: Members
  • Member No.: 17316
  • Posts: 261
  • Joined: 25-February 08
  • Location:Evanston, IL

Posted 23 May 2008 - 12:42 AM

View PostYoric, on May 22 2008, 11:39 PM, said:

Ah, well then I applaud the work done. Upon hearing just how much physical damage was done to the systems Friday night, I can understand the situation for Saturday. I work in IT, so I know how long it can take to set up a server and get the entire network running off of it, so I again, I applaud the hard work and long hours put in to fixing everything as best as can be expected during a network crisis. Keep up the good work, and I have faith that all this will be fixed for '09.


Thanks Yoric. I'm sorry if I seemed a little defensive in my last post. It's been a very frusterating saga with this attack and all the fallout from it. =)
Riker
Night Shift Dispatcher, IRT
Programmer and Network Tech, IT Dept.
Convention Committee Member, Operations Section


Property of SongstressLenne ♥

Orbis Arangiam Nostram Est

#29 User is offline   Caerulea Windseeker 

  • Lurker
  • Pip
  • Group: Purge for Inactivity
  • Member No.: 18564
  • Posts: 45
  • Joined: 19-May 08

Posted 23 May 2008 - 01:04 AM

View PostRiker, on May 23 2008, 12:42 AM, said:

Thanks Yoric. I'm sorry if I seemed a little defensive in my last post. It's been a very frusterating saga with this attack and all the fallout from it. =)


I appreciate the work you did trying to make the problem better Saturday. That kind of stuff can be maddening to do even when you're not pressured for time. My main concern has been line management, since lines will happen at conventions and there's no getting around it. This is why I have complained so much about how the lines are handled, not so much that there were lines in the first place. I kind of figured those who had systems in place to fix last years' problems would be quite aggravated at the new crop of problems this year. :( I hope that you guys can catch whoever damaged con property and force them to pay restitution.

#30 User is offline   Yoric 

  • Newbie
  • Pip
  • Group: Newbie
  • Member No.: 6489
  • Posts: 19
  • Joined: 10-March 06
  • Location:Zion, IL

Posted 23 May 2008 - 01:06 AM

View PostRiker, on May 23 2008, 06:42 AM, said:

Thanks Yoric. I'm sorry if I seemed a little defensive in my last post. It's been a very frusterating saga with this attack and all the fallout from it. =)


I can totally understand, not from experience, but just thinking about being in your position makes me shudder at what kind of stuff is coming at you and the rest of the staff.
Co-founder/Host - Gold Coin Panel: Nerd/Geek Podcast
www.facebook.com/goldcoinpanel

  • 2 Pages +
  • 1
  • 2
  • You cannot start a new topic
  • This topic is locked

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users