All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2, 3  Next
Author Message
 Post subject: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 12:23 pm 
User avatar
Carbon Dev

Joined: Wed Nov 02, 2011 7:00 pm
Posts: 4813
Image

If you've been playing (or trying to play) AirMech the past week and weekend you probably noticed we've had a lot of server downtime and restarts. As a gamer, when that happens to me with a game I want to play nothing makes me more upset. So I totally understand why people are either mad or frustrated with the game lately. But I want to take a minute to explain why this is actually a good thing.

As a super small development team, we built the backend systems for AirMech "as needed". Our programmers have experience working on everything from small projects to massive MMOs, and the plan from the start was to build things in a way to get them up and working, even if you know it might fail as it scales up. Everything fails as it scales up in fact, but the more grand your plans the more time it takes to build, and as devs we've had the experience of building grand things which never end up getting used. So AirMech was very purpose built from the start, which is why we released very early versions of it and involved the community directly in the development process.

The server structure we have been using all this time has been great because we can actually replace and restart servers without people noticing. Great for active development, but also masked some problems. Ever seen the bug in chat where you can't see what you type, but everyone else can? Little things like that are known issues, and not simple fixes without rebuilding everything. In reality, we've been restarting the servers quite a bit, just most of the time players don't notice. But that can't continue forever.

With AirMech having grown as big as it has, we could now see points of failure coming up on the horizon. Combined with our existing bugs, we knew it was time to restructure things. Scary words like "sharding" came up, and grand plans to take AirMech to the next level were made. It's been months of work to get to this point, and we started rolling out the new servers.

Then things started crashing.

Now you might say "just put the old servers back", but that doesn't help us move forward. Even with the new servers, we could just restart them, but that fixes no problems. So when something crashes or locks, we need to actively debug what caused the problem as quickly as possible, then restart the server and work on a fix. When the fix has been made, we push out new servers, restart everything, and wait for it to crash again.

The good news is we are isolating and fixing a ton of bugs that have been around for a long time, and are more likely to be triggered with the new server structure. So things are actually getting fixed! The bad news is we are not at the bottom of the barrel of bugs yet. We're working around the clock to fix them as we catch them.

I hope this explains a bit more about what is going on, and why extended downtime is actually better than quick blips and restarts. This is why we still have the Beta label--we wouldn't want to do this to the game post Beta if possible. We really appreciate your support and patience, and if we have caused you a lot of problems we sincerely apologize. Please reach out to us through (support@carbongames.com) or on the forums if there is something we can do to help.


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 12:36 pm 
Beta Lord

Joined: Sat Oct 06, 2012 10:00 pm
Posts: 159
^-^
Well said, informative. Airmech is moving forward.

_________________
Image


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 12:37 pm 
User avatar
Diamond VIP

Joined: Thu Mar 07, 2013 11:16 pm
Posts: 1875
Location: Kentucky, USA
Good Guy James takes the time to explain why servers being down is actually a good thing.

On a personal note, I really only cry when the login server goes down, and that's only because it seems to log me out forcfully each time that happens, which effectively locks me out of posting until they come back up... XD

_________________
Image http://jrurl.at/jrtxservers
Image


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 12:40 pm 
User avatar
Beta Lord

Joined: Fri Sep 21, 2012 9:42 am
Posts: 1293
Location: Twitch.tv/OPTVLive
Agreed....very informative and helpful for everyone to understand what is going on!

I'm sure that once people read this they will be much more understanding as to why we've had some "hiccups" for about a week now, and will be very supportive during this transitional period to a new server infrastructure :)

It also reassures everyone as well that it's not the "Airmech Apocalypse" and that the world is ending! :lol:

Thank you to James & The Team for being the type of DEVS who actually communicate with their playerbase and explain what is going on....it really shows us that you care and just aren't here to "empty people's wallets" (aka EA) ;)

_________________
Image


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 1:34 pm 

Joined: Wed Apr 18, 2012 8:49 am
Posts: 6
Hey Devs,
Do you still use Go for your new backend code?

Cheers


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 1:40 pm 
User avatar
Diamond VIP

Joined: Thu Oct 04, 2012 6:03 am
Posts: 335
Location: Fort Wayne, IN
Thank you for this James, i was really wondering what was going on to make the game mess up so much and this gave me a lot of insight. I feel bad for you guys though, having to do all this work to fix the servers, i wish i could help but sadly i don't know the first thing about computer stuff :lol:

P.S: Steve, I see what you did there ;) :lol:

_________________
Image


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 2:57 pm 
User avatar
Carbon Dev

Joined: Wed Nov 02, 2011 7:00 pm
Posts: 4813
codditor wrote:
Hey Devs,
Do you still use Go for your new backend code?

Cheers

Yes, Go is awesome. The fact that one programmer can architect a whole backend to support a game the size of AirMech is proof that it's pretty amazing. :)


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 3:42 pm 

Joined: Wed Apr 18, 2012 8:49 am
Posts: 6
CarbonJames wrote:
codditor wrote:
Hey Devs,
Do you still use Go for your new backend code?

Cheers

Yes, Go is awesome. The fact that one programmer can architect a whole backend to support a game the size of AirMech is proof that it's pretty amazing. :)


Just one guy for the whole backend? I feel sorry for him :/


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 4:15 pm 

Joined: Tue Jan 08, 2013 2:47 pm
Posts: 1293
codditor wrote:
CarbonJames wrote:
codditor wrote:
Hey Devs,
Do you still use Go for your new backend code?

Cheers

Yes, Go is awesome. The fact that one programmer can architect a whole backend to support a game the size of AirMech is proof that it's pretty amazing. :)


Just one guy for the whole backend? I feel sorry for him :/


BOBLAN

_________________
Image


Top
Offline Profile  
 
 Post subject: Re: Why Servers Crashing is a Good Thing
PostPosted: Mon Nov 25, 2013 5:52 pm 
User avatar
Carbon Dev

Joined: Wed Nov 02, 2011 2:24 pm
Posts: 445
We've narrowed down the possible causes, and the most likely culprit is friend support. I've taken the feature offline until we can confirm this is the case.

_________________
https://github.com/mendsley


Top
Offline Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: Bing [Bot], Housekeeping and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Jump to:  
AirMech Site | Follow us on Twitter | Like us on Facebook | YouTube | Google+ | Download AirMech | Carbon Games
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group