Server Failover And Load Balancing

Some scenarios for #211.

Redundancy and load balancing will be performed by clusters.

--- The rest of this page is horribly out of date. A newer version will be up soon.

This document describes how servers will function together in order to ensure high availability of all game objects. In general, the RealmServer? is going to have the most to do with distributing the simulation load across servers. RealmServers? will keep a list of servers that are available to simulate parts of the realm and it will periodically query them for their ping time and load. The RealmServer? has the abillity to direct its SimulationServers? to simulate any area/object within the realms they are responsible for. It will be up to the Realm Server to ensure that there is always one master and at least one secondary server for every area/object. The SimulationServers? will take orders from the RealmServer?, but will also be able to transfer Objects across servers if the objects leave the areas the server is primary for.

The role that the DirectoryServer? plays in all of this is similar to the RealmServer?, but instead of making sure all areas have Primary and Backup, it makes sure all Realms have Primary and backup.

Basics

Object Load Balancing

The question here is whether or not it makes sense for one server to own an object that is simulated within an area that another server is the master for. When a server is the slave server for an area, it is really a slave server for all objects in that area as it will be fully simulating them.

When creating an object in an environment where one server is master of the objects area and the other is slave, the object could be created on either server, but would exist on both since when it is added to the area, the master server will get a copy (since it's the only server that can add it) and it'll send a copy to the slave server. The master server should automatically set the slaveservers for the object to all of the slave servers for the area it is in.

When an object leaves an area, (no matter what server it is owned by), it should be added to its next area before removing it from the previous. That way, it won't be transferred across the network more than once. When removing from an area, the area should remove the appropriate secondary servers from the object. The new area should add its secondary areas when the object is added.

Master areas will check periodically to make sure that any objects within their area that they are not master for are still available from their server. If a master object is not available, the master area should take it on as the new master for that object. It should make sure to announce to all slave areas of the change so that they can Replace it.

Additional Musings

Last night, I was enjoying the slowness that is running 3 MV3D servers and 1 MV3D client on the same machine (don't try this at home unless you have at least a dual CPU box). Mostly, I was making sure the unit-tests for redundancy weren't lying when they said they completed successfully. Things seemed to work pretty well with a few minor bugs (when telling it to find a new master server for an area, the current master gets dropped to slave-level but doesn't get listed as an official slave server on the realm).

Then it went and did something quite unexpected (to me at least). When it found a new master server for an area, the new master was definitely the master server for the area, but all the contents of the area remained at a cache level. To explain, there are three levels of ownership. Master level is reserved for the one server that owns the item. Slave level is for any number of servers that are doing full simulation on the item. Cache level is below Slave level and just means the server is recieving updates on the item. The theory is that slave level items can take over for the master object at any time without losing a beat. Items don't have to care about who is Caching them, and servers that cache items don't have to do anything to them except recieve updates. A future enhancement to the system will be to explicitely deny Caches of Cached items. That way, you know that your cached item always comes from either a master or a secondary.

Now that that tangent is over, back to what happens to things inside areas. I really shouldn't have been so surprised things happened the way they did. That is, after all, what the code should have done. It's probably not the desired outcome though. The real question then is: what is the desired outcome? I haven't a clue. To get a clue, I think going over the reasons to transfer ownership of an area from one server to another. Shall we say from server A to server B?

1. Server A is overloaded
2. Server A is going offline
3. Server A is offline and server B is secondary

Interestingly enough, in all of those instances, you'd want to transfer ownership of all master objects within the area on server A over to server B. Are there any instances when you wouldn't want to do that? Yes.

1. Server A is splitting the area with Server B
2. Some Object in the area on Server A is in another area on Server A at the same time

Those are both two fairly tough cases. Especially if in #2, the other area the object is in gets moved to server C. Then what? Let's forget about those odd cases for the moment and figure out how load balancing, redundancy, and failover should work for objects.

With areas, the realm the area is a part of keeps track of a list of areas and their master and slave servers. It also has a list of simulation servers that are available for simulating its areas. It uses those two lists to make sure that all areas have a master and a slave, and that master/slave servers are all online. If there's a problem, it is able to shuffle areas around amongst its simulation servers.

There need to be functions similar to those for areas which have the ability to move objects from server to server. On a higher level, functionality should exist to recursively transfer the contents of something (an area, an object, whatever) to another server. In addition, similar functions to recursively upgrade objects from cache to slave, slave to master, and then down from master to slave and slave to cache should be implimented. However, this does bring up the problem that two objects in the same area can have different masters. The answer of what to do in the situation probably lies in figuring out who is in charge of orchestrating the switch.

For example, when the switch is load related, the realm server first initiates finding a new master for the area.

There are other problems with this whole set up. When an area (or an object) is sent from one server to the other, it includes all subobjects and all parent objects. This is bad. It means that sending an area or object will be very slow and will cause the server to pause while it receives all the data. In an area with 500 objects, this could be a 5 second pause. Not good.

There are two alternatives. In the first one, containers would not actually link directly to their contents or what they are contained by. Instead, they would just include the IDs of those items. The simulation server, then, would contain a big hash of ID:item. This could possibly also help speed up servers that have tons of objects if the Iteration of objects was handled by the Simulation Server instead of individual areas. That would also ensure that an object which existed in multiple areas would not be Iterated twice in a step. When sending an area, just the list of items would be sent and it would be up to the receiving server to figure out when to download them.

The other option is to just have Cacheable send the IDs for contained (and contained by) items when sending them over the network. The problem with this solution is that it would then be Cacheable's problem to reconstitute the object. It would have to do that in an Async way, but there is no way for SetCopyableState? to do that.

I think the first option is the best. The next question is whether or not a realm should become responsible for the priorities of the objects within it. I suspect it is better for the area to be left up to that job. It makes more sense because the problem with realms doing it is that a realm is large. It may be 1000 or more areas and each area could have 500 objects in it, so that would be 500,000 objects. That's a big dictionary.

So if the area is in charge of keeping track of ownership of its objects, that means the master server for an area should be tasked with making sure that all objects in that area have working master servers (and slave servers). If a master server is non functional, the area should make a secondary server the master for that object. If there is no secondary server, it should make itself the master for that object. If a slave server is non functional, that slave server should be removed. If the object has no more slave servers, one should be assigned. The master server for the area should be a master or slave server for all objects within the area.

Higher Level Redundancy

At the higher levels, load balancing doesn't come into play as much, and the most important aspect is redundancy. Directory Servers will be in charge of keeping realms redundant, but there are other things that need to be redundant such as account servers and directory servers. To keep from a chicken and egg problem where there always needs to be a higher level, directory servers should handle their own redundancy. They should assign their first slave server as the 'next of kin' :) and if that slave notices the master going out, it becomes the master. At that level, it's probably ok for the game to require some manual intervention in re-adding the dead directory server.

Account servers are a little different. There should probably be two master account servers and any number of other account servers (that don't have all the accounts, but can get them --- like DNS). The two master account servers could work off of the same database (or two different replicated databases) and all accounts would not be stored in memory, but would be in the database.

That makes a lot of sense. In fact, this should be used for Directory servers as well. Some number of master servers that all use the same database, and other servers query them. Non master directory or account servers keep a list of their master servers and remove any servers from that list if they can't connect to them. They can update the list once connected by asking the master server. Cool!

Back to TechnicalDocumentation.