867 concurent players
Last saterday, we had a new record: 867 players, and counting the player run bots, about 950 connections.
I think that this new record is a little bit inflated, it was Stakhanov's day (*3 harvesting experience) and some people also brought their alts on, to level them up faster.
Nevertheless, it was an interesting experience from a development standpoint; the CPU usage almost doubled compared to just 800 connections (it went up to 22%).
A big part of the increase was due to the operating system itself, such as the time spent dealing with intrerupts (hdd access, networking).
Now, 22% CPU is not a lot, but it seemed that the increase was almost exponential, which means that we might not be able to accomodate more than 2K players on a single machine. So in a year or two, we might have to have another server.
This is a logistic nightmare in itself (we would need new moderators for that server, new forums subsections for each server, more time spent to update the servers, and so on).
On the other hand, we need more data, especially to see what happens when we reach 1K connections on a daily basis. We are currently not using full optimizations during the server compilation (we use O1, not O3). We can also try a few more tricks that would reduce the server load a little bit, but it is really hard to predict how the CPU usage will look like with 2K connections. If we are lucky, we might be able to push the limit to 3K players, especially because we ony use a P4 @3Ghz, which is an OLD CPU (about 2 years or so). A newer CPU with more cache might make a big difference.
But one thing is clear, in the future we will have to have more than one server, and need for that is an indication of our success. And by the time we will need that, we hope our financial status will allow us to hire a few full time team members to take care of the moderation issues associated with our future growth.
I think that this new record is a little bit inflated, it was Stakhanov's day (*3 harvesting experience) and some people also brought their alts on, to level them up faster.
Nevertheless, it was an interesting experience from a development standpoint; the CPU usage almost doubled compared to just 800 connections (it went up to 22%).
A big part of the increase was due to the operating system itself, such as the time spent dealing with intrerupts (hdd access, networking).
Now, 22% CPU is not a lot, but it seemed that the increase was almost exponential, which means that we might not be able to accomodate more than 2K players on a single machine. So in a year or two, we might have to have another server.
This is a logistic nightmare in itself (we would need new moderators for that server, new forums subsections for each server, more time spent to update the servers, and so on).
On the other hand, we need more data, especially to see what happens when we reach 1K connections on a daily basis. We are currently not using full optimizations during the server compilation (we use O1, not O3). We can also try a few more tricks that would reduce the server load a little bit, but it is really hard to predict how the CPU usage will look like with 2K connections. If we are lucky, we might be able to push the limit to 3K players, especially because we ony use a P4 @3Ghz, which is an OLD CPU (about 2 years or so). A newer CPU with more cache might make a big difference.
But one thing is clear, in the future we will have to have more than one server, and need for that is an indication of our success. And by the time we will need that, we hope our financial status will allow us to hire a few full time team members to take care of the moderation issues associated with our future growth.
posted by Radu at 2:56 AM
10 Comments:
O3 optimizations won't help much when the server is spending time handling interrupts. I have multiple gentoo systems, and I was running an O3 compiled system for over 2 years. Now I am running O2 because people generally suggest that O3 is for the desperate (at least when you compile your whole system with that flag ;), more problems which are hard to locate and blame on the compiler optimization. Anyway, I am now running on O2 since about an year, and I didn't notice an signifcant performance decrease after the switch, but maybe it helps more if you just have one big performance hog that you compile with that flag.
Generally, I'd rather suggest investing in big CPU caches, PCI-X boards / NICs and hdd raids with hardware raid controllers and lots of cache to improve interrupt handling issues.
While having another server would surely be fun, too, I think it might be hard to handle. Server synchronisation is a new level of programming problems and I am not so sure it's fun coping with such kind of hazzles. Do you plan on just split the continents to different servers? This might make synchronisation the easiest, but balancing the load would not be optimal. Maybe some kind of balancing at the OS level would be easier to implement and make balancing more even, and allow to add more servers without having to change code. Some kind of EL cluster or something :-)
Yes, I know that O3 is a little desperate, but we can try it and see if it works. Last time we used O3 (about 2 years ago) there were a few problems, not sure if bugs in our code or compiler problems.
And of course, the intrerupts won't change much, but we have a few ideas for that as well (trying to reduce the network traffic a little bit).
And when we have multiple servers, we won't split the continents between servers. Instead, we will have a brave new world on the new server, and players won't be able to log in from one server to another (unless if you create a new account there, and you start as a newbie).
I was thinking about this just yesterday, it would be an incredibly exciting feature to have several EL worlds available on the Net:
* every new server = a new planet
* lots of new space to explore
* possible customizations (different races, cultures, magic systems, supplementary abilities, etc.)
* distributed overhead = less risks of saturation
Of course it would be great to be able to keep the same character and move from server to server (teleport to X planet spell?). I think you shouldn't rule this out:
* a "consistency" system would be needed, to be sure that basic rules be the same among planets
* possible customizations should also be evaluated and eventually accepted by a central authority to make sure that they do not unbalance the game
* more radical customizations could however lead to "isolated" planets
* I don't think you'd need different forums, only a new section in existing forum to deal with specific planet/server issues
Looking forward to developments in this area!
Rehdon
Brendan:
The path finding was never the biggest CPU hog, it was and still is the second biggest one.
The biggest one is the routine that calculates who sees who (there are lots of calculations to do, considering that there are 4K entities, such as players and bots and AI and NPCs).
Of course, we don't check each entity against all the other entities, but considering that we have to check that 4 times a second, it does add a lot of CPU usage.
Anonymous: Unfortunately, it is very difficult to manage an economy on such a scale, there are lots of problems with new planets, such as what happens if all the players decide to meet on one planet at a time for some event? How would you make them spread evenly? It's simply not possible to do it in a reliable way.
The only game that has over 2K players on a server (that I know of) is EVE online, where it is easier to mitigate those problems, considering that it is a space game, with lots of star systems and stuff.
Radu, there are several way you could prevent planet overcrowding:
* link character login to only one planet: i.e. a character can login only to one planet and move to other by means of spells, portals, etc.
* make interplanetary travel (love the sound of it :) possible but not too easy: a medium to high level spell, an expensive "toll gate", etc.
* a simple limit to incoming requests on the server: shouldn't be hard to do
All in all I'd say that ensuring consistency between different servers would be a tougher task, one involving trusting other people as well. Of course I'm no programmer, so you know better than me.
Rehdon (the not-so-anonymous poster :)
How would you make them spread evenly?
You control travel between servers (worlds/continents); as a server becomes full travel tickets become harder to get, fewer ships leaving for there. You still need to make an assessment on inactive characters, but the problem is manageable.
The new server could be reserved for serious Role Players only, were people would have to choose fantasy names and l33t names would be kicked and deleted. Everyone would have to speak in character with OOC talking being prefaced with && (or some type of indicator) in local chat. Encounters with other players in areas like PK, would have to be roleplayed, which could be as simple as, I am an evil fighter, or a mad wizard.
@Rehdon The different worlds is an interesting idea that we did consider, but they all share some common areas, such as the newbie island, the wraith, etc. the same way a building shares a lobby area. So putting too many words would together might not be the best idea, although it would be nice if it were possible.
@Anonymous A dedicated RPing server would also be nice, but:
1. We don't have enough hard core Rpers to justify that.
2. We don't have the resources (people/time) to enforce that.
When the server becomes saturated with players, do you have any plans to add preferential login like Tibia or SilkRoad?
No, we don't have such plans, we want to provide enough servers so that everyone can log in whenever they want to.
Post a Comment
<< Home