Some more bugs and problems

A few players pointed out that the odds of creating various items have changed.
This led me to do some research, like looking at the entropy samples in the server generated logs. It does, indeed, seem that there is a slight bias towards the lower numbers.
My host confirmed that in FreeBSD 5 some things changed:
The history (in the man pages) says it all...
A random device appeared in FreeBSD 2.2. The early version was taken
from Theodore Ts'o's entropy driver for Linux. The current implementa-
tion, introduced in FreeBSD 5.0, is a complete rewrite by Mark R V
Murray, and is an implementation of the Yarrow algorithm by Bruce
Schneier, et al.

So I guess now we should look at developing our own pseudo random numbers generator algorithm, to have some consistency next time we change the OS.

One other problem is that today, when I logged in, my health was half of the normal health... And I don't remember losing over 600 points of health, especially that I didn't visit PK maps, and with my stats, it is highly improbable some monster got me so bad.
There have been some other reports in the past regarding bots being either dead, or with very low health in non PK maps.
Finally, the problem that I've mentioned in my previous entry, is that the new machine, while theortically 3 times faster, it makes very little difference in how much CPU time the game server takes. Basically, on the old machine we had about 15% with 500 players and bots on, and with the new machine we have pretty much the same CPU usage..
Other things, however, such as compiling the server, or backing up the users, are significantly faster.

My theory is that what's really using the CPU is not the actual instructions, but the waiting for the memory. Due to the nature of a MMO server, a lot of memory is iterated many times each second, which makes the CPU cache less effective. This is pretty difficult to solve, because there is no effective way to use less memory.
Me and learner discussed about various improvements that can be done (most of the CPU is used by one little function) and despite Learner's best effort to optimize it, he only got a marginal speed gain.

In case you are curious what that function does, it is the function which keeps track of who sees who. It has to be called 4 times each second, and it looks sort of like this:

for(number of players+number of ai)
for(players on the current map)
check to see if the players see eachother, only when one of the players moved
do stuff

Given the fact that we have 4K players+ai (in total) this can lead to pretty large loops.

Of course, the function is more complext than that, and yes, it skips the players that are not logged on, or on a different map, etc.
We discussed various optimizations, such as using threads, or possibly rewritting the whole function in assembly..

The server migration is complete

Well, sort of, there are still some people on the old server, see below.

The whole moving process was relatively easy, but not painless. One of the problems was that the old server was running FreeBSD 4.x and the new one is running FreeBSD 6.1
While they are both FreeBSD, the binaries didn't work due to some missing libraries. So we had to recompile all the stuff, which included the programs that facilitate viewing the player statistics, the guild dump program, the Small/Pawn compiler, and various other utlities we have. Doing all that took a lot of time, as we had to modify quite a few things, misplaced some makefiles, and so on.

One other difficult aspect was the fact that the new server had a different address, so we need some player interaction, where they had to manually change the server address. Now, this is a very trivial thing to do, and takes a few minutes at most. I incorrectly assumed that everyone that uses a computer knows how to edit a text file. Well, this is not quite true. Besides for multiple and ample warnings, given a week before the server change, teaching people how to do it, many people didn't give a shit about it, and didn't even know we were going to change the server.

After transfering all the data on the new server, and after restarting both servers (the old and the new one) I wanted to make it obvious that the old server is the old server, and that the new server is the new server. So I modified their respective opening screens, to denote which server is which. On the old server, I even put the URL to the forums post explaining how to change the server address.

Obviously, that was not enough, as quite a few people just ignored the opening screen. Some people didn't know the English language, so we sent some people speaking Portugese, Spanish, German, Dutch, Polish, etc. to explain their compatriots about the server change. Some player even made a script to announce that all over the server, in 5 languages (I gave him mod access so he can send global messages every 5 minutes or so).

[17:30:42] #Message from ttlanhil: This is the OLD server. You MUST go to the new server to play EL. Use the command '%server_address game.eternal-lands.com' and then exit and restart EL.
[17:30:54] #Message from ttlanhil: Dit is de OUDE server. Je MOET naar de nieuwe server gaan om EL te spelen. Gebruik het commando '%server_address game.eternal-lands.com' en dan herstart je
[17:31:06] #Message from ttlanhil: Este servidor Ú antigo. Devem mudar para o novo. Usem o comando '%server_address game.eternal-lands.com' e entrem novamente
[17:31:18] #Message from ttlanhil: Ez a rÚgi szerver. Az ·j beßllÝtßsßhoz add ki a '%server_address game.eternal-lands.com' parancsot Ús lÚpjki, majd vissza
[17:31:31] #Message from ttlanhil: To jest stary serwer. Aby grac musisz przeniesc sie na nowy serwer. Uzyj komendy '%server_address game.eternal-lands.com', nastepnie wyjdz i restartuj
[17:31:44] #Message from ttlanhil: Das ist der alte Server. Zum spielen muesst ihr auf den neuen Server gehen. Benutze '%server_address game.eternal-lands.com' und starte neu


Now, every resonable person would think that this is enough. Afterall, the messages were clear; type: %server_address game.eternal-lands.com and then restart the client, and you are on the new server.

Well, it turns out that one idiot was ignoring all those messages, because, in his oppinion, they were annoying. Her eis the dialogue (his name was edited out to protect him).

[17:35:08] idiot: can i have free stuff
[17:35:17] idiot: plz
[17:35:19] marthman: you need to switch to the new server
[17:35:30] idiot: whats that
[17:35:59] marthman: have you noticed this blue messages comming over and over again?
[17:36:10] idiot: ya
[17:36:32] marthman: and have you read them? :)
[17:36:56] marthman: this server here is the old one. all exp and all stuff you get is lost. You need to go to the new server
[17:36:57] idiot: no not really i think they get annoying
[17:37:12] marthman: this is the reason!! :) ppl should read them
[17:38:10] marthman: just type %server_address game.eternal-lands.com
[17:38:20] marthman: then log off and log on again and you are on the new server
[17:38:42] idiot: wat was the first thing
[17:39:03] marthman: what first thing?
[17:39:17] idiot: the fist thing you said
[17:39:45] idiot: to get to new world
[17:39:45] marthman: i asked you if you need help to switch
[17:39:56] idiot: ya i do
[17:40:17] marthman: oh. just type: %server_address game.eternal-lands.com
[17:40:32] idiot: wats that mean?


So as you can see, we had a lot of fun...

One last thing:
The new server is a Dual P4 Xeon 3Ghz
The old server is a Dual P3 1.2 Ghz

I expected the new server two be at least twice as fast as the previous server, and it is faster when no users are logged in (under 1% CPU, while the old server uses about 3%). However, when the players start to log in, the CPU usage is comparable with the previous machine, which kind of puzzles me..
We run all kind of stress tests, and noticed that with 2K pending connections, it went as high as 50% CPU. So we need to look further into this issue, because I am not happy with it at all. I have some ideas in mind, but will post about them next time.

We finally got the new server

Today our host installed the new server in the datacenter, and me and Learner were playing with it. We (actually, more him than me) made it so that we can have over 1K connections at the same time. The default limit is 1K, but at least on FreeBSD 6.1 (and some other versins as well) it is trivial to increase the limit. All you need to do is compile the SDL_net and your application with a special flag, and that's it.

We also tested the limit, and we were able to successfully create 2K connections. Not bad at all.
This new server is a dual Xeon 4Ghz, 1GB RAM, and 3 36 GB HDDs. The OS is FreeBSD 6.1
Apart for having to make some very minor header modifications, it seems that everything is working fine. I want to have a 1 week testing first, so we made that new server the official test server, shuting down the port 2001 test server. The idea is to check if everything is stable, as we are using a new OS version and a totally new hardware.

After everything is tested, we will have to migrate the servers, which will be done like this:
1. We will modify the downloads to include the new address in el.ini
2. We will announce everyone a day or two in advance on how to modify their el.ini to store the new address.
3. The day of the migration, we will shut down the old server, do the backups, transfer the data and stuff, then change the opening screen to tell everyone to modify their el.ini to change the address. After a few days, the old server will be totally offline, to encourage people to go on the new server. A proeminent message will also appear on the main page of our website, for those who missed the warnings and don't read the forums.

On an unrelated note, yesterday I told the players that all the hydrogenium bars donations for the invasion warning system will be matched by me. So far, they donated over 80 bars, which is quite impressive, so with my matched donations, there will be about 200 bars out of 500. So maybe in a month or so it will be done :)

Bluetooth headset modification

Ok, this is not really 100% home made, because it's just a modded "behind the ear" BT headset, but it was still a cool and rewarding project.

Why did I do it?
Well, the original BT headset was kind of crappy. People were complaining that they can't hear me properly, and the audio volume was not the best either. Plus, wearing it behind the ear is not very comfortable, at least not for me.
Another reason is that I like to hack things, just for the fun of it.
I want to mention that this is a prototype, so it's not looking as good as it could. I use it only at home, so the aesthetic aspect was totally unimportant for me. All I cared about was for it to be comfortable and reliable.

Warning:
1. I guess it is obvious that I am not responsible if you decide to give it a try and damage something/someone.
2. While working with Lithium Ion batteries, ALWAYS wear eye protection. I usually don't follow warnings, but this is a warning I ALWAYS follow. Those batteries can be very mean, and if you shortcut them they can explode and burn stuff. While burning your hands is unpleasant, it is at least repairable. Your eyes are harder to repair.

Thing you will need:
1. A BlueTooth headset that you don't care about.
2. A pair of headphones with a microphone. The pair I used was broken, and they didn't work (I suspect the problem was a broken wire near the connector).
3. Optional: A bigger Lithium Ion battery.
4. Optional: A solar panel that can output ~6V and a blocking diode.
5. Optional but very useful: A multimeter.
7. Soldering iron, solder.
8. A pliers, screwdriver, anything you can use to open the damn thing. More on this later.

The BT headset I used is a Logitech HS02-V07. It is pretty cheap, usually under 30 USD.
Now as you might guess, each headset is different, and might have different requirements for a microphone. So there is a risk that the microphone your BT headset uses is totally incompatible with the one in your headphones, in which case you will get bad noise, bad sound, etc. So again, this is a risk you have to take. But my model worked fine with the microphone from my headphones.

The first step, which is also the most dangerous, is to open the BT headset. They are pretty small, and usually they don't have screws. Everything is glued together.
What I did was get a pair of pliers and press on the region near the microphone at an angle that caused the glue bound to break. Then I inserted a flat screwdriver and pushed the case aside. This operation requires a great deal of attention, because everything is so small and a wrong move can damage the circuit board.
Here is how the opened headset looks like:


Next we will focus on the headphones. What we want to do is to cut the wire leading to them about 15 CM from the headphones end. Most of the headphones with a microphone have 5 wires, 3 for the headphones, 2 for the mic. My specific pair had the mic wires together, in an isolated casing (see the picture for details)




What you want to do is group the wires together, the ones for the speakers, and the ones for the microphone. The speakers have 3 cables, one for the left, one for the right, and one is the ground for both of them (common). So a simple way to connect them is by ignoring the ground wire, and connect the BT speaker output to the left and right headphone speakers. This will have the effect that they will be connected in series with your BT headset, which doubles their impedance. Don't worry about that, a mismatched impedance in this context is more that adequate for voice. But of course, you can connect them in parallel if you want, or just use a resistor in parallel with the speakers to reduce the impedance.
Connecting the microphone is easy as well, just remove the original microphone and connect the wires to the headphone's mic wires.



Now that we have the wires connected, it's a good idea to do a test. Start the BT headset, connect it to whatever device you are using (PC, PDA, cellphone, etc.) and see if you get audio in the headphones, and then try to call someone to evaluate if the mic is working properly.
Alternatively, if you are using a computer, you can start Sound Recorder (or the equivalent for your OS), set the input and output to use the BT connection, and record something then listen to it. If you hear a lot of noise in the background, don't worry, Skype for example removes all the noise.

The following steps are optional, and not necessary for the project, but it's cool nonetheless.
The original battery was very small. This is not surprising, as the size and weight of the headset is very limited. No one wants to put a brick on their ear.
I don't know how many mA the original battery has, but I would guestimate that not more than 300 and not less than 200. So while this battery can run your headset for quite a while, why not use a bigger battery, especially if you have one that you don't really use?
It happened that I had a bigger Lithium Ion battery from an MP3 player that was broken. I don't know how many mA it can store, but I guess it's about twice the capacity of the original battery.
I also happened to have a few unused 6V solar panels, and since they are pretty small and lightweight, I thought it would be cool to use one on my new headset, just for the hell of it. I mean, who else has a solar powered BT headphone with a microphone? :)

So basically just glue the Lithium Ion battery on the back of the solar panel, as shown. A multimeter is very useful, you want to wire the + of the solar panel to the + of the battery, and the - to the -. A blocking diode should be placed between the battery and the solar panel to allow the electricity to come in the battery, but not to go out in the solar panel. A multimeter is useful to determine that.
See the image on how mine looks like:



What we need to do next is cut the wires from the original battery, and connect them to the new battery. Usually red is +, black is -, but just to be on the safe side, use the multimeter to determine the polarity.
You might need an 'extension cable' to go to the new battery, depending how far the circuit board is from the new battery.
One side note, if you are using a solar panel and a blocking diode, be sure to connect the wires from the circuit board battery input directly on the +/- of the new battery, before the diode. Otherwise the diode will prevent the current flow to your BT.




We are almost done now. Do a final test to make sure everything works, then glue the battery to the headphones top. Note: if the bluetooth won't start, connect the power supply to the power port, then try again.
One other thing, it seems that at least in my case, removing the battery cleared the memory so now my computer won't ask me for the pairing key. I am not sure if other headsets will do the same thing.

Now all you have to do is to use some adhesive tape and glue the extra wires to the frame of the headphones, so they won't hang allover.



Conclusion:
My device is a prototype and looks like one, therefore it can be considered ugly. That doesn't bother me, as I use it only at home, for Skype. But if you like cute looking devices, you have various options, such as building a small box for the project, cut the wires at the exact length so they won't hang out, etc.

The sound quality is very good for VOIP, but if you think about listening to music with it, forget about it and get some stereo BT headphones. This was not meant for anything but voice.
The sound is louder, and since the microphone is closer to your mouth, the other party can hear you better as well.
The whole project took about two hours, and it was pretty fun. So I recommend it to others. And remember, even if you fail, you still get some experience :)

Your own MMO development kit?

It would seem that the MMO market might get a boost with this new technology. Their offer is pretty interesting, but, nevertheless, making a MMO is still a lot of work, even if you have an engine and artwork. And mainaining it is also a lot of work, and requires a lot of experience as well.

Anyway, the invasion system is complete and tested (the testing was not very extensive, but I did check most of the stuff). I'll upload it today on the test server and restart it somtime this evening, so people can also do a little testing, and if everything works fine it will go on the live server by the end of the week.

Recently I was looking for a single player game engine, because I would really like to work in my sparetime at RPG game.
What I am looking for is an engine that meets the following requirements:
1. Multiplatform.
2. Integrated development tools (map editors, terrain editors, importers, etc.)
3. Resonably priced (under 5K USD)
4. Have an embeded scripting language that can be used to create a full game.
5. Good documentation.
6. Resonably wasy to use.

The Torque engine seems pretty close to what I want, but I couldn't find what fuctions does the scripting language provide. For example, does it have stuff such as "line of sight" or "path finding" available to the scripting language?
I e-mailed them asking a few questions, and I got no response at all, which was dissapointing.
Other engines I checked were C4, A6, Crystal Space... But none match my needs.
Originally I wanted to modify the EL engine for a single player game. Unfortunatly, it lacks a lot of stuff that is hard to implement, such as collision detection, scripting support (I'd have to implement it all by myself) and various other goodies.
But eventually I might have to just do it myself, since there seem to be no other options.

The invasions, almost compelete

The invasions and warning NPC are almost complete. I tested them today and except for a few bugs, they work great.
I didn't totally finish them, but tomorrow they should be done, so they will be on the live server this week.

The PK contest went pretty well, although it seemed quite short. It last 30 minutes, but those 30 minutes went by very fast. A video of it is available here.
I guess we will have another PK contest, with different rules, in 2 weeks or so.

Meanwhile, I managed to fuck up my back, I guess it was due to a bad matress combined with lifting a crate that wasn't even that heavy. Now if I try to bend, I have a sharp pain in the left side of the body, in two places. FIY, the correct way of lifting anything is to go near the object, bend your knees, then extend your hand and pick it up, with your back straight. Even if you lift a paper from the floor.
It will eventually go away, but this is a good lesson, so remmeber kids, lift stuff the proper way.

The last few days

I just order the Iron Maiden's last album, "A Matter of Life and Death" from Amazon, and I can't wait to get it. I listened to only one preview song from this new album, Reincarnation Of Benjamin Breeg, and although I didn't particularly like it more than other songs, Iron Maiden is one of the very few bands that I buy albums without first downloading the songs off various P2P networks.
I know that Iron Maiden can't dissapoint, that is impossible.

On Eternal Lands, I am working at the invasion system, and it should be done tomorrow or the day after, and then I need to test it, so if everything goes well it should be in the game somewhere next week.

Working at a MMO can be fun at times, but it's getting tiring. Especially when you have to deal with stuff like this. A bunch of 12-13 year old kids thinking they can outsmart the man with the logs. It is funny, but since similar things happen every day (usually they contact me via PM), it's getting annoying after a while.

Tomorrow (actually today, it is already Friday now) we are having the PK contest/event. It's going to be a big event, a lot of people bought rostogol stones and stuff for it, no one wants to die and lose their expensive gear. On the other hand, they want to bring their best gear in order to survive and have a chance to actually win the contest.
So I will grab a cup of coffee, make some popcorn, and watch the slaughter :)

Random Invasions

One cool but hardly original idea in a MMO is to have monsters invade cities every once in a while.
That's good for a few reasons:
1. Adds more fun to the game (yes, you can die and lose your stuff, but FFS, it's just agame)
2. Promotes 'involuntary exchange of items' (you die, someone takes your stuff) :D
3. Brings players together to fight against those monsters.

One more or less revolutionary thing in how it will be done in EL is that there will be an NPC where the whole community will be able to donate money in order to have an advanced warning system. There will be an initial building cost, and then there will be a maintanance fee. Depending on how much the players are willing to pay for the maintanance fee, the advanced warning system will be more or less useful. For example, if they pay the minimum cost, the system will announce the invasion when the monsters are invading a main map. On the other hand, if it is well maintained, it will warn two minutes before the monsters spawn in a not so main map, which will help the fighters to go and kill them before they spawn on the main map.

The Random Number Generator

I am writting this entry offline, because my sweet, beautiful DSL is down. It's been down pretty much the whole day, which is really annoying. Not only that I can't do my normal surfing, but I can't give people the stuff they ordered.

The good thing is that since I can't go online and waste my time, I can do other things, such as actually working at the game :)
I did a lot of stuff today, but the most important is that I finally decided to test the random number generator.
After implementing a new 'feature', where some items break during the manufacturing (items such as the hammer, the molds, needle, etc.), I wanted to test and see if it works, so I set the chance of the ring mold to 5000 out of 10000 (so 50%), and starting making rings. To my surprise, it appeared that the chance of the items to break was aparently much higher than 50%. In my estimates, it was something like 60-70%.
The code was pretty simple, even trivial. There couldn't have been a mistake in the code:
if(required_to_have_1!=-1)
{
rand_no=my_rand(10000);
if(rand_no<items[required_to_have_1].damage_when_hit)
{
//secret code, but basically break the item
}
}
As you can see, there is little room for error.

The function my_rand() is OS specific. Under Windows, it uses rand(), while under Linux and BSD it uses random()
Here is the code:

int my_rand(int max)
{
#ifdef WINDOWS
return rand()%(max+1);
#else
return random()%(max+1);
#endif
}

Yes, I know, there are better ways to do it than using %.

So I decided to write a test function:

void test_random_generator()
{
FILE *f = NULL;
int treshold=80;
int count=10000;
int rand_no;
int i;
int zero_count=0;
int one_count=0;
char str[200];

for(i=0;i<count;i++)
{
rand_no=my_rand(10000);
if(rand_no<treshold)zero_count+=1;
else one_count+=1;
}
sprintf(str,"\nCounting to %i, treshold of %i, we got zero: %i and one: %i",count,treshold, zero_count, one_count);

f=fopen("debug.txt", "a");
fwrite(str, strlen(str), 1, f);
fclose(f);
}

The results were:
Counting to 10000, treshold of 5000, we got zero: 5420 and one: 4580
Counting to 10000, treshold of 5, we got zero: 5 and one: 9995
Counting to 10000, treshold of 5: 6 and one: 9994
Counting to 10000, treshold of 80, we got zero: 92 and one: 9908
Counting to 10000, treshold of 80, we got zero: 89 and one: 9911

This means that the higher the treshold is, the higher the bias towards the end. However, in EL, most of the random tests are compared towards the 0 range. In addition, I heard that random() is somewhat more reliable than rand(), but at home I only have Windows, so I will have to test it on our BSD machine as well, when Mr. DSL will decide to work.

If the BSD machine will show a similar bias as well, I guess we will eventually have to use something better than %.
Nevertheless, the result of my tests are encouraging, and the multiple reports from the players on how stuff breaks more than it should are most likely due to bad luck, not due to errors in the code.

Moving the webserver to it's final resting place

Or, hopefully the final resting place.
The dedicated machine in France was set up thanks to our great game server host, Klug. He got us a deal with some other French company, which offer dedicated web servers much cheaper than he does. He also set up most of it, and Learner helped too (in fact, did most of the work related to transfering the sites).
I modified the DNS about 30 minutes ago, so now we are waiting for the changes to propagate.

Once this whole thing is over, we will have to focus on moving the game server as well. That will be much more complicated, as we have to eliminate the 1024 connections/process limit. We will be using a newer FreeBSD kernel, so hopefully there will be no problems related to that.

Meanwhile, I am working at making the crown of mana and the crown of life manufacturable. They will be under Crafting, and will require quite a few rare items to make.