Monday 13/09/2010, 16:52
We did an operation on the servers between 4 am and 11 am to solve the massive slowdowns problem. During this period of time we had to deactivate missions so you can keep playing.
Some more explanations about the problem :
The number of missions unlocked by players is permanently increasing. For some time now, when a certain number of players online is reached, the server doesn’t have enough room to keep all necessary data in memory. It had to keep on doing round trips to the hard disk (which is far slower than the memory). The result is that it was taking more and more time to give back missions and the other data were pending.
After a lot of optimizations, observations, false bright ideas, real bright ideas... we manage to isolate the problem and then find the appropriate solution.
Last night, we changed the data base structure so the missions requests go faster. We also reorganized the server to give more memory (double it) to the database so it can keep more data in memory.
We will also duplicate this server and split «the work» on several machines. We will optimize the data structure to avoid the slowdowns when there will be even more players than now.
We apologize for the inconvenience. Be sure we are doing our best to solve these problems as fast as we can. Unfortunately, some problem are more difficult and longer to solve than others.