Collecting Garbage - Development Log #520

Michi talks about two performance problems and their solutions.

Michi (molp)

March is here, which not only means that the weather in Germany gets nicer, but also that we have a new monthly economic report for you ;)

Click the image for the full report

This week I looked into the servers' performance once more. I noticed that we were running at almost 90% heap usage. In and of itself, this is not bad per se, but it resulted in frequent runs of the garbage collector, which would use up CPU resources. In order to see if having more head-room regarding the heap would reduce the garbage collector cycles, I added another server to the cluster. It did definitely help with heap usage, but it consumes another critical resource: money. So I took a snapshot of the heap of one of the servers and started looking for potential optimizations. Eventually I found something: Many entities in the game have a reporting behavior that allows us to track game-related events (think company creations, tour completions, ..). We cache some parts of these events, so the database we send these to doesn't receive the same event twice or even multiple times. It turned out that we kept everything, instead of just a few-days worth of events that are necessary for that behavior to work properly. Over time some entities collected a notable number of events :) After deploying a fix and only keeping what is really necessary, the behavior is significantly leaner now.

I was also looking into another performance-related issue: The servers' logs contained a warning from time to time that a message could not be serialized because it exceeded the configured maximum size of a few megabytes. This was a rather odd log message, because usually our messages are rather small. There are a few exceptions, of course, like the message containing the information about all star systems. It took me a while, and two deployments of additional logging, to find the culprit. The CXM command shows commodity information across all commodity exchanges. It has a mandatory commodity ticker parameter and an optional planet parameter. If the planet parameter is specified, an additional column shows the distance from the specified location to the commodity exchange. Instead of not requesting any planet information if the planet parameter is not set, the command tried to acquire the full planet list with the details of ~5000 planets. Not only does it take time to collect that data, but it also creates a lot of garbage along the way that has to be cleaned up, unnecessary additional stress for the garbage collector.

The work on the faction influence foundation is coming along as well, I hope to be able to go into detail on the actual generation in the next devlog.

As always, we'd love to hear what you think: join us on Discord or the forums!

Happy trading!