3/13/2023 - devlog
Michi recounts the complexity of last week's maintenance release, and your swag is in the mail!
Michi (molp)
This week we released the first maintenance release of the year. You can find its release notes on the forums.
If you have been playing (or at least tried to do so) on release date, you noticed that the game has been extremely sluggish and was even unavailable for the better part of the day. Here is a quick explanation what was going on and how we fixed it.
A week or two before the release happened, I noticed that the faction agents, that offer contracts to all players on a regular basis, had been very busy and thus required a big amount of memory on the server. That also means, that they are made up of thousands of events that had to be reloaded from the database once they restarted. These restarts usually happen when the servers restart as well. So, in order to reduce the loading times, I decided to implement snapshotting for them, just as we have done for many other types of game entities in the past. What the snapshotting does—is basically to store a snapshot of the current state of the agent. Once it needs to load from the database, it first loads the latest snapshot plus the events that happened since. This makes the loading process way faster than just loading all the events.
The problem was though, that two out of the twenty faction agents got so big, that once they were loaded completely and started creating a snapshot of themselves, the corresponding server node ran out of memory and stopped working. The system then marked that server node as unreachable and tried to move the load to another server node, effectively rendering that node unreachable as soon as it managed to load the faction agent and tried to create a snapshot. You can see that is a kind of cascading failure that led to the extended downtime on release day.
The solution was to disable snapshotting for faction agents for now. Most of them have successfully created a snapshot and can be loaded from that snapshot, so we do have the performance gain from these. Of course, the underlying problem is not solved yet. So right now I am trying to improve the situation on two fronts: First, I want to improve the serialization / deserialization performance in general. It has been implemented for events, that usually are only a few kilobytes in size, not snapshots of a hundred megabytes. There is definitely room for improvement. Second, the faction agents have many contracts that they don't need anymore (players left the game for example) that could be terminated and thus reduce the agents memory footprint.
Nick
Last week I had a fun conversation with Odysseus/AnotherOneAnother, where we worked out some details about the Prosperous Universe Lottery community event. I will be assisting with selecting the winning tickets at random. This will absolve anyone that may have a stake in the lottery from drawing the winning tickets and will be a nice opportunity to also have some team interaction with the players. I will have a fun time making the video for the winners this week. If you haven't gotten your tickets yet, make sure to check out the event here!
Aside from that, we are finally at the end of the Name THAT Planet event! We now have a list of name for OT-889b, so go on over and cast your vote for your favorite planet name. The winning name will be announced soon!
Martin
I wanted to make a rare appearance on the devlog to let you know that I handed over the second batch of goodie bags to DHL last weekend. This means that all goodie bag shipments that were pending have left simulogics HQ. From now on, we'll likely ship perks in smaller batches and sooner after the respective support tiers have been purchased. If you want to receive our incredibly cool swag as well, make sure to purchase a Star support tier or higher!
To anyone who's purchased a Cluster tier or higher: Note that art prints (both exclusive and non-exclusive) are not part of the goodie bags and will get shipped separately.
And last but not least, if you are a Star supporter or higher but haven't received your goodie bag yet, make sure to enter your shipping address in account management.
As always: we'd love to hear what you think: join us on Discord or the forums!
Happy trading!