Make Tests Fast Again - Development Log #545

In this week's devlog Michi reports from simulogics HQ, improves the performance of the test suite and fixes the inflation command.

Avatar Michi

Michi (molp)

This week I was visiting simulogics HQ. Usually Martin, Fabian and I use these rare in-person meetings to talk about technical topics and game design, but this time we welcomed a new team member. Ipek joined simulogics as an intern! Welcome! She will work on an AirlineSim map feature, and you can read about it in AirlinSim's weekly devlogs.

I spent the remainder of this week on smaller issues.

The server project of Prosperous Universe now has roughly 1200 automated tests in its suite. They range from small unit tests to larger actor tests and even full integration tests. A unit test covers a very small part of the code base, for example making sure a behavior like the fleet behavior correctly processes certain elements of space flight. Actor tests are way larger: They spin up a test universe with a handful of systems and their corresponding planets. They can be used to test the behavior between entities within the game. A typical use case is for example: start the world, set up a company, gain PRO license status, build an admin center, run for office, vote and make sure we actually become a member of parliament. Integration tests go even further and spin up a database, so we can validate that the correct data is written to the database. We use them very sparingly though.

As you can imagine, the actor tests, while extremely helpful, take quite some time to run. On my development machine the whole suite takes about ten minutes to run. In and of itself this wouldn't be a big problem: I can just run the few tests that belong to the feature I am currently working on, but eventually I want to check that I didn't break anything outside the feature's scope.

So I took the time and dove into the performance of the tests. I was able to reduce the runtime to about five minutes on my machine by optimizing test-related settings and reduce polling retries. For example: an admin center tries to determine its currency once it is built. If the first try doesn't work, it'll retry five seconds later. On production that is no issue, but potentially waiting five more seconds in every automated test that contains an admin center quickly builds up. I was also able to fix a handful of flaky tests that failed sporadically.

I also looked into the INFL command again. There was a discussion on Discord a few days ago, that the inflation reports are off. It seems that a clearing of the order book caused a massive spike in the inflation data and on the next day the prices went back to normal. Since the weight of a period (day in that case) was based on the trade volume the spike disproportionately increased the inflation values. Even when prices went back to the previous level the next day, that period has a much smaller weight, because it didn't have that volume behind it. The change I made, and this is already live in production, is to not use the period's trade volume as a weight, but rather a sliding window of the last 30 days' worth of trade volume. This should buff out these huge spikes. The change was applied retroactively, so the spike from last week should be gone now. Check out the INFL command and let me know whether you think it more accurately represents the actual inflation.

As always, we'd love to hear what you think: join us on Discord or the forums!

Happy trading!