Hello all,
As the Halloween celebration continues it seemed that the server has been spooked so hard it’s stopped working on many occasions!
We have fixed a ton of related issues and added many optimizations, and this should result in a more stable gameplay experience for everyone, and with performance to spare for higher player counts.
All that focus on optimization did take some time away from creating Part 2 of our Halloween event, however… But don’t worry, there will still be a Part 3.
GHOST GULAG EVENT
— Part Two of our halloween celebrations!
The spirits of Gulag have staged an uprising! Throw pumpkins in the new Gulag scenario to give your opponents levitation. Race to the mini-nuke while hurling your opponents into the air!
This mode is available in Showers TDM, and Gulag KOTH flavors.
SEPARATE DATABASE SERVER
Previously, our MySQL database was hosted on the frontend server (the same server as the BungeeCord). When there were DDoS attacks on the server, the DDoS protection on our host was able to prevent the attack on existing players, but the protection algorithm also blocked the database requests of our backend servers. Hence the attackers were able to take down our PVP and Mob-Arena servers because they ran on the backend and could not access the database.
We have moved the server’s MySQL database to a separate VPS that only hosts the database and nothing else, and it’s firewalled off from the internet and only listens to traffics from our other servers. This means that everything will now stay up and work normally despite a DDoS attack.
DATABASE RECODE
We have noticed that there was a rare deadlock issue with our current database system causing a server to become unresponsive to player logins until it is restarted.
We have recoded the way data is accessed from the database, ensuring that synchronization is only done when necessary to avoid two simultaneous queries being executed on the same thread. This not only improves response times overall, but also avoids any deadlocks.
THREAD POOL ALLOCATION
To simplify matters, each player has 1 dedicated MySQL thread that handles all of their data loading and saving operations. Previously, which thread to use was determined by the first 3 bits of the player’s UUID. This was not perfect as players could be placed on an already-busy thread increasing response times, and some threads could be serving more players than others.
Now, we have recoded it so that a MySQL thread is dynamically assigned to the player when they log in. This algorithm prioritizes threads that are not already busy, so you can join a game server more quickly, and evens out the number of players handled by each thread, which reduces lag and avoids data saving issues.
The thread assigned to offline players will continue to be dependent on their UUID.
OTHER IMPROVEMENTS
- Updated anti-cheat, hopefully this will reduce the false positives (dragbacks)
- High Gravity skill reworked to simply disable jump on players. For mobs it will still drag them down to the ground
BUG FIXES
- The Discord NPC at spawn now provides the correct link to the Discord server
- Fixed a glitch which caused Quest Progress and Stats to not display properly sometimes (this also affected Time Played and PVP Kills shown in chat)
- Bandaid Fix to the glitch where the time remaining displayed in PVP is often incorrect and in the millions. We are still investigating the underlying issue, but this fix should give the correct behavior for now
- High Gravity skill will now work on mobs properly (its intended behavior is to drag flying mobs down, not make every mob fly up into the air…)
- Fixed a glitch where the Magnum Sniper and Schmidt Scout in the DustPCG scenario would glitch out if fired while scoped
- Fixed a glitch that caused the gun plugin to rarely be loaded improperly on server start
- Fixed a glitch where the Halloween event NPC did nothing when clicked
- Fixed a glitch where weapons with skins had the Shooter kit in Mob-Arena, instead of the correct kit for the weapon
- Fixed a console glitch with the chat packet cancelling code in our plugin
- Fixed a glitch where spawning in Mob-Arena for the first time would still cause a lag spike to the entire MA server
Hotfix 10/28/20:
We have fixed a data saving glitch that arose due to the update, as well as many fixes aimed at reducing recent occurrences of bandwidth-related lag (both serverside, and clientside on players with bad internet).
- Fixed a glitch where database accesses were sometimes in the wrong order, which caused inventory saving in particular to be inconsistent (this mainly affected quest rewards and crate opening)
- Completely rewrote bullet tracer particles with the aim of reducing bandwidth
- The gap between two bullet tracer particles is now increased based on the angle of the bullet relative to you to send less particles & save bandwidth, while still ensuring that the tracer line looks continuous
- The range of bullet tracer particles has been increased to 230 blocks (from 200) for the player firing the gun, but reduced to 180 blocks (from 200) for other players
- If the bullet tracer will stay outside a player’s FOV for the entirety of its flight, no tracer particles will be sent
- Other particle lines (such as the supersonic bullet particle) will also no longer display if outside your field of view
- network-compression-threshold for the BungeeCord server lowered from 256 to 128 to reduce bandwidth usage for players with poor connections
- For the lobby server, which runs on the same machine as BungeeCord, network compression has been disabled to save CPU
- Game servers, which are more bandwidth limited than CPU (by far), have been set to compress more packets
- Reduced PVP server view-distance back to 12 (from 16). It was nice to see further in certain maps (e.g. AlpineTrench), but these maps also tended to be the ones that lagged the most, and one’d take performance over render distance any day.
- Created a new network profiler which counts packets of each type getting sent to a player, intended to combat any other bandwidth-related issues
