Print

website security lock

Our Rust server has now been running for almost a year and it's time that we do some maintenance on the physical server itself. While this maintenance shouldn't take much more than a few hours we are making some BIG changes to the operating systems and networking structures installed on this server, as a result we have scheduled an extended maintenance window to ensure that if we run into issues we have time to sort them out.

The server will go offline somewhere around 9-10am UTC on the 2nd November and will return to operation after the forced map wipe on the 3rd-4th Nov as usual. 

NOTE: The server may show as online prior to this but will be passworded during a testing phase. Once the password is removed you are free to use the server

Current Status

Task  Status
Waiting for Rust to empty (no players) COMPLETED
Backup Rust files (8Gb) to offline source  COMPLETED
Backup mySQL DB running player notices COMPLETED
Wipe and Reinstall base operating system COMPLETED
Update pioneerx DNS servers COMPLETED
Create virtual networking (switches, firewalls, routers, etc) COMPLETED
Create Rust virtual machine COMPLETED
Reinstall Rust instance COMPLETED
Test Rust COMPLETED
Install Host and VM Monitoring COMPLETED
Open for community COMPLETED

 

UPDATE: A missed setting on a virtual firewall had us chasing our tails for a few hours but that was finally identified and resolved. The server ran fine before the mandated update but failed after the update. On further investigation we found this was not an issue on our side but a problem in the linux server issued by FacePunch. We have implemented a workaround as detailed on the Facepunch Forum and this appears to be holding. Our server checks every 10 minutes for an update from Facepunch so may restart unexpectedly if they issue a fix outside the normal schedule.

Other than this we are up and running as expected.

For those interested in more details of what we are doing see below....those not interested in the technical side dont need to read further.


For Rust, our current hardware consists of a rented HP Blade Server with Dual E5420 cpus giving us 8 cores @ 2.5Ghz across two processors. In addition we added 16Gb of high bandwidth ECC RAM on top. We have Debian Linux installed as the operating system and our Rust server is running directly on top of this. This has two issues, the first being we ideally want a layer of security between the Rust servers public access and the underlying operating system. While we have some of this it is now a year old and could be done better. The second relates to the wasted power of this physical machine. We spend a lot of money renting this amount of power (thanks for all those who donated to the running costs) and our Rust server uses ~10% of it. The rest of the power gets wasted since at this time we cant run other things on the server and still protect the resources allocated to Rust. For example if we also put ARK-Evolved on the server today it would be possible for ARK to consume all the RAM/CPU and therefore negatively impact the Rust server.

We want to fix both these situations so we are installing virtualisation onto the physical hardware. This will allow us to split up the one physical machine into multiple virtual machines and therefore allocate dedicated resources to various things. This will also allow us to run the network traffic through filtering and firewalling better than we do today and therefore better protect our service.

virtual architecture

The 'Traditional Architecture' shows our current setup with 'Application' being our Rust server. The 'Virtual Architecture' shows how we can make us of the 90% extra power our server has by allowing for more 'Applications' to use this extra power. This is not a new thing for PioneerX as our core network (where this website is for example) is already running on virtual machines. The biggest difference being that the PioneerX core network consist of 5 dedicated server machines that are clustered together to form one very large super machine with the virtual machines running on top of that. We are NOT adding this physical machine to the PioneerX network because of the physical location difference. The PioneerX core network is located in a data center in the UK where as our Rust server is located in the USA. Maintaining a full cluster arrangement across the Atlantic introduces complications that are unnecessary to us at this time. However the virtualisation software we are using allows us to include this feature at a later date if we desire.

We have a lot of system monitoring in place today that allows us to see the real-time status of all our systems and keep an eye on things so we dont overload any one system. 

zabbix

This monitoring will be installed at both the base hardware level and the virtual machines that exist above it therefore allowing direct real-time stats for both the physical hardware and the virtual hardware above it. If you would like more information or advice on how you could also do such things please feel free to leave a comment, support ticket or forum post and we will be happy to explain.

Kubik-Rubik Joomla! Extensions