[Done] Emergency maintenance (Meltdown and Spectre)

As you are probably aware by now a number of critical CPU vulnerabilities have recently been discovered dubbed Meltdown and Spectre. If you haven't already done so you may want to visit https://meltdownattack.com/ for a high level overview of these issues. Long story short: These vulnerabilities can be used by attackers to read memory to which they would otherwise not have access (even of another VPS running on the same host system). Sensitive information such as private keys and passwords is therefore at risk.

Meltdown attack

Especially critical is the Meltdown variant to which all CPU's made by Intel are vulnerable. A mitigation has been made available which works around this issue by splitting up user and kernel page tables using a feature called page table isolation (PTI). We have deployed kernel updates containing this feature to all Tilaa infrastructure and made changes to our platform to expose the PCID CPU feature to a VPS which will limit the performance hit caused by PTI, though despite that some reduced performance is to be expected.

Spectre attack

The two Spectre variants seem to be more difficult to successfully exploit but are unfortunately also more difficult to fix. The full impact of the Spectre vulnerabilities is still not completely known and work to mitigate Spectre attack vectors is still in progress by developers world-wide, including Intel and the Linux community. The fixes that have been made available for known attacks have been tested and deployed, but we expect more updates to become available over the next couple of months and will evaluate and deploy them as soon as we can.

Performance impact

It's difficult to determine the performance hit caused by the mitigation patches since it depends on the OS and the specific workload running on the VPS, but worst case you can expect to see between 10% and 30% performance loss.

Unfortunately there isn't much we can do about that except hope that future patches will restore some of the lost performance. You might have to (temporarily) upgrade the CPU's of your VPS to be able to restore its performance to previous levels.

Platform reboot

Due to the high risk of exploitation of these security bugs we have to patch these issues as soon as possible. Unfortunately this will cause you downtime, because this requires reboots of our host servers (to load a PTI enabled kernel and new CPU microcode) as well as the VPS's (to enable PCID). You can expect to see about one hour of downtime for each VPS.

The reboots will take place over the course of next week (January 15th -  January 19th) during working hours starting Monday morning (9:00 to 18:00 in the Europe/Amsterdam timezone). Our whole team will be standby throughout the week to help out with any issues that might surface.

Due to the complexity of this large operation it's unfortunately impossible for us to exactly determine when each host server will be rebooted. We will send out a maintenance notification before each host server reboot so that you are informed when the maintenance each specific VPS is starting. We will try our best to prevent nodes of high availability clusters to be rebooted simultaneously.

We understand that rebooting your VPS during working hours is inconvenient and we are doing our utmost to ensure you experience as little inconvenience as possible during this process.

Questions?

We trust we have informed you sufficiently. If you have questions, please contact our support department. You can reach us by phone, email and through our social media channels. Thank you for your patience and understanding.

Maintenance completed

All hosts have been rebooted and the maintenance has been completed. Fortunately the performance hit seems to be barely noticeable and the updates seem to be stable.

Have more questions? Submit a request

0 Comments

Article is closed for comments.