As a part of our regular maintenance, we have migrated a hypervisor from one rack to another rack on Tuesday at 13:17. We have changed our port configuration to reflect this migration, but unfortunately, we have made a mistake in the configuration. This caused the VPSs in both racks to receive IPv6 addresses from both racks, which caused a routing issue for IPv6.
We immediately issued a rollback to the previous configurations, but since the additional IPv6 addresses were already advertised, this did nothing to remedy the situation.
We have ultimately corrected the configuration of the switch and removed the additional IP addresses from the hypervisors at 11:21 on Wednesday, which resolved the immediate issue, but did require a restart of the network service on the VPSs.
This obviously is much longer than we would have liked. Unfortunately, we were set on the wrong path because a rollback didn't yield the desired result, and the configuration error was very easy to overlook.
This was further complicated by the fact that this issue shouldn't be possible: our switch shouldn't allow machines to be in two pods at the same time. It seems we have run into an issue in the firmware of the switch itself.
Following this change, our name-servers in one of the pods have been been unreachable because they also had an incorrect IPv6 address. We have fixed this issue at 13:29, after which everything became stable again.
We have updated our procedures to prevent this from happening again, and updated our monitoring to be able to correct it up quickly if it happens nonetheless.
We apologise for any inconvenience this may have caused, and strive to do better in the future.
[April 7th - 1:31 PM]
At 1:30 PM the issues with the DNS-resolver has been resolved.
Further information will follow soon in this article.
[April 7th - 1:15 PM]
At approximately 12PM we've received reported that customers are have issues resolving DNS requests. Our engineers are currently looking into this issue; please follow this page for additional updates.
We suggest temporarily changing the DNS servers to a substitute;
Some useful substitutes are:
- Google DNS (188.8.131.52 / 184.108.40.206)
- Open DNS (220.127.116.11 / 18.104.22.168)
- CloudFlare (22.214.171.124)
If you have any questions please feel free to contact us on firstname.lastname@example.org