We are experiencing a serious network issue. Our network is partially unreachable from the internet. We are investigating this problem, we will give updates every 15 minutes.
Looks like one of our upstream network providers is experiencing problems. We're shutting down the BGP session.
The problem is solved by shutting down one of our upstream links towards an internet exchange in the Netherlands
The root cause is currently still unknown, but apparently we were not the only ones impacted by this incident.
The router of one of our upstream network providers was not functioning correctly anymore. After restarting the router the problem was solved at their site. We're not restoring BGP sessions at the moment in order to make sure an unexpected repetition of the problem doesn't impact Tilaa. When the router turns out to be stable again we will restore BGP sessions. Until then traffic will be rerouted via other upstream providers. We're expecting to restore BGP sessions for this upstream network provider somewhere tomorrow.
We have decided to keep the BGP sessions to this upstream network provider disabled until we are confident their routers are stable again and we have had a satisfactory post mortem root cause analysis of the incident.
We're glad we've made the decision to keep BGP sessions to this upstream network provider disabled since they're experiencing the same issues at the moment. The Tilaa network was not impacted this time.
Our upstream provider has notified us that they are going to replace the faulty hardware tonight during an emergency maintenance window. Because it's still uncertain that replacing the hardware will actually resolve this issue we will continue to keep the BGP sessions down over the weekend.
We will re-evaluate the situation on monday. If the situation is stable we will consider re-enabling the BGP sessions to this provider.
Our upstream provider has replaced the faulty hardware on 25/10/2014 0:00 and the network has proven stable since. We have therefore re-enabled BGP sessions to this provider. All connectivity has now been fully restored.