[SOLVED] Issue with an upstream network provider 22/10/2014

22/10/2014 23:11

We are experiencing a serious network issue. Our network is partially unreachable from the internet. We are investigating this problem, we will give updates every 15 minutes.

++++

22/10/2014 23:22

Looks like one of our upstream network providers is experiencing problems. We're shutting down the BGP session.

++++

22/10/2014 23:25

The problem is solved by shutting down one of our upstream links towards an internet exchange in the Netherlands

The root cause is currently still unknown, but apparently we were not the only ones impacted by this incident.

++++

23/10/2014 0:18

The router of one of our upstream network providers was not functioning correctly anymore. After restarting the router the problem was solved at their site. We're not restoring BGP sessions at the moment in order to make sure an unexpected repetition of the problem doesn't impact Tilaa. When the router turns out to be stable again we will restore BGP sessions. Until then traffic will be rerouted via other upstream providers. We're expecting to restore BGP sessions for this upstream network provider somewhere tomorrow.

++++

23/10/2014 13:04

We have decided to keep the BGP sessions to this upstream network provider disabled until we are confident their routers are stable again and we have had a satisfactory post mortem root cause analysis of the incident.

++++

23/10/2014 22:20

We're glad we've made the decision to keep BGP sessions to this upstream network provider disabled since they're experiencing the same issues at the moment. The Tilaa network was not impacted this time.

++++

24/10/2014 16:30

Our upstream provider has notified us that they are going to replace the faulty hardware tonight during an emergency maintenance window. Because it's still uncertain that replacing the hardware will actually resolve this issue we will continue to keep the BGP sessions down over the weekend.

We will re-evaluate the situation on monday. If the situation is stable we will consider re-enabling the BGP sessions to this provider.

++++

27/10/2014 10:27

Our upstream provider has replaced the faulty hardware on 25/10/2014 0:00 and the network has proven stable since. We have therefore re-enabled BGP sessions to this provider. All connectivity has now been fully restored.

 

Have more questions? Submit a request

0 Comments

Article is closed for comments.