Summary
Our Service Assurance team have reported that 2 wholesale partners experiencing 50% loss when trying to connect to customers across into our network. This is effecting customers via the LTS at Harbour exchange. Initial investigations have found no issues. However, A partner have brought down their BGP tunnels and brought them back again to no avail. They are advising that they have customers over the link working ok however once they drop and instigate a new tunnel thatâs when the issues start . This started to happen around 23:00 yesterday evening.N/A
Latest Update This incident has been upgraded to a p2 incident due to the impact this is happening on the partners. A further update will be provided once NOC have done further investigations. Â
Latest Update This incident has been upgraded to a P2 incident due to the impact this is happening on the partners. A further update will be provided once NOC have done further investigations.
Latest Update NOC have engaged with Network Support regarding the issue. A further update will be provided once investigations have been done. Â
Latest Update NOC have engaged with Network Support regarding the issue. A further update will be provided once investigations have been done. Â
Latest Update This incident has been upgraded to a p1 incident due to the impact being seen and partners reporting this issue. A further update will be provided following a conference call with MIM , IM , B2B , NOC and Network Support. Â
Latest Update A bridge call has been set up with TT MIM, NOC, Netowrks and TTB. We have been advised that a number of partners and circuits have been impacted including Verizon, Andrew & Arnolds, GCI, Maintel, Fluid One, Timico, Daisy, Fujitsu. Networks have investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors. In order to restore service, the Network support team reloaded card 10 which have now cleared. TTB are now engaging with customers to retry and confirm service is now restored. Â
Latest Update A bridge call has been set up with TT MIM, NOC, Netowrks and TTB. We have been advised that a number of partners and circuits have been impacted. Networks have investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors. In order to restore service, the Network support team reloaded card 10 which have now cleared. TTB are now engaging with customers to retry and confirm service is now restored.
Latest Update A number of wholesale partners have now confirmed service has been restored and they are now back to BAU. We are currently waiting on a further partner to confirm who are in the process of testing. The Networks Support team have advised they have also raised a case with third party vendor Juniper under reference 2020-0123-0070 for card 10. Â
Technical / Suspected Root Cause Our Service Assurance team had reported that around 23:30 last night (22/01/20), wholesale partners were experiencing 50% loss when trying to connect to customers across into our network. This was effecting customers via the LTS at Harbour exchange. Initial investigations had found no issues, however a partner had brought down their BGP tunnels and brought them back again. They were advising that they have customers over the link working ok however once they drop and instigate a new tunnel thatâs when the issues start. This started to happen around 23:00 yesterday evening. We had been advised that a number of partners and circuits had been impacted including Verizon, Andrew & Arnolds, GCI, Maintel, Fluid One, Timico, Daisy, Fujitsu. The Networks support team investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors. In order to restore service, the Network support team reloaded card 10 which had then cleared. Confirmation has now been received from wholesale partners via TTB that service has now been restored. Â