Summary
NOC have identified Multiple backhaul links down with NPE001.NOR and NPE001.LEI. This is affecting multiple sites and exchanges including: Corby , Great Oakley , Moulton , Broughton , Kettering , Oundle and Thrapston. Affected exchange codes: EMCRRBY, EMGRETO , EMMOULT , EMOUNDL , EMTHRAP , EMBROUG , EMKTTER.
exchangecodeEMCRRBY,EMGRETO,EMMOULT,EMOUNDL,EMTHRAP,EMBROUG,EMKTTER
Latest Update Preliminary investigations are underway with our NOC engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until our NOC engineers have completed further diagnostics. Â
Latest Update IVR deflection has been added to all impacted exchanges to protect the contact centres. NOC have raised ticket 1-1208069558383 into Openreach for investigations into the cause and fix for the impacted backhauls. There is currently no estimated restoration time for this incident. A further update will be provided by 21:30. Â
Latest Update Openreach investigations have identified three cut cables as the cause of the loss of service. The cables route through a travelling communities site and Openreach engineers are currently engaging BT security and the police to gain access to the site to investigate further. Openreach have declared MBORC due to where the cable damage is onsite and there is currently no estimate restoration time because of this. A further update will be provided by 23:00. Â
Latest Update Openreach Fibre team have now gained access to the location of the damaged cables and they are in the process of completing their assessment of whether the cables can be repaired or not. If the cables cannot be repaired then temporary cables will be laid over ground and re-spliced. NOC have confirmed reroutes to restore service are not possible. A further update will be provided by 03:00. Â
Latest Update The Openreach Fibre team have now located all cables and they have been pulled back away from the travelling communities site however, due to health and safety reasons traffic management is now required as the site is on a 60mph road and this will need to be agreed with the council. As there are currently no re-route options available due to the location of the damage Openreach are arranging emergency traffic management but have advised this will not be available until morning. The repair will continue once traffic management has been arranged and is at site. A further update will be provided by 09:30. Â
Latest Update Openreach Fibre team are at site awaiting approval from the local authority to go ahead with the traffic management. The repair work will continue once traffic management is in place. A further update will be provided by 11:00. Â
Latest Update Openreach have advised that the approval from the local authority to put the Traffic Management in-place is believed to have been approved. Fibre teams onsite are continuing to confirm exactly where the work needs to commence once approval has been confirmed. The repair work will continue once traffic management is in place and further updates on the progress will continue to be provided. Â
Latest Update NOC are continuing to liaise with Openreach who have advised that the Traffic Management is still not currently in-place. OR MIT are awaiting confirmation and estimate traffic-management to be in-place by approximately by 13:00hrs. Fibre teams onsite have completed pre measurements of the fibre break and identified exactly where the work needs to commence once traffic-management is in place and safe for the work to start. Â
Latest Update NOC are continuing to liaise with Openreach who have advised that the Traffic Management is still not currently in-place. Openreach have advised that this is related to them being reliant on contractors for TM who have been slow to respond and missed SLA. Openreach advised that they believe the TM should be in place by approximately 14:30hrs and our NOC team are currently speaking to the OR Duty manager to confirm that latest status. Â
Latest Update NOC are continuing to liaise with Openreach who have advised that Traffic Management arrived at site at approximately 15:10 hrs and began setting up. Openreach Fibre teams are continuing to work onsite to restore service and further updates on the progress of the restoration work will be provided as soon as available. Â
Latest Update IP Access Operations team have successfully re-routed traffic at Moulton, Eye and Orton exchanges to restore service to all impacted circuits. FTTC traffic has also been successfully re-routed at Thrapston exchange to restore service. There are no options to re-route traffic at the remaining impacted exchanges. Openreach have confirmed that the traffic management is now in-place on the A43 and the repair of the cables has commenced as of 17:30. Openreach estimate the repair work will take approx. 3 to 4 hours to complete and have provided an estimate restoration time of 21:30. Â
Latest Update IP Access Operations team have successfully re-routed traffic at Moulton, Eye and Orton exchanges to restore service to all impacted circuits. FTTC traffic has also been successfully re-routed at Thrapston exchange to restore service. There are no options to re-route traffic at the remaining impacted exchanges. Openreach have confirmed that the traffic management is now in-place on the A43 and the repair of the cables has commenced as of 17:30. Openreach estimate the repair work will take approx. 3 to 4 hours to complete and have provided an estimate restoration time of 21:30. Â
Latest Update Openreach engineers remain on site and are continue jointing and splicing of the 3 cut fibre cables. Due to the severity of the damage Openreach have revised their there targeted service restoration time to the before the start of business hours tomorrow. A further update will be provided by 01:00. Â
Latest Update NOC have confirmed following a combination of re-routes completed by IP Access and repair work completed by Openreach service to all impacted customers has been restored as of approx. 20:41. Openreach engineers remain on site and are continuing jointing and splicing of the 3 cut fibre cables. Due to the severity of the damage Openreach have revised their there targeted service restoration time to before the start of business hours tomorrow. Once Openreach have fully completed the fibre repair, IP Access will be engaged to remove the re-routes completed earlier today. A further update will be provided by 08:00. Â
Latest Update NOC have confirmed following a combination of re-routes completed by IP Access and repair work completed by Openreach service to all impacted customers was restored as of approx. 20:41 (21/04). Openreach engineers have successfully completed the jointing and resplicing activities and confirmed service restoration to the 3 impacted fibre cables. IP Access Operations team to now remove all re-routes completed yesterday. A further update will be provided by 10:30. Â
Technical / Suspected Root Cause This incident as raised due to being identified that Multiple Backhaul Links were down. Investigations identified that this was an Openreach fault and was raised to them to investigate further. Openreach investigations identified three cut cables as the cause of the loss of service. Resolution was delayed slightly due to Traffic Management which was required to be put in place to allow OR fibre repair work to be completed safely. NOC confirmed following a combination of re-routes completed by IP Access and repair work completed by Openreach, service to all impacted customers was restored as of approx. 20:41 (21/04). Openreach engineers have successfully completed the jointing and resplicing activities and confirmed service restoration to the 3 impacted fibre cables. As no further issues have been identified we can now confirm closure of this incident. Â