TalkTalk Wholesale lines are currently off line. We are investigating.
Some lines are coming back online..
Other wholesalers are having similar problems
Most lines are back online.
We had notified TalkTalk at the start of this, they have replied saying that they have identified some network alarms. We'll update when we get more information. For now this fault is closed as lines are back online.
TalkTalk confirmed that this affected many other of their wholesalers.
We've not had the report from TT yet, but we have chased TT for it.
This is the report from TalkTalk regarding this incident: Problem Management have carried our further investigations. Please see the root cause below and preventative measures: On Monday 21/10/2013, following some network device reboots, traffic was lost on two aggregate interconnects. We confirmed this using graphs on Cacti. Investigations were carried out by the B2B team and at 3:53pm on Monday 21st October a configuration addition was implemented on the ldn-vc1.thn device to try and load balance traffic over the two interconnects. Unfortunately this change affected the global routing of all traffic on the box (including management and customer data traffic). As part of the standard B2B procedure, the commit confirmed feature was used on the device when the change was made which resulted in the device returning back to normal service (management and data traffic) after 2 minutes. Following a review of this incident by B2B it was determined that the change was high risk. B2B processes have now been changed to always perform risk analysis on any future configuration changes. As well as this, any future configuration changes deemed necessary should be completed out of hours via the TTT PEW change process. Sincere apologies for any inconvenience this has caused.
Summary of events:
Lines went off at 15:53
The started coming back online at 15:56
Most, if not all lines were back online by 16:04
We will have a report from TalkTalk as to the reason for this later today (22nd)