We are seeing some customers who are still down following a blip within TalkTalk. We currently have no root cause but are investigating.
A small number of lines are still down, however most have now resumed service. We are still communicating with TalkTalk so we can restore service for all affected lines.
Looks like we're seeing another blip affecting many more customers this time. We are still speaking to TalkTalk to determine the cause of this.
TalkTalk have raised an incident with teh following information:
"We have received reports from a number of B2B customers (Wholesale ADSL) who are experiencing a loss of their Broadband services. The impact is believed to approximately 600 lines across 4 or 5 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our IP operations team to progress. "
As a result, we'll move TalkTalk traffic away from the Harbour Exchange datacentre to see if it helps. This move will be seamless and will not affect other customers.
Our TalkTalk traffic has now been moved away from HEX89, if There are still a small number of customers offline, if they reboot their router/modem that may force a re-connection and a successful login.
At 12:29 we saw around 80 lines drop, most of these are back online as of 12:37 though. The incident is still open with TalkTalk engineers.
TalkTalk are really not having a good day. We're now seeing packetloss on lines as well as a few more drops. We're going to bring the HEX89 interconnect back up in case that is in any way related, we're also chasing TT on this.
This is still an open incident with TalkTalk, it is affecting other ISPs using TalkTalk as their backhaul. We have chased TalkTalk for an update.
Update from TalkTalk: "Network support have advised that service has been partially restored. Currently Network Support are continuing to re-balance traffic between both LTS’s (HEX & THN). This work is currently being completed manually by our Network support team who ideally need access to RadTools to enable them to balance traffic more efficiently. We are currently however experiencing an outage of RadTools which is being managed under incident 10007687. We will continue to provide updates on the progress as soon as available."
Probably as a result, we are still seeing low levels of packetloss on some TalkTalk lines.
It's looking like the low levels of packetloss stopped at 16:10. Things are looking better.
There are a handful of customers that are still offline, we have sent the list of these circuits to TalkTalk to investigate.
Update from TalkTalk: "We have received reports from a number of B2B customers (Wholesale ADSL & FTTC) who are experiencing authentication issues with their Broadband services. The impact is believed to approximately 100 lines across 2 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our Network support team to progress."
We have actaully already taken down our Harbour Exchange interconnect but this has not helped.
Over half of these remaining affected lines logged back in at 2017-06-08 10:38
The remaining customers offline should try rebooting their router/modem and if still not online then please contact Support.
From TalkTalk: The root cause of this issue is believed to have been caused by a service request which involved 3 network cards being installed in associated equipment at Harbour exchange. This caused BGP issues on card (10/1). To resolve this Network Support shut down card (10/1) but this did not resolve all issues. This was then raised this to Ericsson who recommended carrying out an XCRP switchover on the LTS. Once the switchover was carried out all subscribers connections dropped on the LTS and the majority switched over to the TeleHouse North LTS. Network support then attempted to rebalance the traffic across both LTS platform however were not able to due to an ongoing system incident impacting Radius Tools. Network support instead added 2 new 10G circuits to the LTS platform to relieve the congestion and resolve any impact. As no further issues have been identified this incident will now be closed and any further RCA investigation will be carried out by problem management.
Regarding the problem with a few circuits not able to establish PPP. the report from TalkTalk is as follows: Network support have advised that they have removed HEX (harbour exchange) from the radius to restore service until a permanent fix can be identified. Network support are liaising with Ericsson in regards to this and investigations are ongoing.