17 May 12:00:00
[Broadband] Peak time packetloss on some TalkTalk connected lines - Closed
Details
Posted: 26 Apr 11:01:07
We have noticed packetloss between 8pm and 10pm on Tuesday (25th April) evening on a small number of TalkTalk connected lines. This may be related to TalkTalk maintenance. We will review this again tomorrow.
Update
26 Apr 16:41:43
We are seeing packet loss this afternoon on some of these lines too. We are contacting TalkTalk.
Update
26 Apr 16:44:23
Update
26 Apr 17:58:06
We have moved TalkTalk traffic over to our Harbour Exchange interconnect to see if this makes a difference or not to the packet loss that we are seeing...
Update
26 Apr 20:50:41
Moving the traffic made no difference. We've had calls with TalkTalk and they have opened an incident and are investigating further.
Update
26 Apr 20:55:14
The pattern that we are seeing relates to which LAC TT are using to send traffic over to us. TT use two LACs at their end, and lines via one have loss whilst lines via the other have no loss.
Update
26 Apr 21:32:30
Another example, showing the loss this evening:

Update
26 Apr 22:26:50
TalkTalk have updated us with: "An issue has been reported that some Partners are currently experiencing quality of service issues, such as slow speed and package (SIC) loss, with their Broadband service. From initial investigations the NOC engineers have identified congestion to the core network connecting to Telehouse North as being the possible cause. This is impacting Partners across the network and not specific to one region, and the impacted volume cannot be determine at present. Preliminary investigations are underway with our NOC and Network Support engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until the engineers have completed further diagnostics."
Update
28 Apr 09:43:43
Despite Talktalk thinking they had fixed this we are still seeing packetloss on these circuits between 8pm and 10pm. It's not as much packetloss as we saw on Wednesday evening, but loss nonetheless. This has been reported back to TalkTalk.
Update
4 May 15:42:04
We are now blocking the two affected Talkalk LACs on new connections, eg a PPP re-connect. This means that it will take a bit longer for a line to re-connect (depending upon the broadband router perhaps a minute or two).

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

Update
5 May 16:48:13
We've decided to not block the LACs that are showing packetloss as it was causing connection problems for a few customers. We have had a telephone call with TalkTalk today, and this issue is being escalated with TalkTalk.
Update
5 May 19:47:37
We've had this update from TalkTalk today:

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

Update
8 May 09:26:21
Planned work has been scheduled for 16th May for this, details and updates of this work is on https://aastatus.net/2386
Update
5 Jun 11:49:24
The planned work took place on the 16th May which appears to have been a success.
Broadband Users Affected 20%
Started 26 Apr 10:00:00
Closed 17 May 12:00:00