INITIAL

We have noticed packetloss between 8pm and 10pm on Tuesday (25th April) evening on a small number of TalkTalk connected lines. This may be related to TalkTalk maintenance. We will review this again tomorrow.

UPDATE

We are seeing packet loss this afternoon on some of these lines too. We are contacting TalkTalk.

UPDATE

UPDATE

We have moved TalkTalk traffic over to our Harbour Exchange interconnect to see if this makes a difference or not to the packet loss that we are seeing...

UPDATE

Moving the traffic made no difference. We've had calls with TalkTalk and they have opened an incident and are investigating further.

UPDATE

The pattern that we are seeing relates to which LAC TT are using to send traffic over to us. TT use two LACs at their end, and lines via one have loss whilst lines via the other have no loss.

UPDATE

Another example, showing the loss this evening:

UPDATE

TalkTalk have updated us with: "An issue has been reported that some Partners are currently experiencing quality of service issues, such as slow speed and package (SIC) loss, with their Broadband service. From initial investigations the NOC engineers have identified congestion to the core network connecting to Telehouse North as being the possible cause. This is impacting Partners across the network and not specific to one region, and the impacted volume cannot be determine at present. Preliminary investigations are underway with our NOC and Network Support engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until the engineers have completed further diagnostics."

UPDATE

Despite Talktalk thinking they had fixed this we are still seeing packetloss on these circuits between 8pm and 10pm. It's not as much packetloss as we saw on Wednesday evening, but loss nonetheless. This has been reported back to TalkTalk.

UPDATE

We are now blocking the two affected Talkalk LACs on new connections, eg a PPP re-connect. This means that it will take a bit longer for a line to re-connect (depending upon the broadband router perhaps a minute or two).

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

UPDATE

We've decided to not block the LACs that are showing packetloss as it was causing connection problems for a few customers. We have had a telephone call with TalkTalk today, and this issue is being escalated with TalkTalk.

UPDATE

We've had this update from TalkTalk today:

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

UPDATE

Planned work has been scheduled for 16th May for this, details and updates of this work is on https://aastatus.net/2386

UPDATE

The planned work took place on the 16th May which appears to have been a success.