TalkTalk did some work on the Telehouse side of our interconnect on Friday as follows:

"The device AA connect into is a chassis with multiple cards and interfaces creating a virtual switch. The physical interface AA plugged into was changed to another physical interface. We suspect this interface to be faulty as when swapped to another it looks to have resolved the packet loss."

We will be testing both of our interconnects individually over the next couple of days.

TalkTalk have raised an incident with teh following information:

"We have received reports from a number of B2B customers (Wholesale ADSL) who are experiencing a loss of their Broadband services. The impact is believed to approximately 600 lines across 4 or 5 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our IP operations team to progress. "

As a result, we'll move TalkTalk traffic away from the Harbour Exchange datacentre to see if it helps. This move will be seamless and will not affect other customers.

Update from TalkTalk: "Network support have advised that service has been partially restored. Currently Network Support are continuing to re-balance traffic between both LTS’s (HEX & THN). This work is currently being completed manually by our Network support team who ideally need access to RadTools to enable them to balance traffic more efficiently. We are currently however experiencing an outage of RadTools which is being managed under incident 10007687. We will continue to provide updates on the progress as soon as available."

Probably as a result, we are still seeing low levels of packetloss on some TalkTalk lines.

Update from TalkTalk: "We have received reports from a number of B2B customers (Wholesale ADSL & FTTC) who are experiencing authentication issues with their Broadband services. The impact is believed to approximately 100 lines across 2 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our Network support team to progress."

We have actaully already taken down our Harbour Exchange interconnect but this has not helped.

We have completed further tests and we are seeing congestion manifesting itself as slow throughput at peak times (evenings and weekends) on VDSL (FTTC) lines that connect to us through a certain Talk Talk LAC.

This has been reported to senior TalkTalk staff.

To explain further; VDSL circuits are routed from TalkTalk to us via two LACs. We are seeing slow thoughput at peak times on one LAC and not the other.

Tests over the weekend suggest that speeds are good when we only use our Harbour Exchange interconnect.

TalkTalk are moving the interconnect we have at Telehouse to a different port at their side so as to rule out a possible hardware fault.

We now have samples of lines which are affected by the slow throughput and those that are not.

Since 9pm Sunday we are using the Harbour Exchange interconnect in to TalkTalk and so all customers should be seeing fast speeds.

TalkTalk are doing some work this evening and will be reporting back to us tomorrow. We are also going to be carrying out some tests ourselves this evening too.

Our tests will require us to move traffic over to the Telehouse interconnect, which may mean some customers will see slow (single thread) download speeds at times. This will be between 9pm and 11pm

TalkTalk are arranging some further tests to be done with us which will happen Monday or Tuesday evening this week.

TT have confirmed that they have made a configuration change on the switch at their end in Telehouse - this is the reason for the speed testing this evening.

Here are the results from last night:

And BT Circuits:

Some of the results are rather up and down, but these lines are in use by customers so we would expect some fluctuations, but it's clear that a number of lines are unaffected and a number are affected.

Here's the interesting part. Since this problem started we have rolled out some extra logging on to our LNSs, this has taken some time as we only update one a day. However, we are now logging the IP address used at our side of L2TP tunnels from TalkTalk. We have eight live LNSs and each one has 16 IP addresses that are used. With this logging we've identified that circuits connecting over tunnels on 'odd' IPs are fast, whilst those on tunnels on 'even' IPs are slow. This points to a LAG issue within TalkTalk, which is what we have suspected from the start but this data should hopefully help TalkTalk with their investigations.

We have been testing the Telehouse interconnect this evening with TalkTalk engineers. This involved a ~80 minute conference call and setting up a very simple test of a server our side plugged in to the switch which is connected to our 10G interconnect, and running iperf3 tests against a laptop on the TalkTalk side.

The test has highlighted a problem at the TalkTalk end with the connection between two of their switches. When plugged in to the second switch we got about 300Mbit/s, but when their laptop was in the switch directly connected to our interconnect we got near full speed or around 900Mb/s.

This has hopefully given them a big clue and they will now involve the switch vendor for further investigations.

Results from last night when we changed over to test the Telehouse interlink:

This shows that unlike the previous times, when we changed over to use the Telehouse interconnect at 11PM speeds did not drop.

We're still awaiting details from TalkTalk as to what the fix was and if it is a temporary or permanent fix.

Speed tests against customers over the weekend do not show the peak time slow downs, this confrims that what TalkTalk did on Thursday night has fixed the problem. We are still awaiting the report from TalkTalk regarding this incident.

The graph above shows iperf3 speed test results taken once an hour over the weekend against nearly 30 customers. Although some are a bit spiky we are no longer seeing the drastic reduction in speeds at peak time. The spikyness is due to the lines being used as normal by the customers and so is expected.

This testing is planned for 16th May.

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.