TalkTalk did some work on the Telehouse side of our interconnect on Friday as follows:

"The device AA connect into is a chassis with multiple cards and interfaces creating a virtual switch. The physical interface AA plugged into was changed to another physical interface. We suspect this interface to be faulty as when swapped to another it looks to have resolved the packet loss."

We will be testing both of our interconnects individually over the next couple of days.

Sadly, we are still seeing very low levels of packetloss on some TalkTalk connected circuits in the evenings. We have raised this with TalkTalk today, they have investigated this afternoon and say: "Our Network team have been running packet captures at Telehouse North and replicated the packet loss. We have raised this into our vendor as a priority and are due an update tomorrow."

We'll keep this post updated.

Update from TalkTalk regarding their investigations today:- Our engineering team have been working through this all day with the Vendor. I have nothing substantial for you just yet, I have been told I will receive a summary of today's events this evening but I expect the update to be largely "still under investigation". Either way I will review and fire an update over as soon as I receive it. Our Vendor are committing to a more meaningful update by midday tomorrow as they continue to work this overnight.

Here is an example - the loss is quite small on individual lines, but as we are seeing this sort of loss on many circuits and the same time (evenings) it make this more severe. It's only due to to our constant monitoring that this gets picked up.

Saturday 15th and Sunday 16th evenings were a fair bit worse than previous evenings. On Saturday and Sunday evening we saw higher levels of packet loss (between 1% and 3% on many lines) and we also saw slow single TCP thread speeds much like we saw in April. We did contact TalkTalk over the weekend and this has been blamed on a faulty card that TalkTalk had on Thursday that was replaced but has caused traffic imbalance on this part of the network.

We expect things to improve but we will be closely monitoring this on Monday evening (17th) and will report back on Tuesday.

We'll close this incident as lines have been stable for an hour. We'll update the post with further information as to the cause and any action we will be taking to help stop this type of incident from happening again.

We would like to thank our customers for their patience and support this evening. We had many customers in our IRC channel who were in good spirits and supportive to our staff whilst they worked on this incident.

TalkTalk have raised an incident with teh following information:

"We have received reports from a number of B2B customers (Wholesale ADSL) who are experiencing a loss of their Broadband services. The impact is believed to approximately 600 lines across 4 or 5 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our IP operations team to progress. "

As a result, we'll move TalkTalk traffic away from the Harbour Exchange datacentre to see if it helps. This move will be seamless and will not affect other customers.

Update from TalkTalk: "Network support have advised that service has been partially restored. Currently Network Support are continuing to re-balance traffic between both LTS’s (HEX & THN). This work is currently being completed manually by our Network support team who ideally need access to RadTools to enable them to balance traffic more efficiently. We are currently however experiencing an outage of RadTools which is being managed under incident 10007687. We will continue to provide updates on the progress as soon as available."

Probably as a result, we are still seeing low levels of packetloss on some TalkTalk lines.

Update from TalkTalk: "We have received reports from a number of B2B customers (Wholesale ADSL & FTTC) who are experiencing authentication issues with their Broadband services. The impact is believed to approximately 100 lines across 2 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our Network support team to progress."

We have actaully already taken down our Harbour Exchange interconnect but this has not helped.

We have completed further tests and we are seeing congestion manifesting itself as slow throughput at peak times (evenings and weekends) on VDSL (FTTC) lines that connect to us through a certain Talk Talk LAC.

This has been reported to senior TalkTalk staff.

To explain further; VDSL circuits are routed from TalkTalk to us via two LACs. We are seeing slow thoughput at peak times on one LAC and not the other.

Tests over the weekend suggest that speeds are good when we only use our Harbour Exchange interconnect.

TalkTalk are moving the interconnect we have at Telehouse to a different port at their side so as to rule out a possible hardware fault.

We now have samples of lines which are affected by the slow throughput and those that are not.

Since 9pm Sunday we are using the Harbour Exchange interconnect in to TalkTalk and so all customers should be seeing fast speeds.

TalkTalk are doing some work this evening and will be reporting back to us tomorrow. We are also going to be carrying out some tests ourselves this evening too.

Our tests will require us to move traffic over to the Telehouse interconnect, which may mean some customers will see slow (single thread) download speeds at times. This will be between 9pm and 11pm

TalkTalk are arranging some further tests to be done with us which will happen Monday or Tuesday evening this week.

TT have confirmed that they have made a configuration change on the switch at their end in Telehouse - this is the reason for the speed testing this evening.

Here are the results from last night:

And BT Circuits:

Some of the results are rather up and down, but these lines are in use by customers so we would expect some fluctuations, but it's clear that a number of lines are unaffected and a number are affected.

Here's the interesting part. Since this problem started we have rolled out some extra logging on to our LNSs, this has taken some time as we only update one a day. However, we are now logging the IP address used at our side of L2TP tunnels from TalkTalk. We have eight live LNSs and each one has 16 IP addresses that are used. With this logging we've identified that circuits connecting over tunnels on 'odd' IPs are fast, whilst those on tunnels on 'even' IPs are slow. This points to a LAG issue within TalkTalk, which is what we have suspected from the start but this data should hopefully help TalkTalk with their investigations.

We have been testing the Telehouse interconnect this evening with TalkTalk engineers. This involved a ~80 minute conference call and setting up a very simple test of a server our side plugged in to the switch which is connected to our 10G interconnect, and running iperf3 tests against a laptop on the TalkTalk side.

The test has highlighted a problem at the TalkTalk end with the connection between two of their switches. When plugged in to the second switch we got about 300Mbit/s, but when their laptop was in the switch directly connected to our interconnect we got near full speed or around 900Mb/s.

This has hopefully given them a big clue and they will now involve the switch vendor for further investigations.

Results from last night when we changed over to test the Telehouse interlink:

This shows that unlike the previous times, when we changed over to use the Telehouse interconnect at 11PM speeds did not drop.

We're still awaiting details from TalkTalk as to what the fix was and if it is a temporary or permanent fix.

Speed tests against customers over the weekend do not show the peak time slow downs, this confrims that what TalkTalk did on Thursday night has fixed the problem. We are still awaiting the report from TalkTalk regarding this incident.

The graph above shows iperf3 speed test results taken once an hour over the weekend against nearly 30 customers. Although some are a bit spiky we are no longer seeing the drastic reduction in speeds at peak time. The spikyness is due to the lines being used as normal by the customers and so is expected.

This testing is planned for 16th May.

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

We are mostly ready to switch over to the new system - probably happening on Monday. The wording of the messages will change slightly with the new system. There are already more options on the control pages, and once we have switched over these will replace the old settings.

Once done, we will start using the new system for more and more aspects of the control pages, ordering, and faults, over the coming weeks.

Affected lines are looking like this. This shows the fault started just after 9am, but from 3pm there is severe packet loss.

It looks like TalkTalk completed their work at around 06:30. We've not yet received an update from TalkTalk as to how it went from their side, but we'll update this post when we do.

There may be a very small number of customers who are still off line, these may need a router reboot so as to force it to log back in.

We are now expecting this works to happen on the evening of the 16th and the early hours of the 17th May.

We will move traffic away from Telehouse, where TalkTalk are doing there work so as to minimise the number of customers who will be affected.

We are expecting this work to cause some disruption, in that circuits will be disconnected and will reconnect a couple of times during this work.