01 Aug 2017 06:30:00
[Broadband] TalkTalk DSL core network changes will cause drops from midnight to 4AM 1st August - Completed
Details
Posted: 22 Jul 2017 20:21:08

The following is from TalkTalk and relates to this incident post: https://aastatus.net/2401 and also similar work on 3rd August https://aastatus.net/2415

"The wholesale LTS platform has been suffering from packet loss and congestion over the last few months. Juniper have now advised that this is due to a limitation with the hardware we have and we need to carry out some essential vendor recommended maintenance on our DSL Interconnects platform/switch at Telehouse North in order to fix customer slow throughput.

This RFC is to rebuild ldn-vc1.thn from a mixed EX4200/EX4550 estate to a entirely EX4550 virtual switch.

We have a limitation with our EX4550's - running in a mostly EX4200 VC (mixed mode), whereby the VC cables/modules will not support any more bandwidth past 16gbps (each way). This limitation is because we are running a mixed mode VC. We therefore need to upgrade this VC to run all EX4550s which will mean the VC cables/modules will support 32gbps and the hope is that low speed issues will be resolved."

During this work AAISP will move traffic over to our Harbour Exchange interconnect so as to minimise the impact on our customers, however there still may be drops between midnight and 4am.

Update
31 Jul 2017 16:25:24
TalkTalk traffic has been moved over to Harbour Exchange, away from where this overnight work is happening.
Update
01 Aug 2017 03:25:09
A number of TT lines dropped their PPP session at 00:38, byt 01:30 most had reconnected. The planned work window is still open and there may be a further drop.
Update
01 Aug 2017 07:17:14

It looks like TalkTalk completed their work at around 06:30. We've not yet received an update from TalkTalk as to how it went from their side, but we'll update this post when we do.

There may be a very small number of customers who are still off line, these may need a router reboot so as to force it to log back in.

Update
01 Aug 2017 10:42:11
TalkTalk called us shortly after 9am to confirm that they had completed the work in Telehouse successfully. We will move traffic over to Telehouse later today and will be reporting back the outcome in status post: https://aastatus.net/2401
Update
01 Aug 2017 10:45:19
Closed 01 Aug 2017 06:30:00
Previously expected 01 Aug 2017 04:00:00