3 Aug 05:07:34
[Broadband] TalkTalk DSL core network changes will cause drops, early hours of 3 August - Completed
Details
Posted: 24 Jul 13:32:48

The following is from TalkTalk and relates to this incident post: https://aastatus.net/2401 and also the other similar planned work for 1st August: https://aastatus.net/2414 This is happening between 00:00 and 06:00 Wednesday 3rd August.

"The wholesale LTS platform has been suffering from packet loss and congestion over the last few months. Juniper have now advised that this is due to a limitation with the hardware we have and we need to carry out some essential vendor recommended maintenance on our DSL Interconnects platform/switch at Harbour Exchange in order to fix customer slow throughput.

This RFC is to rebuild ldn-vc1.thn from a mixed EX4200/EX4550 estate to a entirely EX4550 virtual switch.

We have a limitation with our EX4550's - running in a mostly EX4200 VC (mixed mode), whereby the VC cables/modules will not support any more bandwidth past 16gbps (each way). This limitation is because we are running a mixed mode VC. We therefore need to upgrade this VC to run all EX4550s which will mean the VC cables/modules will support 32gbps and the hope is that low speed issues will be resolved."

During this work AAISP will move traffic over to our Telehouse interconnect so as to minimise the impact on our customers, however there still may be drops between midnight and 4am.

Update
3 Aug 02:00:12
This work is under way, we do have a number of lines that dropped and have not come back yet.
Update
3 Aug 02:30:23
There are still a few hundred circuits offline, they are reconnecting slowly. We suspect this work by talk talk has put strain on other parts of their network which is causing delays in logging back in.
Update
3 Aug 05:09:09
The work has now finished, any customers still off line should try restarting their router so as to force it to try logging in.
Resolution TalkTalk confirm that their work completed successfully. From our side, we did see some line drop PPP and reconnect, however some lines did take a long time (up to 2 hours) to reconnect. We have raised this with TalkTalk. Further updates to the evening packetloss issue which this work should fix will be posted against this incident: https://aastatus.net/2401
Closed 3 Aug 05:07:34
Previously expected 3 Aug 00:04:00