Initial reports suggest one of our fibre links to TalkTalk is down. This is affecting broadband lines using TalkTalk backhaul.
00:05 TT Lines drop, looked like we had a router blip and a TT fibre blip - reasons yet unknown 00:15 Lines start to log back in However, we are getting reports in intermittent access to some sites on internet - possible MTU related.
MTU is still a problem. A workaround for the moment, is to lower the MTU setting in your router to 1432. Ideally this should not be needed, but try this until the problem is resolved.
Other wholesalers using TT are reporting the same problem. TT helpdesk is aware of planned work that may be causing this. We have requested that that pass this MTU report on to the team involved in the planned work.
TT tell us they think the problem with MTU has been fixed. We're still unsure at this moment, and will work with customers who still have problems.
This is still a problem affecting customers using TT backhaul. TT are aware and are investigating. This is a result of a router upgrade within TT which looks to have been given incorrect settings. Where possible, customers can change the MTU on their routers to be 1432
We have been in contact with the TT Service Director who will be chasing this up internally at TT.
Customers with bonded lines using TT and BT can turn off their TT modem or router for the time being.
We are looking at re-routing TT connections through our secondary connection to TT...
Traffic is now routing via our secondary connection to TT, this looks like it is not being routed via the faulty TT router and it is looks as if lines are passing traffic as normal
Some customers are working OK, some are not. The reason being is that we have 2 interconnects to TT. We are still seeing connections from both of them, however, we have a 1600 byte path from one but only 1500 from the other. The 1500 one is the one that TT did an upgrade on last night. So it looks like TT forgot to configure jumbo frames on an interface after the upgrade. Needless to say, we've passed this information on to people at various levels within TT
We are working on only accepting connections from TT via the working interconnect.
We are forcing TT lines to reconnect, this should mean they then reconnect over the working interconnect and not the one with the faulty TT router.
We are blocking connections from the faulty TT router and only accepting from the working one. This means when customers connect they have a working connection. However, this does mean that logins are being rejected from customers until they are routed via the working interconnect. It may take a few attempts for customers to connect first time.
Some lines are taking a long time to come back. This is because they are still coming in via the broken interconnect - that we're rejecting. Unfortunately, affected lines just have to be left until they attempt to log in via the working interconnect. So, if we appear to be rejecting your login please leave your router to keep trying and it should fix itself.
TT are reverting their upgrade from last night. This looks like it's underway at the moment.
Latest from TT: "The roll back has been completed and the associated equipment has been restarted. Our (TT) engineers are currently performing system checks and a retest before confirming resolution on this incident. Further information will be provided shortly. "
TT have completed their downgrade. It looks like the faulty link is working OK again, we'll be testing this before we unblock the link our side.
We've re-enabled the faulty link, we are now back to normality! We do apologise for this outage. We will be discussing this fault and future upgrades of these TT routers with TT staff.