We have 5 routers for transit and we are seeing low levels of packet loss on one of these routers.
The cause of this is excess traffic due to our direct interconnect with Amazon being down and Amazon traffic being sent over our transit connections instead of the interconnect.
We have contacted Amazon and hope to have the interconnect working again soon, but we're not yet sure why they have taken it down.
This problem was caused by our interconnect to Amazon going down due to maintenance at the Amazon side. Amazon have a policy not to route traffic over exchange points such as LINX and LONAP (where we have a lot of excess capacity) but instead would send traffic over our transit links (which have less excess capacity).
To prevent this from happening again: We are already in the process of increasing our transit capacity anyway, and we will investigate having a second interconnect in to Amazon.
Further update: a second interconnect to Amazon is underway.
Lines reconnected by 12:10. We suspect a fault somewhere in TalkTalk's network, we'll post an update when we hear back from them.
TalkTalk say: We have some transmission issues around Hove, its being investigated.
Further info from TalkTalk: Multiple Core links flapped circuits have recovered and being monitored.
We've had some problems today with Apple related traffic causing some of our routers to get busy. This has affected some customers with some latency and packetloss to some destinations on the Internet. The problem is that the recent software updates from Apple are being downloaded via our 'transit' links instead of our 'peering' links.
The bulk of our traffic comes to us over our peering links where we have higher capacity connections in to exchange points such as LINX and LONAP. However, even though we have direct peering with Apple of these higher capacity links, today's Apple traffic is being set over our smaller transit links.
We have tried to balance the traffic across the transit links we have, but this hasn't been perfect. We've emailed Apple, requesting that they send traffic to us over the peering links but we've only had an automatic ticket confirmation reply so far.
Traffic levels Friday evening were the highest we've ever seen, however the traffic was spread over multiple routers and were at levels that didn't impact our service negatively. We've had no word back from Apple about them using our peering links. We'll be looking at upgrading our transit capacity as a result of this so as to be able to cope better in the future.
We also understand that there was various reports in the media about Apple having problems yesterday and there were widely reported problems with update downloads failing. We're not sure is this is related to the reasons why Apple traffic was being set via transit instead of our direct peering with Apple though.
2020-11-18 Further update: We're in positive discussions with Apple in regard to improving the connectivity between our networks.