We're investigating connectivity problems.
Network is recovering
We are seeing some packet loss on our network again, we're still investigating this.
The Network is stable at the moment but the cause of this is still under investigation.
For information, the problems have occurred for minutes around: 08:55, 10:00, 10:10, 10:30. Customers would have had 'routing' problems during these times - ie problems reaching websites and places on the internet. More information to follow.
The disruption today has been caused by problems with our 'route servers'. These are BGP routers that manage all our internal and external IP address announcements and routing information for our network - they exchange IP address information between all our LNS routers and our edge transit/peering routers. For resilience, we run two route servers, in separate data-centres. They both exchange routes with all the other routers on our network. Each route server is also physically connected to two network switches.
The disruptions today were a result of both of our route dropping their Ethernet connections to their switches at a similar time, which caused a BGP sessions to drop which cascaded to the routing problems that were experienced at the time.We are still not 100% sure of the cause and investigations are continue. However, later software does manage the Ethernet ports slightly different and at 11:30 we upgraded the software on one of the router servers.
We are still monitoring the networkNetwork has remained stable since 10:30AM
We had a similar occurrence of this problem at around 11PM on 30th March - however, due to the software upgrade on one of our route servers, the network was unaffected. We are still investigating the root cause, but our software changes have made our routers handle this problem better.