General internet connectivity
MAJOR Closed General
STATUS
Closed
CREATED
Mar 30, 10:08 AM (1½ years ago)
AFFECTED
General
STARTED
Mar 30, 10:00 AM (1½ years ago)
CLOSED
Mar 30, 11:00 AM (1½ years ago)
REFERENCE
42530 / AA42530
INFORMATION
  • INITIAL
    1½ years ago by Andrew

    We're investigating connectivity problems.

  • UPDATE
    1½ years ago by Andrew

    Network is recovering

  • UPDATE
    1½ years ago by Andrew

    We are seeing some packet loss on our network again, we're still investigating this.

  • UPDATE
    1½ years ago by Andrew

    The Network is stable at the moment but the cause of this is still under investigation.

  • UPDATE
    1½ years ago by Andrew

    For information, the problems have occurred for minutes around: 08:55, 10:00, 10:10, 10:30. Customers would have had 'routing' problems during these times - ie problems reaching websites and places on the internet. More information to follow.

  • UPDATE
    1½ years ago by Andrew

    The disruption today has been caused by problems with our 'route servers'. These are BGP routers that manage all our internal and external IP address announcements and routing information for our network - they exchange IP address information between all our LNS routers and our edge transit/peering routers. For resilience, we run two route servers, in separate data-centres. They both exchange routes with all the other routers on our network. Each route server is also physically connected to two network switches.

    The disruptions today were a result of both of our route dropping their Ethernet connections to their switches at a similar time, which caused a BGP sessions to drop which cascaded to the routing problems that were experienced at the time.

    We are still not 100% sure of the cause and investigations are continue. However, later software does manage the Ethernet ports slightly different and at 11:30 we upgraded the software on one of the router servers.

    We are still monitoring the network

  • UPDATE
    1½ years ago by Andrew

    Network has remained stable since 10:30AM

  • UPDATE
    1½ years ago by Andrew

    We had a similar occurrence of this problem at around 11PM on 30th March - however, due to the software upgrade on one of our route servers, the network was unaffected. We are still investigating the root cause, but our software changes have made our routers handle this problem better.

  • Closed