Power outage at datacentre affecting services
MAJOR Closed Power Outage
STATUS
Closed
AFFECTED
Power Outage
STARTED
Aug 18, 04:30 AM (1 month ago)
CLOSED
Aug 18, 09:40 PM (1 month ago)
REFERENCE
36892 / AA36892
INFORMATION
  • INITIAL
    1 month ago by Andrew

    The London LD8 datacentre suffered a loss of power at 4:30AM. Our status page was affected and this is a new post which will have some updates as services are being restored.

  • UPDATE
    1 month ago by Andrew

    This post is lacking lots of detail of the problems that we've had today as our status page, although hosted in Amsterdam on someone else's network, routing to it was also affected.

  • UPDATE
    1 month ago by Andrew

    Power has just been restored, and this will be causing some connectivity problems for a short while whilst the network settles down and we update our configuration of various routers.

  • UPDATE
    1 month ago by Andrew

    We are in the process of rebalancing lines - this involved moving lines back to the LNSs that have been down all day. This means some lines will have a short (minute or two) drop whilst they reconnect.

  • UPDATE
    1 month ago by Andrew

    Service is mostly back to normal now.

  • RESOLUTION
    1 month ago by Andrew

    Brief summary:

    • From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
    • TalkTalk Ethernet services were off line for the whole duration
    • Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
    • broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
    • A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
    • The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
    • Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
    • Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
    • From 10:30PM, we consider our network stable.
    Although many customers were unaffected by this incident others were affected in various way. We do apologise for this. We'll be posting more information about the actual cause of the power loss when we receive it.

  • Closed