RESOLUTION

3½ years ago by Andrew

Brief summary:

• From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
• TalkTalk Ethernet services were off line for the whole duration
• Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
• broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
• A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
• The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
• Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
• Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
• From 10:30PM, we consider our network stable.

Although many customers were unaffected by this incident others were affected in various way. We do apologise for this. We'll be posting more information about the actual cause of the power loss when we receive it.