AFFECTED

STARTED

CLOSED

DESCRIPTION

Many of our TalkTalk ADSL and FTTC lines dropped at 10:08. Most lines reconnected by 10:11, and the remaining by around 10:18. Cause is yet undetermined, we're awaiting updates from TalkTalk. They did report a 'network flap' that affected a smaller number of lines at 9:52, but we're not sure if these two sets of blips are related.)

Further update: TalkTalk do say that that drops were due to link flaps on fibres between their datacentres.

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

Here is a brief summary of this incident and what we are planning to do to improve matters.

1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
3. 01:52 about 30% of lines still had problems staying connected
4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.

AFFECTED

STARTED

CLOSED

DESCRIPTION

AFFECTED

STARTED

CLOSED

DESCRIPTION

• From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
• TalkTalk Ethernet services were off line for the whole duration
• Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
• broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
• A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
• The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
• Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
• Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
• From 10:30PM, we consider our network stable.