Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed General Internet Access
AFFECTED
General Internet Access
STARTED
May 29, 12:35 PM (20 days ago)
CLOSED
May 29, 03:00 PM (19¾ days ago)
DESCRIPTION
We're investigating problems on our network which will affecting connectivity for customers.
Resolution: Sorry for the delay posting details - this is one of those occasions where there are not a lot of details we can post. The problems mainly related to our authoritative DNS servers which therefore caused issues with almost all services.

Closed: 19¾ days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
May 19, 12:51 PM (29¾ days ago)
CLOSED
May 19, 01:21 PM (29¾ days ago)
DESCRIPTION
We're seeing a high number of broadband lines dropping and reconnecting, we're investigating
Resolution: Line drops were caused when staff bought up an interconnect between our two London datacentres. The configuration of this circuit was intended to be separate from the rest of our network but somehow it did carry some traffic and cause this outage for some customers. We continue to investigate the cause in more detail.

Closed: 29¾ days ago
View full details

MAJOR Closed Broadband Lines
AFFECTED
Broadband Lines
STARTED
Mar 29, 01:10 AM (2½ months ago)
CLOSED
Mar 29, 05:00 AM (2½ months ago)
DESCRIPTION
We're seeing high numbers of TalkTalk lines disconnected, we're investigating
Resolution:

Closed: 2½ months ago
View full details

MAJOR Closed General
AFFECTED
General
STARTED
Feb 25, 12:22 PM (3¾ months ago)
CLOSED
Feb 25, 12:28 PM (3¾ months ago)
DESCRIPTION
At around 12:22 we had a routing problem which affected general internet access.
Resolution:

Closed: 3¾ months ago
View full details

MAJOR Closed General Internet Access
AFFECTED
General Internet Access
STARTED
Feb 18, 06:00 PM (4 months ago)
CLOSED
Feb 18, 06:10 PM (4 months ago)
DESCRIPTION
We saw a few minutes of packet loss and connectivity problems at around 6PM. We're investigating further.
Resolution:

Closed: 4 months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 29, 09:00 AM (7½ months ago)
CLOSED
Oct 29, 10:00 AM (7½ months ago)
DESCRIPTION
We had overnight problems with a faulty switch in London. details here: https://aastatus.net/37744 Bringing the replacement switch back in to service is causing some disruption to services though.
Resolution:

Closed: 7½ months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (9 months ago)
CLOSED
Sep 15, 04:00 PM (9 months ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 9 months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (9¼ months ago)
CLOSED
Sep 06, 08:00 PM (9¼ months ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

Closed: 9¼ months ago
View full details

MAJOR Closed Power Outage
AFFECTED
Power Outage
STARTED
Aug 18, 04:30 AM (10 months ago)
CLOSED
Aug 18, 09:40 PM (10 months ago)
DESCRIPTION
The London LD8 datacentre suffered a loss of power at 4:30AM. Our status page was affected and this is a new post which will have some updates as services are being restored.
Resolution: Brief summary:
  • From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
  • TalkTalk Ethernet services were off line for the whole duration
  • Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
  • broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
  • A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
  • The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
  • Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
  • Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
  • From 10:30PM, we consider our network stable.
Although many customers were unaffected by this incident others were affected in various way. We do apologise for this. We'll be posting more information about the actual cause of the power loss when we receive it.

Closed: 10 months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 12, 04:30 AM (10 months ago)
CLOSED
Aug 12, 05:15 PM (10 months ago)
DESCRIPTION
We have a number of DSL lines down in and around Scotland. BT say: "Due to the severe weather in Scotland early this morning there was a major outage at Edinburgh Exchange causing loss of 20CN & 21CN / ETHERNET, MEAS & BROADBAND SERVICES. Engineers are currently on site recovering services. We're sorry for any inconvenience that this has caused."
Resolution:

Closed: 10 months ago
View full details