Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (7¼ days ago)
CLOSED
Sep 15, 04:00 PM (6½ days ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 6½ days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (15½ days ago)
CLOSED
Sep 06, 08:00 PM (15½ days ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

Closed: 15½ days ago
View full details

MAJOR Closed Power Outage
AFFECTED
Power Outage
STARTED
Aug 18, 04:30 AM (1 month ago)
CLOSED
Aug 18, 09:40 PM (1 month ago)
DESCRIPTION
The London LD8 datacentre suffered a loss of power at 4:30AM. Our status page was affected and this is a new post which will have some updates as services are being restored.
Resolution: Brief summary:
  • From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
  • TalkTalk Ethernet services were off line for the whole duration
  • Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
  • broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
  • A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
  • The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
  • Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
  • Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
  • From 10:30PM, we consider our network stable.
Although many customers were unaffected by this incident others were affected in various way. We do apologise for this. We'll be posting more information about the actual cause of the power loss when we receive it.

Closed: 1 month ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 12, 04:30 AM (1¼ months ago)
CLOSED
Aug 12, 05:15 PM (1¼ months ago)
DESCRIPTION
We have a number of DSL lines down in and around Scotland. BT say: "Due to the severe weather in Scotland early this morning there was a major outage at Edinburgh Exchange causing loss of 20CN & 21CN / ETHERNET, MEAS & BROADBAND SERVICES. Engineers are currently on site recovering services. We're sorry for any inconvenience that this has caused."
Resolution:

Closed: 1¼ months ago
View full details

MAJOR Closed SIM2SIM
AFFECTED
SIM2SIM
STARTED
Jul 13, 09:01 AM (2¼ months ago)
CLOSED
Jul 14, 09:16 PM (2¼ months ago)
DESCRIPTION
We are aware of problems affecting our SIP2SIM service - this will be affecting calls and SMS for those using our SIP2SIM service. This may be related to problems with our carrier as there is a known problem with calls relating to Jersey Telecom but we are still investigating.
Resolution:

Closed: 2¼ months ago
View full details

MAJOR Closed VoIP
AFFECTED
VoIP
STARTED
Jun 22, 01:00 AM (3 months ago)
CLOSED
Jun 22, 02:00 AM (3 months ago)
DESCRIPTION
We're investigating a problem with VoIP registrations and calls failing.
Resolution:

Closed: 3 months ago
View full details

MAJOR Closed General
AFFECTED
General
STARTED
May 04, 03:10 PM (4½ months ago)
CLOSED
May 04, 04:00 PM (4½ months ago)
DESCRIPTION
We're investigating a connectivity problem.
Resolution: Service was restored at around 4PM. The disruption was caused by a number of our our BGP routers crashing due to unusual network traffic. This is similar to what we saw a couple of weeks ago when one of our LNS routers crashed. We have since upgraded the software on all our LNS routers and that has helped - our LNSs did not crash during this incident. We are planning to upgrade the software on our BGP routers this week. It is fair to say that with staff working from home that the time taken to recover from this incident was impaired and we'll be looking at ways to improve this too. We would like to apologise to our customers for this outage.

Closed: 4½ months ago
View full details

MAJOR Closed General
AFFECTED
General
STARTED
Feb 27, 08:00 AM (6¾ months ago)
CLOSED
Feb 27, 09:15 AM (6¾ months ago)
DESCRIPTION
We're investigating networking problems with our Maidenhead datacentre. This will be affecting hosted services and access to some of our services such as VoIP and outbound email.
Resolution: Sorry for the delay providing more information. Due to the nature of the incident we prefer not to go in to too much detail publicly. Staff are willing to talk to customers who may be concerned about this, so please do get in touch. The incident lasted just over an hour and did cause various amounts of disruption. There are lessons we can learn from this incident and will be working on further improvements.

Closed: 6¾ months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jan 22, 11:10 PM (8 months ago)
CLOSED
Jan 23, 09:05 AM (8 months ago)
DESCRIPTION
We're investigating a problem with some broadband lines showing 50% packetloss from just after 11PM.
Resolution: The problem was resolved just after 9AM. the reason for this outage is in https://aastatus.net/33308 "Networks have investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors."

Closed: 8 months ago
View full details

MAJOR Closed General
AFFECTED
General
STARTED
Dec 18, 09:40 AM (9 months ago)
CLOSED
Dec 18, 12:04 PM (9 months ago)
DESCRIPTION
We're investigating a faulty switch in Telehouse. This will have some impact on services, we're reviewing this at the moment.
Resolution: The switch is back in service. We're still investigating the initial cause of the problem.

Closed: 9 months ago
View full details