Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 29, 09:00 AM (5½ months ago)
CLOSED
Oct 29, 10:00 AM (5½ months ago)
DESCRIPTION
We had overnight problems with a faulty switch in London. details here: https://aastatus.net/37744 Bringing the replacement switch back in to service is causing some disruption to services though.
Resolution:

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (7 months ago)
CLOSED
Sep 15, 04:00 PM (7 months ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (7¼ months ago)
CLOSED
Sep 06, 08:00 PM (7¼ months ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 12, 04:30 AM (8 months ago)
CLOSED
Aug 12, 05:15 PM (8 months ago)
DESCRIPTION
We have a number of DSL lines down in and around Scotland. BT say: "Due to the severe weather in Scotland early this morning there was a major outage at Edinburgh Exchange causing loss of 20CN & 21CN / ETHERNET, MEAS & BROADBAND SERVICES. Engineers are currently on site recovering services. We're sorry for any inconvenience that this has caused."
Resolution:

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jan 22, 11:10 PM (1 year ago)
CLOSED
Jan 23, 09:05 AM (1 year ago)
DESCRIPTION
We're investigating a problem with some broadband lines showing 50% packetloss from just after 11PM.
Resolution: The problem was resolved just after 9AM. the reason for this outage is in https://aastatus.net/33308 "Networks have investigated and advised they found a number of errors on a card in LOH which was erroring and causing back plain errors."

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Nov 19, 12:01 AM (1¼ years ago)
CLOSED
Nov 19, 01:00 AM (1¼ years ago)
DESCRIPTION
We are seeing line drops on some TalkTalk lines since midnight. We're investigating...
Resolution: This was a fault within TalkTalk. The 'routing engine' in one of their routers failed.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 09, 02:11 AM (1½ years ago)
CLOSED
Oct 09, 09:06 AM (1½ years ago)
DESCRIPTION

On 09/10/19 at 02:11 our monitoring picked up that a few thousand broadband lines dropped.

After some investigation it appears the drops occurred due to all the B-sides of our BTW hostlinks flapping causing all sessions over these links to re-establish

Lines did immediately start to reconnect over our other hostlinks; however due to the number of attempted re-connections in some cases it took ~30minutes for lines reconnect

We are not aware of any BTW planned works scheduled for this time and are currently chasing BT for an explanation for this blip


Resolution:

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 09, 04:53 PM (1½ years ago)
CLOSED
Aug 09, 05:46 PM (1½ years ago)
DESCRIPTION
We have identified a blip in our network at 16:53 with multiple customers going offline and not recovering within a reasonable period of time. We are currently investigating and will update this post with any news.
Resolution: We believe that all affected circuits are currently back online. Our carriers suggest that this outage was due to power issues across the UK. If you are still experiencing issues, then we would suggest rebooting your router, or contacting Support at 03333 400 999 / support@aa.net.uk.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jun 27, 01:35 AM (1¾ years ago)
CLOSED
Jun 27, 02:16 AM (1¾ years ago)
DESCRIPTION
Looks like all lines in TT backhaul dropped. Lines are reconnecting. We’ll update later when we have more details.
Resolution: Line reconnections seem to have stopped and traffic levels back to normal.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Mar 21, 11:20 AM (2 years ago)
CLOSED
Mar 21, 11:56 AM (2 years ago)
DESCRIPTION
At 11:20 we saw a large number of BT lines drop
Resolution: