Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed General
AFFECTED
General
STARTED
Nov 21, 07:25 PM (5¼ days ago)
CLOSED
Nov 21, 07:45 PM (5¼ days ago)
DESCRIPTION
We're investigating a network incident that occured at around 19:30, updates to follow.
Resolution: Service restored and back to normal at around 19:50. This was due to a network issue that affected some of our BGP sessions which caused routing problems to certain destinations.

Closed: 5¼ days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 20, 10:20 AM (1 month ago)
CLOSED
Oct 20, 10:45 AM (1 month ago)
DESCRIPTION
Lines connected on the witless LNS dropped and are in process of reconnecting.
Resolution: The Witless LNSs suffered a crash which was caused by a bug in our CQM graphing code. This bug has been fixed and is in the process of being tested and we will then roll out the new software to the the Witless LNSs overnight as 'emergency' planned work.

Closed: 1 month ago
View full details

MAJOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 30, 03:24 PM (1¾ months ago)
CLOSED
Sep 30, 04:36 PM (1¾ months ago)
DESCRIPTION
at 15:24 we saw most of our TalkTalk circuits drop. They are in the process of reconnecting.
Resolution: Lines reconnected within a couple of minutes and we're investigating this further with TalkTalk. Further updates about this will be on https://aastatus.net/42468

Closed: 1¾ months ago
View full details

MAJOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 27, 10:00 PM (2 months ago)
CLOSED
Sep 28, 08:00 AM (2 months ago)
DESCRIPTION
Following on from the problems this morning, TalkTalk are performing some emergency work overnight, tonight. The work will start at 10PM and will carry on through to 6AM. They say that their work before midnight should not impact circuits, but the work after midnight may cause some reconnects.
Resolution: TalkTalk have emergency overnight work that failed and turned in to an outage causing circuits to drop. We took our Telehouse connection out of service and lines reconnected via our Equinix link and have been stable since 08:30. We'll leave the Telehouse link down for the moment in case TalkTalk schedule further work on it.

Closed: 2 months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Jul 20, 01:00 AM (4¼ months ago)
CLOSED
Jul 20, 07:29 AM (4 months ago)
DESCRIPTION
We're investigating a problem causing some lines to be reconnecting.
Resolution: A bug has been found in the LNS code which erroneously limited the number of connections that would successfully connect. This has been fixed. We will update the software on our test LNS (B.Doubtless) overnight. We'll then reschedule an upgrade of the production LNS (X.Witless) and bulk-move of customers for later in the week.

Closed: 4 months ago
View full details

MAJOR Closed Internet
AFFECTED
Internet
STARTED
Jul 06, 02:45 PM (4½ months ago)
CLOSED
Jul 06, 03:08 PM (4½ months ago)
DESCRIPTION
at around 14:45 there was some sort of problem that affected traffic, we're investigating the cause.
Resolution: We caused a network loop between two of our switches in one of our London datacentres. This was caused by loading wrong software variant on to one of our test FireBrick FB9000 routers; the software variant that was loaded didn't have the LACP feature and when the FireBrick booted it caused a network loop. This caused a broadcast storm and general chaos until the switch disabled the ports. We do apologise for the inconvenience this caused customers this afternoon, lesson learnt and software builds have been changed to include LACP in all variants.

Closed: 4½ months ago
View full details

MAJOR Closed access
AFFECTED
access
STARTED
Nov 25, 02:43 AM (1 year ago)
CLOSED
Nov 25, 02:48 AM (1 year ago)
DESCRIPTION
We're investigating a routing problem at the moment.
Resolution:

Closed: 1 year ago
View full details

MAJOR Closed TalkTalk Broadband
AFFECTED
TalkTalk Broadband
STARTED
Jul 09, 10:08 AM (1¼ years ago)
CLOSED
Jul 09, 10:18 AM (1¼ years ago)
DESCRIPTION
We've just seen many TalkTalk connected lines disconnect.
Resolution:

Many of our TalkTalk ADSL and FTTC lines dropped at 10:08. Most lines reconnected by 10:11, and the remaining by around 10:18. Cause is yet undetermined, we're awaiting updates from TalkTalk. They did report a 'network flap' that affected a smaller number of lines at 9:52, but we're not sure if these two sets of blips are related.)

Further update: TalkTalk do say that that drops were due to link flaps on fibres between their datacentres.


Closed: 1¼ years ago
View full details

MAJOR Closed General Internet Access
AFFECTED
General Internet Access
STARTED
May 29, 12:35 PM (1¼ years ago)
CLOSED
May 29, 03:00 PM (1¼ years ago)
DESCRIPTION
We're investigating problems on our network which will affecting connectivity for customers.
Resolution: Sorry for the delay posting details - this is one of those occasions where there are not a lot of details we can post. The problems mainly related to our authoritative DNS servers which therefore caused issues with almost all services.

Closed: 1¼ years ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
May 19, 12:51 PM (1½ years ago)
CLOSED
May 19, 01:21 PM (1½ years ago)
DESCRIPTION
We're seeing a high number of broadband lines dropping and reconnecting, we're investigating
Resolution: Line drops were caused when staff bought up an interconnect between our two London datacentres. The configuration of this circuit was intended to be separate from the rest of our network but somehow it did carry some traffic and cause this outage for some customers. We continue to investigate the cause in more detail.

Closed: 1½ years ago
View full details