Last 10 major outages posts, sorted by last updated
Order posts by limited to posts

MAJOR Closed BT Circuits
AFFECTED
BT Circuits
STARTED
Mar 26, 01:13 AM (3¼ days ago)
CLOSED
Mar 26, 03:30 AM (3 days ago)
DESCRIPTION
We're investigating the cause of major stability issues. Update to follow ASAP.
Resolution:

The cause of this disruption was BT planned work to carry out 'invasive testing' on our links. They have confirmed that the work has been completed.

They failed to inform us of this. We already have a formal complaint regarding previous lack of notifications, and BT have since been sending us notification of works (eg the one for 27th March) manually to us. This is being followed up with our account manager.

We do apologise to our customers who were affected by this.

We're furious.

We have had further information from BT about their work. The work was on a transmission link between two datacentres, and as part of that all ports on devices that use the link also have their ports disabled and enabled. As a result we saw one port on each pair of our host links go down and up around 15 times each - at the same time. As this was not cleanly shutdown by BT it caused traffic to break and customers to drop and reconnect multiple times between midnight and 3:30AM.


MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Feb 27, 12:00 PM (30¾ days ago)
CLOSED
Feb 27, 12:31 PM (30¾ days ago)
DESCRIPTION
A number of lines dropped and reconnected at around noon. We're investigating.
Resolution:

The X.Witless LNS hung and restarted which caused customers to disconnect and reconnect.

This incident is related to https://aastatus.net/42608 X.Witless had been running without incident for 104 days. However, it is not fitted with an NVMe drive and was running software that pre-dates our NVMe drive fixes. We suspect the hang was caused by these two factors.

Further work on our LNSs is being planned and updates will be posted to the status page in due course.


MAJOR Closed BT
AFFECTED
BT
STARTED
Jan 08, 11:40 PM (2½ months ago)
CLOSED
Jan 09, 12:29 AM (2½ months ago)
DESCRIPTION
The Drop and reconnect of a large number of BT circuits was due to planned work: https://aastatus.net/42596
Resolution: Affected lines are back online. This post will be closed now - but refer back to the parent post for further updates of BTs work on 9th January in the early hours of the morning: https://aastatus.net/42596. The cause of this drop has been identified as a mis-configuration within BTs network which prevented our traffic from failing over to our links to BT in Telehouse. This has been fixed by BT and we will retest.

MAJOR Closed Routing - General
AFFECTED
Routing - General
STARTED
Apr 26, 09:30 AM (11 months ago)
CLOSED
Apr 26, 10:00 AM (11 months ago)
DESCRIPTION
One of our routers has restarted and is causing some routing problems. Investigations underway...
Resolution:

MAJOR Closed General
AFFECTED
General
STARTED
Mar 30, 10:00 AM (12 months ago)
CLOSED
Mar 30, 11:00 AM (12 months ago)
DESCRIPTION
We're investigating connectivity problems.
Resolution:

MAJOR Closed L2TP and Routing
AFFECTED
L2TP and Routing
STARTED
Jan 26, 08:57 AM (1 year ago)
CLOSED
Jan 26, 09:40 AM (1 year ago)
DESCRIPTION
We say some L2TP sessions drop at around 08:57 - we are investigating this and a possible routing problem.
Resolution:

MAJOR Closed Line Drops
AFFECTED
Line Drops
STARTED
Dec 15, 01:00 AM (1¼ years ago)
CLOSED
Dec 15, 01:58 AM (1¼ years ago)
DESCRIPTION
We're investigating dropping connections and routing problems that started at 1AM (15th December)
Resolution: Many of our routers lost their BGP connection to our internal 'route servers' - this caused some of our LNSs and edge routers to loose connectivity and their connection. This resulted in customers being unable to reach many places on the internet for a short while. Logs and other data have been captured and will be further investigated to find the cause.

MAJOR Closed General
AFFECTED
General
STARTED
Nov 21, 07:25 PM (1¼ years ago)
CLOSED
Nov 21, 07:45 PM (1¼ years ago)
DESCRIPTION
We're investigating a network incident that occured at around 19:30, updates to follow.
Resolution: Service restored and back to normal at around 19:50. This was due to a network issue that affected some of our BGP sessions which caused routing problems to certain destinations.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 20, 10:20 AM (1¼ years ago)
CLOSED
Oct 20, 10:45 AM (1¼ years ago)
DESCRIPTION
Lines connected on the witless LNS dropped and are in process of reconnecting.
Resolution: The Witless LNSs suffered a crash which was caused by a bug in our CQM graphing code. This bug has been fixed and is in the process of being tested and we will then roll out the new software to the the Witless LNSs overnight as 'emergency' planned work.

MAJOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 30, 03:24 PM (1¼ years ago)
CLOSED
Sep 30, 04:36 PM (1¼ years ago)
DESCRIPTION
at 15:24 we saw most of our TalkTalk circuits drop. They are in the process of reconnecting.
Resolution: Lines reconnected within a couple of minutes and we're investigating this further with TalkTalk. Further updates about this will be on https://aastatus.net/42468