Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Planned BT Ethernet
AFFECTING
BT Ethernet
STARTING
Oct 06, 01:00 AM (3 days )
DESCRIPTION
BT have informed us of the following work being carried out that will affect Ethernet circuits (Etherway/Etherflow) . They expect a 2 minute outage in the early hours of October 6th sometime between 1AM and 6AM:

We've found a problem in our core network, so we need to make some changes at multiple sites. Your service outage is expected to be a minor outage lasting less than 2 minutes. There will be loss of capacity during the works as each switch is replaced This will be between 01:01 - 06:00 All work will be completed by 06:00


MAJOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 30, 03:24 PM (2 days ago)
CLOSED
Sep 30, 04:36 PM (2 days ago)
DESCRIPTION
at 15:24 we saw most of our TalkTalk circuits drop. They are in the process of reconnecting.
Resolution: Lines reconnected within a couple of minutes and we're investigating this further with TalkTalk. Further updates about this will be on https://aastatus.net/42468

MINOR Open TalkTalk
AFFECTING
TalkTalk
STARTED
Sep 30, 01:30 PM (2¼ days ago)
DESCRIPTION

We do apologise for the inconvenience these drops are having on our customers.

There have been a number of problems affecting TalkTalk ADSL and VDSL circuits this week. They have been a mix of overnight planned work, overnight planned work going wrong and causing disruption in the mornings and drops during the day.

These are less of a problem where customers have bonded lines with services from different back-haul providers (TalkTalk and BT) but they are disruptive to single-line customers.

Earlier in the week we took our Telehouse interconnect out of service as it was connected to equipment at TalkTalk that was having serious problems. However, we have still seen a few drops during the day. These drops are not affecting all circuits but also don't seem to be localised to particular areas of the country.

TalkTalk say they have no more work to do on the Telehouse side and so over this weekend we will bring back the downed interconnect which will restore resilience. This should not cause a blip, but will be done out of hours.

We are still in dialogue with TalkTalk about the unexplained drops and will report here the findings and outcome.

MINOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 29, 03:04 PM (3 days ago)
CLOSED
Sep 29, 03:30 PM (3 days ago)
DESCRIPTION
We saw a large number of TalkTalk lines drop and reconnect. Investigations underway.
Resolution:

MAJOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Sep 27, 10:00 PM (4¾ days ago)
CLOSED
Sep 28, 08:00 AM (4½ days ago)
DESCRIPTION
Following on from the problems this morning, TalkTalk are performing some emergency work overnight, tonight. The work will start at 10PM and will carry on through to 6AM. They say that their work before midnight should not impact circuits, but the work after midnight may cause some reconnects.
Resolution: TalkTalk have emergency overnight work that failed and turned in to an outage causing circuits to drop. We took our Telehouse connection out of service and lines reconnected via our Equinix link and have been stable since 08:30. We'll leave the Telehouse link down for the moment in case TalkTalk schedule further work on it.

MINOR Closed TalkTalk DSL
AFFECTED
TalkTalk DSL
STARTED
Sep 27, 09:00 AM (5¼ days ago)
CLOSED
Sep 27, 12:00 PM (5¼ days ago)
DESCRIPTION
Overnight and this morning we've seen a number of TalkTalk circuits dropping and reconnecting.
Resolution: Overnight planned work by TalkTalk and then problems afterwards caused some lines to drop and reconnect. TalkTalk have given the all-clear and will send further details of the specific problem at a later time.

MINOR Closed Routing
AFFECTED
Routing
STARTED
Sep 23, 11:10 PM (8¾ days ago)
CLOSED
Sep 23, 11:51 PM (8¾ days ago)
DESCRIPTION
One of our routers is having problems which has caused some packet loss to some destinations. This is being looked in to.
Resolution: One of our BGP routers locked up during routine maintenance and needed manual intervention. This caused some packetloss for a few minutes. The router has been taken out of service whilst we investigate it further. The routing it was doing has been moved on to other routers and service is back to normal.

MAINTENANCE Completed LNS Upgrades
AFFECTING
LNS Upgrades
STARTED
Sep 21, 12:00 AM (11¾ days ago)
CLOSED
Sep 22, 04:36 PM (10 days ago)
DESCRIPTION
We are performing routine router upgrades overnight, including a "rolling" LNS upgrade. This will cause customer lines on the 'Witless' set of LNSs to drop and reconnect when thy are on is updated. We will be doing one LNS a night and the work starts from 1AM. The 'Witless' LNSs manage our FTTP estate as well as some 80M FTTC lines. More information about LNS switchovers here: https://support.aa.net.uk/LNS_Switches
Resolution: This work has been completed successfully.

MAINTENANCE Completed General
AFFECTING
General
STARTED
Sep 16, 07:00 AM (16½ days ago)
CLOSED
Sep 16, 07:19 AM (16½ days ago)
DESCRIPTION
We will be bringing up a new cross-datacentre fibre link between our two London datacentres. (Telehouse and Harbour Exchange) We already have two links, this is a third to further increase resilience. No customer impact is expected.
Resolution: Change reverted due to problems.

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 12, 11:41 AM (20¼ days ago)
CLOSED
Sep 12, 12:10 PM (20¼ days ago)
DESCRIPTION
We saw a number of line disconnect and reconnect at around 11:41. We're investigating.
Resolution: This drop happened whist we were moving BT traffic from one datacenrre to another. This was being carried out as BT are performing maintenance on their side of the links on one of the datacentres later this week. There is usually no drop in service when we move traffic in this way, but this time a small portion of our customers were affected. We do apologise to those who had their service interrupted.

MINOR Closed Data SIMs
AFFECTED
Data SIMs
STARTED
Sep 07, 01:10 PM (25¼ days ago)
CLOSED
Sep 07, 01:34 PM (25¼ days ago)
DESCRIPTION
We're seeing some data SIMs disconnecting and reconnecting. This looks to be upstream of us, we're investigating further.
Resolution: SIMs had all reconnected by 13:34. This was planned work by our carrier to move connections between their routers.