Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Open LNS Upgrades
AFFECTING
LNS Upgrades
STARTED
Feb 26, 01:00 AM (2 days ago)
DESCRIPTION
We are performing routine router upgrades overnight, including a "rolling" LNS upgrade. This will cause customer lines to drop and reconnect when the LNS they are on is updated. We will be doing one LNS a night and the work starts from 1AM. Due to the number of LNSs this will last a few weeks. More information about LNS switchovers here: https://support.aa.net.uk/LNS_Switches

MAJOR Closed General
AFFECTED
General
STARTED
Feb 25, 12:22 PM (2½ days ago)
CLOSED
Feb 25, 12:28 PM (2½ days ago)
DESCRIPTION
At around 12:22 we had a routing problem which affected general internet access.
Resolution:

MAINTENANCE Completed BT DSL lines
AFFECTING
BT DSL lines
STARTED
Feb 25, 12:01 AM (3 days ago)
CLOSED
Feb 25, 08:30 AM (2½ days ago)
DESCRIPTION

BT have planned work on our links used for broadband lines in our Telehouse datacentre. We have resilient links in Harbour Exchange datacentre and so traffic will be seamlessly moved before BT's work is carried out.

We are not expecting any impact on customers.

Starting: 25th Feb, 00:01. Ending: 25 Feb 06:00


Resolution: BT completed their work at 01:10, and we enabled our links at 08:30.

MAJOR Closed General Internet Access
AFFECTED
General Internet Access
STARTED
Feb 18, 06:00 PM (9¼ days ago)
CLOSED
Feb 18, 06:10 PM (9¼ days ago)
DESCRIPTION
We saw a few minutes of packet loss and connectivity problems at around 6PM. We're investigating further.
Resolution:

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Feb 02, 11:00 AM (25½ days ago)
CLOSED
Feb 14, 09:29 AM (13½ days ago)
DESCRIPTION
When you connect on PPP for broadband your router gets a reply message. Some routers allow you to see this, e.g. a FireBrick. Until now this has simply been a circuit ID, e.g. BBEU and a number. We are changing this to include the speed information (down/up) after the circuit. The plan is for FireBricks to be able to pick up the uplink speed and adjust rates for bonded uplink automatically. E.g. BBEU39074416 16789000/1171000 for an ADSL line that is synced at 16.789M down and 1.171M up.
Resolution:

MINOR Closed General Internet Access
AFFECTED
General Internet Access
STARTED
Feb 11, 09:15 AM (16½ days ago)
CLOSED
Feb 11, 11:30 AM (16½ days ago)
DESCRIPTION

We have 5 routers for transit and we are seeing low levels of packet loss on one of these routers.

The cause of this is excess traffic due to our direct interconnect with Amazon being down and Amazon traffic being sent over our transit connections instead of the interconnect.

We have contacted Amazon and hope to have the interconnect working again soon, but we're not yet sure why they have taken it down.


Resolution:

This problem was caused by our interconnect to Amazon going down due to maintenance at the Amazon side. Amazon have a policy not to route traffic over exchange points such as LINX and LONAP (where we have a lot of excess capacity) but instead would send traffic over our transit links (which have less excess capacity).

To prevent this from happening again: We are already in the process of increasing our transit capacity anyway, and we will investigate having a second interconnect in to Amazon.

Further update: a second interconnect to Amazon is underway.


MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Feb 11, 03:00 AM (16¾ days ago)
CLOSED
Feb 11, 04:29 AM (16¾ days ago)
DESCRIPTION
We will be performing so work on the links we have to BT that provide our broadband services. This work will happen in the early hours of Thursday 11th February. We expect this work to have little impact on customer lines however, there may be a small number of lines that will have a PPP restart.
Resolution: This work has been completed.

MAINTENANCE Completed VoIP and SIMs
AFFECTING
VoIP and SIMs
STARTED
Feb 02, 09:00 AM (25½ days ago)
CLOSED
Feb 03, 05:00 PM (24¼ days ago)
DESCRIPTION
We have been advised that Manx telecom will be carrying out some planned works on their core network which should be considered as at risk. This may affect calls over our SIPSIM service We have a requirement to migrate the voice interconnect of one of our upstream carriers to new connectivity links. Traffic will be changed in 2 phases over 2 days on 2nd and 3rd February. We do not anticipate any service impact however, there is an increased risk of disruption. The service will be closely monitored and will be reverted if problems occur. Maintenance Window From: 02/02/2021 09:00 - 16:30 GMT To: 03/02/2021 09:00 - 16:30 GMT
Resolution:

MAINTENANCE Completed VoIP, Control Pages, Customer collocation
AFFECTING
VoIP, Control Pages, Customer collocation
STARTED
Feb 06, 12:00 PM (21½ days ago)
CLOSED
Feb 08, 10:17 AM (19½ days ago)
DESCRIPTION

Our Maidenhead datacentre is undergoing major electrical infrastructure upgrades which will affect our services in the following ways:

Sunday 31st January, Afternoon. We moved our equipment on to temporary power feeds.

Saturday 6th February, Afternoon: Power at risk as the datacentre replace our original power feeds - We will be running on the temporary power feeds during this work so service should be unaffected.

Sunday 7th February, Afternoon: We will be moving from the temporary power feeds to the new feeds.

Broadband and general internet access will be unaffected.
Resolution:

MAINTENANCE Completed VoIP, Control Pages, Customer collocation
AFFECTING
VoIP, Control Pages, Customer collocation
STARTED
Feb 07, 02:00 PM (20¼ days ago)
CLOSED
Feb 07, 02:45 PM (20¼ days ago)
DESCRIPTION

Our Maidenhead datacentre is undergoing major electrical infrastructure upgrades which will affect our services in the following ways:

Sunday 31st January, Afternoon, We moved all our equipment on to temporary power feeds.

Saturday 6th February, Afternoon: Power at risk as the datacentre replace our original power feeds - We will be running on the temporary power feeds during this work so service should be unaffected.

Sunday 7th February, Afternoon: We moving our equipment, again, this time on the new power feeds. Similar to the work on 31st January there will be a period of a few minutes where some of our services in Maidenhead will be offline. This will be minimal due to most devices having dual power supply feeds and will not need to be powered down during this work.

Broadband and general internet access will be unaffected.
Resolution: This work has been completed with minimal impact to service.

MAINTENANCE Completed VoIP, Control Pages, Customer collocation
AFFECTING
VoIP, Control Pages, Customer collocation
STARTED
Jan 31, 02:00 PM (27¼ days ago)
CLOSED
Jan 31, 03:18 PM (27¼ days ago)
DESCRIPTION

Our Maidenhead datacentre is undergoing major electrical infrastructure upgrades which will affect our services in the following ways:

Sunday 31st January, Afternoon. We will be powering down all our equipment in order to move on to temporary power feeds. There will be a period of up to 30 minutes where our services in Maidenhead will be offline.

Saturday 6th February, Afternoon: Power at risk as the datacentre replace our original power feeds - We will be running on the temporary power feeds during this work so service should be unaffected.

Sunday 7th February, Afternoon: We will be powering down all our equipment again in order to move on to the new power feeds. Similar to the work on 31st January there will be a period of up to 30 minutes where our services in Maidenhead will be offline.

Services that will be affected when power is moved between feeds:

VoIP services,
Access to the Control Page,
Dedicated servers we host for customer (collocation),
Web pages we host for customers

Broadband and general internet access will be unaffected.
Resolution:

MAINTENANCE Completed General Internet Access
AFFECTING
General Internet Access
STARTED
Jan 28, 11:00 PM (30 days ago)
CLOSED
Jan 29, 07:38 AM (29½ days ago)
DESCRIPTION
The London Internet Exchange (LINX) are upgrading the software on some of their network switches in the Harbour Exchange (Equinix LD8) datacentre, to which we have a connection to. So as to minimise connectivity problems we will take down our connection to them beforehand.

We do not anticipate that this work will affect our customers directly as the traffic will be going over our LINX in Telehouse instead.


Resolution: LINX have completed their work, and our connection has re-established.