Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Assumed Completed Maidenhead
AFFECTING
Maidenhead
STARTING
Dec 10, 03:00 AM (5¼ days )
DESCRIPTION

We will be bringing up additional routing between our London sites and Maidenhead. This is a period of 'at risk' as routing to services in Maidenhead (VoIP, Control pages, colocation) may be affected for brief moments whilst the work is being carried out.

Starting: Wednesday 10th December, 3AM

Ending by: Wednesday 10th December, 5AM


MAINTENANCE Completed Internal Routing
AFFECTING
Internal Routing
STARTED
Dec 03, 03:00 AM (1½ days ago)
CLOSED
Dec 03, 04:33 AM (1½ days ago)
DESCRIPTION

We're adjusting some of our internal routing between Maidenhead and London, this is general maintenance and is a period of 'at risk' whilst we do the work as there may be moments when our internal traffic involving maidenhead is routed over different links.

Starting: Wednesday 3rd December, 3AM

Ending by: Wednesday 3rd December, 5AM


Resolution: This work has been completed successfully.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Nov 30, 12:32 AM (4¾ days ago)
CLOSED
Nov 30, 03:06 AM (4½ days ago)
DESCRIPTION
We have two VoIP servers, A.Voiceless and B.Voiceless. Currently B.Voiceless is offline and A.Voiceless has been set to take all traffic whilst we investigate the problem.
Resolution:

MAINTENANCE Completed Control Page
AFFECTING
Control Page
STARTED
Nov 25, 06:00 AM (9½ days ago)
CLOSED
Nov 25, 06:37 AM (9½ days ago)
DESCRIPTION
We're doing some work with the Control Pages that will mean there will be a short moment where the pages will not be available. This is expected to only last a few minutes and will happen between 6AM and 7AM.
Resolution:

MINOR Closed DNS
AFFECTED
DNS
STARTED
Nov 17, 12:30 PM (17¼ days ago)
CLOSED
Nov 17, 03:00 PM (17 days ago)
DESCRIPTION
We've identified a fault in our DNS updates for domains that we host. For the moment, any changes to DNS records made on the Control Pages will not be made live whilst we investigate this.
Resolution:

MINOR Closed Website
AFFECTED
Website
STARTED
Nov 17, 12:18 PM (17¼ days ago)
CLOSED
Nov 25, 11:40 AM (9¼ days ago)
DESCRIPTION
The VoIP rate card is giving errors, this is being investigated.
Resolution: These pages are working again now!

MINOR Closed Domains - email
AFFECTED
Domains - email
STARTED
Nov 17, 11:30 AM (17¼ days ago)
CLOSED
Nov 17, 01:15 PM (17¼ days ago)
DESCRIPTION
Email to the company are currently sending bounce messages - we are investigating this.
Resolution: Email is now ok. This problem is in relation to https://aastatus.net/38227

MAINTENANCE Completed Control Pages
AFFECTING
Control Pages
STARTED
Nov 15, 10:00 AM (19¼ days ago)
CLOSED
Nov 15, 12:17 PM (19¼ days ago)
DESCRIPTION
One of the servers behind the Control Pages is being upgraded on Sunday 15th November from 10AM onwards. During this time parts of the control pages will be unavailable.
Resolution:

MINOR Closed Connectivity
AFFECTED
Connectivity
STARTED
Nov 13, 09:00 AM (21¼ days ago)
CLOSED
Nov 13, 08:00 PM (21 days ago)
DESCRIPTION

We've had some problems today with Apple related traffic causing some of our routers to get busy. This has affected some customers with some latency and packetloss to some destinations on the Internet. The problem is that the recent software updates from Apple are being downloaded via our 'transit' links instead of our 'peering' links.

The bulk of our traffic comes to us over our peering links where we have higher capacity connections in to exchange points such as LINX and LONAP. However, even though we have direct peering with Apple of these higher capacity links, today's Apple traffic is being set over our smaller transit links.

We have tried to balance the traffic across the transit links we have, but this hasn't been perfect. We've emailed Apple, requesting that they send traffic to us over the peering links but we've only had an automatic ticket confirmation reply so far.


Resolution:

Traffic levels Friday evening were the highest we've ever seen, however the traffic was spread over multiple routers and were at levels that didn't impact our service negatively. We've had no word back from Apple about them using our peering links. We'll be looking at upgrading our transit capacity as a result of this so as to be able to cope better in the future.

We also understand that there was various reports in the media about Apple having problems yesterday and there were widely reported problems with update downloads failing. We're not sure is this is related to the reasons why Apple traffic was being set via transit instead of our direct peering with Apple though.

2020-11-18 Further update: We're in positive discussions with Apple in regard to improving the connectivity between our networks.


MAINTENANCE Completed Maidenhead
AFFECTING
Maidenhead
STARTED
Nov 13, 05:30 AM (21½ days ago)
CLOSED
Nov 12, 03:52 PM (22 days ago)
DESCRIPTION
We will be adjusting some of the routing between our London datacentres and Maidenhead. This work will start from 5:30 and should last about 30 minutes. During this time there may be slight disruption to accessing services which we host in Maidenhead. This includes access to Control Pages, VoIP and physical servers that we host for customers.
Resolution: This is been postponed until further notice.

MAINTENANCE Completed TalkTalk FTTC
AFFECTING
TalkTalk FTTC
STARTED
Nov 13, 01:30 AM (21¾ days ago)
CLOSED
Nov 13, 09:15 AM (21¼ days ago)
DESCRIPTION

We have been informed by our supplier that Openreach intend to carry out maintenance within their network. During the maintenance window you may experience one period of outage to the maximum specified below.

This is to advise that Openreach will be applying an urgent network element fix on 13th Nov 2020 from 01:30 to 06:00. Anticipated Downtime : 20 Minutes.

Exchanges listed below


Resolution: Assuming this work was carried out, we saw little to no line drops as a result.

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 13, 01:30 AM (21¾ days ago)
CLOSED
Nov 13, 09:14 AM (21¼ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 13th November from 1:30AM:


Resolution: Assuming this work was carried out, we saw little to no line drops as a result.

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 12, 01:30 AM (22¾ days ago)
CLOSED
Nov 12, 10:55 AM (22¼ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 12th November from 1:30AM:


Resolution:

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Nov 11, 02:45 PM (23 days ago)
CLOSED
Nov 11, 03:08 PM (23 days ago)
DESCRIPTION
We're investigating broadband drops
Resolution: The fault lasted from 14:45 through to 15:08 and was affecting some BT circuits that were on 6 of our 20 LNSs (the routers at our side that broadband lines connect to). Of those affected some had a short blip and others had a high amount of packetloss. The cause of the drops and loss was loss of connectivity to some parts of BTs network. However, this is likely a problem our side rather than at BT. The affected LNSs are on the same set of network switches. We've not yet identified the root cause, but our investigations will continue, and we'll be planning some out of hours work in relation to this.

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 11, 01:30 AM (23¾ days ago)
CLOSED
Nov 12, 10:54 AM (22¼ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 11th November from 1:30AM:


Resolution:

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 10, 01:30 AM (24¾ days ago)
CLOSED
Nov 11, 09:18 PM (22¾ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 10th November from 1:30AM:


Resolution:

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Nov 05, 09:45 AM (29¼ days ago)
CLOSED
Nov 05, 09:44 PM (28¾ days ago)
DESCRIPTION
We're raising a fault with one of our VoIP carriers where some incoming calls are ending after a few seconds. This isn't all calls, it may be specific to calls from some mobile numbers - we're still investigating.
Resolution: This issue only related to ported numbers on one of our carriers, and even then not all ported numbers as it depended on the route calls took getting in to the carrier's network. The carrier fixed the problem with the SIP messages and calls have been fine since 21:45 this evening. Ported numbers we have with our other carriers were unaffected.