Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Nov 30, 12:32 AM (2 days ago)
CLOSED
Nov 30, 03:06 AM (2 days ago)
DESCRIPTION
We have two VoIP servers, A.Voiceless and B.Voiceless. Currently B.Voiceless is offline and A.Voiceless has been set to take all traffic whilst we investigate the problem.
Resolution:

MINOR Closed Website
AFFECTED
Website
STARTED
Nov 17, 12:18 PM (14½ days ago)
CLOSED
Nov 25, 11:40 AM (6½ days ago)
DESCRIPTION
The VoIP rate card is giving errors, this is being investigated.
Resolution: These pages are working again now!

MAINTENANCE Completed Control Page
AFFECTING
Control Page
STARTED
Nov 25, 06:00 AM (6¾ days ago)
CLOSED
Nov 25, 06:37 AM (6¾ days ago)
DESCRIPTION
We're doing some work with the Control Pages that will mean there will be a short moment where the pages will not be available. This is expected to only last a few minutes and will happen between 6AM and 7AM.
Resolution:

MINOR Closed Connectivity
AFFECTED
Connectivity
STARTED
Nov 13, 09:00 AM (18¾ days ago)
CLOSED
Nov 13, 08:00 PM (18¼ days ago)
DESCRIPTION

We've had some problems today with Apple related traffic causing some of our routers to get busy. This has affected some customers with some latency and packetloss to some destinations on the Internet. The problem is that the recent software updates from Apple are being downloaded via our 'transit' links instead of our 'peering' links.

The bulk of our traffic comes to us over our peering links where we have higher capacity connections in to exchange points such as LINX and LONAP. However, even though we have direct peering with Apple of these higher capacity links, today's Apple traffic is being set over our smaller transit links.

We have tried to balance the traffic across the transit links we have, but this hasn't been perfect. We've emailed Apple, requesting that they send traffic to us over the peering links but we've only had an automatic ticket confirmation reply so far.


Resolution:

Traffic levels Friday evening were the highest we've ever seen, however the traffic was spread over multiple routers and were at levels that didn't impact our service negatively. We've had no word back from Apple about them using our peering links. We'll be looking at upgrading our transit capacity as a result of this so as to be able to cope better in the future.

We also understand that there was various reports in the media about Apple having problems yesterday and there were widely reported problems with update downloads failing. We're not sure is this is related to the reasons why Apple traffic was being set via transit instead of our direct peering with Apple though.

2020-11-18 Further update: We're in positive discussions with Apple in regard to improving the connectivity between our networks.


MINOR Closed DNS
AFFECTED
DNS
STARTED
Nov 17, 12:30 PM (14½ days ago)
CLOSED
Nov 17, 03:00 PM (14½ days ago)
DESCRIPTION
We've identified a fault in our DNS updates for domains that we host. For the moment, any changes to DNS records made on the Control Pages will not be made live whilst we investigate this.
Resolution:

MINOR Closed Domains - email
AFFECTED
Domains - email
STARTED
Nov 17, 11:30 AM (14½ days ago)
CLOSED
Nov 17, 01:15 PM (14½ days ago)
DESCRIPTION
Email to the company are currently sending bounce messages - we are investigating this.
Resolution: Email is now ok. This problem is in relation to https://aastatus.net/38227

MAINTENANCE Completed Control Pages
AFFECTING
Control Pages
STARTED
Nov 15, 10:00 AM (16½ days ago)
CLOSED
Nov 15, 12:17 PM (16½ days ago)
DESCRIPTION
One of the servers behind the Control Pages is being upgraded on Sunday 15th November from 10AM onwards. During this time parts of the control pages will be unavailable.
Resolution:

MAINTENANCE Completed TalkTalk FTTC
AFFECTING
TalkTalk FTTC
STARTED
Nov 13, 01:30 AM (19 days ago)
CLOSED
Nov 13, 09:15 AM (18¾ days ago)
DESCRIPTION

We have been informed by our supplier that Openreach intend to carry out maintenance within their network. During the maintenance window you may experience one period of outage to the maximum specified below.

This is to advise that Openreach will be applying an urgent network element fix on 13th Nov 2020 from 01:30 to 06:00. Anticipated Downtime : 20 Minutes.

Exchanges listed below


Resolution: Assuming this work was carried out, we saw little to no line drops as a result.

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 13, 01:30 AM (19 days ago)
CLOSED
Nov 13, 09:14 AM (18¾ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 13th November from 1:30AM:


Resolution: Assuming this work was carried out, we saw little to no line drops as a result.

MAINTENANCE Completed Maidenhead
AFFECTING
Maidenhead
STARTED
Nov 13, 05:30 AM (18¾ days ago)
CLOSED
Nov 12, 03:52 PM (19¼ days ago)
DESCRIPTION
We will be adjusting some of the routing between our London datacentres and Maidenhead. This work will start from 5:30 and should last about 30 minutes. During this time there may be slight disruption to accessing services which we host in Maidenhead. This includes access to Control Pages, VoIP and physical servers that we host for customers.
Resolution: This is been postponed until further notice.

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 12, 01:30 AM (20 days ago)
CLOSED
Nov 12, 10:55 AM (19½ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 12th November from 1:30AM:


Resolution:

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 11, 01:30 AM (21 days ago)
CLOSED
Nov 12, 10:54 AM (19½ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 11th November from 1:30AM:


Resolution:

MAINTENANCE Completed FTTC FTTP
AFFECTING
FTTC FTTP
STARTED
Nov 10, 01:30 AM (22 days ago)
CLOSED
Nov 11, 09:18 PM (20¼ days ago)
DESCRIPTION

Openreach are performing a series of upgrades over a 4-day period commencing 10th Nov, with sets of exchanges being upgraded on different nights. The upgrade will take place on a nightly basis between 01:30 and 06:00 hours.

Anticipated Downtime : 20 Minutes

The exchanges listed below are due to have work carried out on 10th November from 1:30AM:


Resolution:

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Nov 11, 02:45 PM (20½ days ago)
CLOSED
Nov 11, 03:08 PM (20½ days ago)
DESCRIPTION
We're investigating broadband drops
Resolution: The fault lasted from 14:45 through to 15:08 and was affecting some BT circuits that were on 6 of our 20 LNSs (the routers at our side that broadband lines connect to). Of those affected some had a short blip and others had a high amount of packetloss. The cause of the drops and loss was loss of connectivity to some parts of BTs network. However, this is likely a problem our side rather than at BT. The affected LNSs are on the same set of network switches. We've not yet identified the root cause, but our investigations will continue, and we'll be planning some out of hours work in relation to this.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Nov 05, 09:45 AM (26¾ days ago)
CLOSED
Nov 05, 09:44 PM (26¼ days ago)
DESCRIPTION
We're raising a fault with one of our VoIP carriers where some incoming calls are ending after a few seconds. This isn't all calls, it may be specific to calls from some mobile numbers - we're still investigating.
Resolution: This issue only related to ported numbers on one of our carriers, and even then not all ported numbers as it depended on the route calls took getting in to the carrier's network. The carrier fixed the problem with the SIP messages and calls have been fine since 21:45 this evening. Ported numbers we have with our other carriers were unaffected.