Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MINOR Open BT
AFFECTING
BT
STARTED
Jul 23, 11:20 AM (3¾ days ago)
DESCRIPTION
At 11:20 we saw a number of customers drop and reconnect. We're investigating the cause.

MAINTENANCE Completed NAT64
AFFECTING
NAT64
STARTED
Jul 21, 02:00 PM (5¾ days ago)
CLOSED
Jul 21, 03:40 PM (5½ days ago)
DESCRIPTION
We'll be doing a little bit of work on our NAT64 gateways. This will happen during the afternoon/evening of Sunday 21st July. Part of our work will involve testing our failover. There may be short periods of time when the gateway is unavailable. (This is only relevant to those customers who are IPv6 only and specifically make use of our DNS64 and NAT64 services.)
Resolution: This work has been completed.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Jul 18, 09:00 AM (8¾ days ago)
CLOSED
Jul 18, 10:00 AM (8¾ days ago)
DESCRIPTION
There was intermittent problems with some calls unable to be connected. This has been tracked to a problem with our RADIUS servers which manage call routing logic and registrations.
Resolution:

MAINTENANCE Completed VoIP and SIMs
AFFECTING
VoIP and SIMs
STARTED
Jul 15, 07:20 AM (12 days ago)
CLOSED
Jul 17, 03:05 PM (9½ days ago)
DESCRIPTION
We have an upgrade to our VoIP systems to fix an issue where calls from SIP2SIM to SIP2SIM would lose CLI (show "unknown"). This will be tested over the next few days before being made permanent.
Resolution:

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jul 12, 04:33 PM (14½ days ago)
CLOSED
Jul 12, 04:40 PM (14½ days ago)
DESCRIPTION
We're investigating a large number of drops and reconnect on our L2TP service.
Resolution: This looks to be a repeat of the drop and reconnect incident we saw earlier in the week - A port flap on our Primary L2TP router caused our backup L2TP router to take over the routing of the service causing customers to drop and reconnect - then the primary L2TP router re-established itself and customers moved back once more. We'll investigate this in more detail to find the root cause of the initial port problems.

MAINTENANCE Completed TalkTalk
AFFECTING
TalkTalk
STARTED
Jul 11, 11:11 AM (15¾ days ago)
CLOSED
Jul 11, 11:50 AM (15¾ days ago)
DESCRIPTION
At 11:21 we saw a small number of TalkTalk lines drop and reconnect. We're investigating.
Resolution: A TalkTalk backhaul connection failed (twice?) and traffic was re-routed, as a result a small number to TalkTalk connections dropped and reconnected.

MAINTENANCE Completed VoIP
AFFECTING
VoIP
STARTED
Jul 10, 03:00 AM (17 days ago)
CLOSED
Jul 17, 10:00 AM (9¾ days ago)
DESCRIPTION
We're upgrading the software on our VoIP servers, we do not anticipate any problems as customers will be moved off the device before the upgrade happens.
Resolution: This work has completed.

MAINTENANCE Completed VoIP and SIMs
AFFECTING
VoIP and SIMs
STARTED
Jul 08, 10:30 AM (18¾ days ago)
CLOSED
Jul 08, 05:05 PM (18½ days ago)
DESCRIPTION
ONSIM are making changes to address some long standing issues with slow connects and multiple connects for outgoing calls. Sorry for the short notice, they have just advised us. We have been nagging them to fix this for some time. It is possible some calls from mobiles may fail, but it is worth retrying if they do. We are hoping this will only last a few minutes, but not depends on the mobile carrier completing the work promptly.
Resolution: We have not had feedback, but tests suggest calls are being consistently reliable now.

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jul 08, 09:29 AM (18¾ days ago)
CLOSED
Jul 08, 10:00 AM (18¾ days ago)
DESCRIPTION
At around 09:29 we saw a large number of L2TP circuits drops and reconnect. We're investigating the cause as at the moment we're not quite sure!
Resolution:

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jun 29, 10:35 AM (27¾ days ago)
CLOSED
Jun 29, 10:51 AM (27¾ days ago)
DESCRIPTION
We're investigating the cause of a number of L2TP services dropping and reconnecting. Update to follow.
Resolution: At this moment we're unsure of the cause - a drop like this is typical of something breaking out on the internet affecting routing for for some locations for a short moment - We can't see a problem or cause of this on our network.

MINOR Closed Broadband and SIMs
AFFECTED
Broadband and SIMs
STARTED
Jun 28, 11:00 AM (28¾ days ago)
CLOSED
Jun 28, 03:00 PM (28½ days ago)
DESCRIPTION
Just after 11 AM we saw a number of broadband and Data SIMs drops and reconnect. We're investigating.
Resolution: This incident was caused by the 'B.Gormless' LNS resetting at 11AM and then again at 11:20AM, causing some of our customers to drop and reconnect. After the second drop, the LNS was taken out of service. Staff were at the datacentre and near the LNS at the time , and we believe the resets were caused by a static discharge which caused a hardware reset. The LNS still has its low level 'JTAG' debug port connected and it's this connection that made it vulnerable to static.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Jun 14, 10:00 AM (1¼ months ago)
CLOSED
Jul 08, 10:45 AM (18¾ days ago)
DESCRIPTION

As you are probably aware A&A develops its own network hardware under the FireBrick brand. We have been, for some time now, using a mixture of the older "FB6000" and the newer "FB9000" models as BGP routers, L2TP routers and LNSs which terminate customer broadband connections; For the most part the FB9000 LNSs have been is use for customers with faster speed lines. We've been using FB9000s for our L2TP services and as BGP routers for over a year now.

You might have seen that we did have some stability problems earlier in the roll out of the FB9000 for our LNS platform and therefore paused the deployment. For some months now, we have running firmware with well established stability and as such will be rolling out FB9000s in replacement of all older FB6000 LNS. This work will happen in small stages during June and July.

This will be mildly customer impacting, causing a brief outage as each line moves across, but no worse than any common LNS firmware update like we have been doing consistently for years. We will do migrations in the early hours of the morning and according to your preferences in the control pages.

We will post updates to this status post when parts of this wider project are scheduled. eg when we schedule moving a set of customers off a specific LNS on to a new one.


Resolution: These migrations have completed. All broadband customers are now on FB9000 LNSs.

MINOR Closed VoIP and SIMs
AFFECTED
VoIP and SIMs
STARTED
Jun 05, 03:20 PM (1¾ months ago)
CLOSED
Jul 08, 10:32 AM (18¾ days ago)
DESCRIPTION
We are again seeing an issue where calls from ONSIM SIMs are not immediately connecting, and may seem like several call attempts to the recipient. We are chasing ONSIM on this.
Resolution: This is being worked on as a (short notice) planned work now.