Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jul 12, 04:33 PM (2¼ days ago)
CLOSED
Jul 12, 04:40 PM (2¼ days ago)
DESCRIPTION
We're investigating a large number of drops and reconnect on our L2TP service.
Resolution: This looks to be a repeat of the drop and reconnect incident we saw earlier in the week - A port flap on our Primary L2TP router caused our backup L2TP router to take over the routing of the service causing customers to drop and reconnect - then the primary L2TP router re-established itself and customers moved back once more. We'll investigate this in more detail to find the root cause of the initial port problems.

MAINTENANCE Completed TalkTalk
AFFECTING
TalkTalk
STARTED
Jul 11, 11:11 AM (3½ days ago)
CLOSED
Jul 11, 11:50 AM (3½ days ago)
DESCRIPTION
At 11:21 we saw a small number of TalkTalk lines drop and reconnect. We're investigating.
Resolution: A TalkTalk backhaul connection failed (twice?) and traffic was re-routed, as a result a small number to TalkTalk connections dropped and reconnected.

MAINTENANCE Assumed Completed VoIP
AFFECTING
VoIP
STARTED
Jul 10, 03:00 AM (5 days ago)
DESCRIPTION
We're upgrading the software on our VoIP servers, we do not anticipate any problems as customers will be moved off the device before the upgrade happens.

MAINTENANCE Completed VoIP and SIMs
AFFECTING
VoIP and SIMs
STARTED
Jul 08, 10:30 AM (6½ days ago)
CLOSED
Jul 08, 05:05 PM (6¼ days ago)
DESCRIPTION
ONSIM are making changes to address some long standing issues with slow connects and multiple connects for outgoing calls. Sorry for the short notice, they have just advised us. We have been nagging them to fix this for some time. It is possible some calls from mobiles may fail, but it is worth retrying if they do. We are hoping this will only last a few minutes, but not depends on the mobile carrier completing the work promptly.
Resolution: We have not had feedback, but tests suggest calls are being consistently reliable now.

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jul 08, 09:29 AM (6½ days ago)
CLOSED
Jul 08, 10:00 AM (6½ days ago)
DESCRIPTION
At around 09:29 we saw a large number of L2TP circuits drops and reconnect. We're investigating the cause as at the moment we're not quite sure!
Resolution:

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Jun 29, 10:35 AM (15½ days ago)
CLOSED
Jun 29, 10:51 AM (15½ days ago)
DESCRIPTION
We're investigating the cause of a number of L2TP services dropping and reconnecting. Update to follow.
Resolution: At this moment we're unsure of the cause - a drop like this is typical of something breaking out on the internet affecting routing for for some locations for a short moment - We can't see a problem or cause of this on our network.

MINOR Closed Broadband and SIMs
AFFECTED
Broadband and SIMs
STARTED
Jun 28, 11:00 AM (16½ days ago)
CLOSED
Jun 28, 03:00 PM (16½ days ago)
DESCRIPTION
Just after 11 AM we saw a number of broadband and Data SIMs drops and reconnect. We're investigating.
Resolution: This incident was caused by the 'B.Gormless' LNS resetting at 11AM and then again at 11:20AM, causing some of our customers to drop and reconnect. After the second drop, the LNS was taken out of service. Staff were at the datacentre and near the LNS at the time , and we believe the resets were caused by a static discharge which caused a hardware reset. The LNS still has its low level 'JTAG' debug port connected and it's this connection that made it vulnerable to static.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Jun 14, 10:00 AM (30½ days ago)
CLOSED
Jul 08, 10:45 AM (6½ days ago)
DESCRIPTION

As you are probably aware A&A develops its own network hardware under the FireBrick brand. We have been, for some time now, using a mixture of the older "FB6000" and the newer "FB9000" models as BGP routers, L2TP routers and LNSs which terminate customer broadband connections; For the most part the FB9000 LNSs have been is use for customers with faster speed lines. We've been using FB9000s for our L2TP services and as BGP routers for over a year now.

You might have seen that we did have some stability problems earlier in the roll out of the FB9000 for our LNS platform and therefore paused the deployment. For some months now, we have running firmware with well established stability and as such will be rolling out FB9000s in replacement of all older FB6000 LNS. This work will happen in small stages during June and July.

This will be mildly customer impacting, causing a brief outage as each line moves across, but no worse than any common LNS firmware update like we have been doing consistently for years. We will do migrations in the early hours of the morning and according to your preferences in the control pages.

We will post updates to this status post when parts of this wider project are scheduled. eg when we schedule moving a set of customers off a specific LNS on to a new one.


Resolution: These migrations have completed. All broadband customers are now on FB9000 LNSs.

MINOR Closed VoIP and SIMs
AFFECTED
VoIP and SIMs
STARTED
Jun 05, 03:20 PM (1¼ months ago)
CLOSED
Jul 08, 10:32 AM (6½ days ago)
DESCRIPTION
We are again seeing an issue where calls from ONSIM SIMs are not immediately connecting, and may seem like several call attempts to the recipient. We are chasing ONSIM on this.
Resolution: This is being worked on as a (short notice) planned work now.

MAINTENANCE Assumed Completed authoritative DNS
AFFECTING
authoritative DNS
STARTED
Jun 03, 10:35 AM (1¼ months ago)
DESCRIPTION

This is only relevant to customers who run their own authoritative DNS servers and use our secondary-dns.co.uk as an additional nameserver.

Overview: We run a "secondary" DNS service for customers where they run the master DNS server and we are secondary slaves. We have a project underway that involves migrating all our authoritative DNS services to a new platform. As part of this we are needing to disable some of the automation we do for adding and updating the customer's master IP address automatically.

The change: From June 17th, If you run your own master DNS server for your domain(s) and secondary-dns.co.uk is a slave, if you change the IP address of your master you will need to contact support@aa.net.uk to request us to update our side.

We have more information about our Authoritative DNS project on our Support Site: https://support.aa.net.uk/New_Authoritive_DNS