Last 10 Maintenance posts, sorted by last updated
Order posts by limited to posts

MAINTENANCE Completed L2TP
AFFECTING
L2TP
STARTED
Jan 18, 04:00 AM (23½ days ago)
CLOSED
Jan 18, 07:22 AM (23¼ days ago)
DESCRIPTION
We will be upgrading the software on one of our L2TP servers over the weekend. This means moving customers between routers at 4AM on Saturday and 4AM on Sunday. This will cause a brief outage as customers reconnect.
Resolution: The router was upgraded and put back in to service at 7AM Saturday and lines were moved back then (Earlier than originally planned)

MAINTENANCE Completed Office
AFFECTING
Office
STARTED
Nov 28, 02:00 PM (2¼ months ago)
CLOSED
Jan 06, 03:00 PM (1 month ago)
DESCRIPTION

Here are our office opening times over Christmas and new year:

  • Mon 23rd: Normal hours (8-6)
  • Tue 24th: 8-3
  • Wed 25th: Closed
  • Thurs 26th: Closed
  • Fri 27th: 9-5
  • Sat 28th: 10-2
  • Sun 29th: Closed
  • Mon 30th: 9-5
  • Tues 31st: 8-3
  • Wed 1st: Closed
  • Thurs 2nd: back to normal (8-6)

Resolution:

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Nov 19, 09:30 AM (2¾ months ago)
CLOSED
Jan 23, 07:00 PM (17¾ days ago)
DESCRIPTION

The work outlined below will start from Saturday 23rd November.

Background:

Our FireBrick team has been working on the 'hang' problem that we faced with the LNSs earlier in the year.

The nature of the problem has made investigating the problem very time consuming as it is extremely difficult to reproduce. However, we do believe that a plausible cause has been identified, and code changes have been made to mitigate the problem.

We have been testing this new code, both in our test lab and on a few select A&A routers, for over two months. During this time the new code has not caused the hardware to hang, where older versions of the code did.

Our next step is to run the new code on our LNSs, the ones our customers connect to for their broadband connections.

We plan to do this slowly, out of hours and in a couple of phases.

We believe the cause of the hang is related to how memory is initially allocated for the tasks the FireBrick will be performing, this means that if the hardware is going to hang then this will most likely happen over the first couple of days (or first couple of hours).

Stage one: (Completed)

We plan to upgrade only one of our LNSs at first. We will move broadband connections on to it in the early hours of the morning and then move them back off a few hours later. This means that during the day, customers will be on the normal set of LNSs.

Then, each night, over the course of two weeks, the LNS will be power cycled and we will move an increasing number of connections over, until it is at the point of taking twice the amount of connections that we'd normally run on an LNS. (We normally run LNSs at around 40% capacity, so twice the number of connections is not a problem.)

Stage two: (Completed)

Once we have confirmed that the hang is not happening, the second phase would be to run customer connections on the upgraded for a few days at a time.

We will go through a cycle of: move connections off, reboot the LNS, move connections on, wait a few days. Repeat. We will do this with an increasing number of connections until it's at the point of taking a normal amount of connections.

Stage Three:

As of the end of 2024, half our LNSs have been running the new software without any problems. From January 7th we will e doing overnight upgrades of the remaining LNSs.

More information:

So as to minimise impact to customers, the work of moving connections off and on will happen overnight between 1AM and 5AM.


Resolution: All of our FireBrick FB900 LNSs and BGP routers have been upgraded successfully.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Oct 23, 03:00 AM (3½ months ago)
CLOSED
Oct 23, 03:51 AM (3½ months ago)
DESCRIPTION
Due to the problems on Tuesday afternoon we have customers unevenly spread across our pool of LNSs. We will move some customers in the early hours of Wednesday morning. These customers will see a drop and reconnect of their service.
Resolution: This work has been completed.

MAINTENANCE Completed Control Pages
AFFECTING
Control Pages
STARTED
Oct 16, 02:00 PM (3¾ months ago)
CLOSED
Oct 16, 04:18 PM (3¾ months ago)
DESCRIPTION
We're performing a database migration which will mean the display of broadband usage stats, VoIP Call records (CDRs) and SMS logs will be unavailable on one control pages on the afternoon of 16 October.
Resolution: This work has been reverted.

MAINTENANCE Completed Routing
AFFECTING
Routing
STARTED
Oct 07, 05:00 PM (4 months ago)
CLOSED
Oct 15, 02:00 PM (3¾ months ago)
DESCRIPTION
We will be enabling a new transit provider this week - Arelion, AS1299. This is in addition to our three existing transit providers: NTT, Colt (aka Lumen/Level3) and Cogent, as well as multiple peering links. This gives us increased capacity but more importantly it gives us new, diverse routes to the rest of the internet which makes us even better connected to the wider world.
Resolution:

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Oct 07, 01:00 AM (4 months ago)
CLOSED
Oct 07, 10:00 AM (4 months ago)
DESCRIPTION
Due to various BT works over the past few weeks we have a small number of customers on our 'spare' LNS. We will be moving these customers off during the early hours of Monday 7th October. Customers on the i.Gormless LNS will have their line drop and reconnect.
Resolution: This work has been competed.

MAINTENANCE Completed VoIP and SIMs
AFFECTING
VoIP and SIMs
STARTED
Aug 08, 03:16 PM (6 months ago)
CLOSED
Nov 19, 09:00 AM (2¾ months ago)
DESCRIPTION
3G is being phased out by all mobile network operators, and will stop working for our data SIMs in early September. The data SIMs we provide already support 4G. In a small number of cases it is possible that devices are too old to support 4G, in which case the device is sadly obsolete and will need replacing in order to keep connectivity. You will be emailed if we think your SIM is one which is in a device still operating via 3G.
Resolution:

MAINTENANCE Completed Quotas
AFFECTING
Quotas
STARTED
Aug 05, 02:30 PM (6 months ago)
CLOSED
Sep 09, 10:00 AM (5 months ago)
DESCRIPTION
Please see the following news page for changes happening to monthly quotas this month: https://www.aa.net.uk/etc/news/update-to-monthly-usage-quotas/
Resolution:

MAINTENANCE Completed NAT64
AFFECTING
NAT64
STARTED
Jul 21, 02:00 PM (6½ months ago)
CLOSED
Jul 21, 03:40 PM (6½ months ago)
DESCRIPTION
We'll be doing a little bit of work on our NAT64 gateways. This will happen during the afternoon/evening of Sunday 21st July. Part of our work will involve testing our failover. There may be short periods of time when the gateway is unavailable. (This is only relevant to those customers who are IPv6 only and specifically make use of our DNS64 and NAT64 services.)
Resolution: This work has been completed.