Last 10 Maintenance posts, sorted by last updated
Order posts by limited to posts

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Feb 14, 12:00 PM (1¼ months ago)
CLOSED
Feb 18, 03:00 AM (1¼ months ago)
DESCRIPTION

Happening Monday and Tuesday early mornings:

Most of our LNSs have extra diagnostic and debugging hardware installed which has been used to help us track down the problems we were having with them in early 2024. We are now in a position where we can remove this. We have planned to do this work over two nights: the early hours of February 17th and 18th.

This work will involve us moving customers off the LNSs as we do the work. In practice this means customer connections will experience a drop and reconnect at around 1AM and 4AM on the 17th and 18th February.


Resolution: This work was completed at around 3AM pn Tuesday morning.

MAINTENANCE Completed BT
AFFECTING
BT
STARTED
Feb 12, 01:00 AM (1½ months ago)
CLOSED
Jan 29, 11:35 AM (2 months ago)
DESCRIPTION

BT are performing "Simultaneous Upgrades" which will affect all BT ADSL, VDSL and FTTP services during the early hours of February 12th. It is disappointing that this work is a multi-site activity as we have multiple links to BT in multiple datacentres so as to prevent outages when BT have work to do.

They say:

Start: 2025-02-12 00:01. We're simultaneously upgrading some 21C network software across multiple sites. The PW window is from 00:01 until 07:00, and we will start at 00:01 with non-disruptive pre-checks. From 01:00 we will start to re-boot the devices to the new version of code. This will cause an outage to all customers of up to 15 minutes whilst the device restarts. However, if roll-back is required, the outage could exceed 1hr. All work will be complete by 07:00.

In practice, we expect customers with BT provided connections to drop and reconnect a couple of times in the early hours of February 12th.


Resolution: CANCELLED! This notification fas sent to us by BT in error - their work is not expected to affect our services at all.

MAINTENANCE Completed Email
AFFECTING
Email
STARTED
Feb 10, 09:00 AM (1½ months ago)
CLOSED
Feb 18, 02:11 PM (1¼ months ago)
DESCRIPTION
We will be doing work on our email servers which should not affect customers. The work involves rebuilding one of the two back end file stores and will be taking place across multiple days.
Resolution: The initial set of work has been completed successfully and without any customer impact. We'll be doing further work in a couple of weeks time, and will post a new status post nearer the time.

MAINTENANCE Completed L2TP
AFFECTING
L2TP
STARTED
Jan 18, 04:00 AM (2¼ months ago)
CLOSED
Jan 18, 07:22 AM (2¼ months ago)
DESCRIPTION
We will be upgrading the software on one of our L2TP servers over the weekend. This means moving customers between routers at 4AM on Saturday and 4AM on Sunday. This will cause a brief outage as customers reconnect.
Resolution: The router was upgraded and put back in to service at 7AM Saturday and lines were moved back then (Earlier than originally planned)

MAINTENANCE Completed Office
AFFECTING
Office
STARTED
Nov 28, 02:00 PM (4 months ago)
CLOSED
Jan 06, 03:00 PM (2¾ months ago)
DESCRIPTION

Here are our office opening times over Christmas and new year:

  • Mon 23rd: Normal hours (8-6)
  • Tue 24th: 8-3
  • Wed 25th: Closed
  • Thurs 26th: Closed
  • Fri 27th: 9-5
  • Sat 28th: 10-2
  • Sun 29th: Closed
  • Mon 30th: 9-5
  • Tues 31st: 8-3
  • Wed 1st: Closed
  • Thurs 2nd: back to normal (8-6)

Resolution:

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Nov 19, 09:30 AM (4¼ months ago)
CLOSED
Jan 23, 07:00 PM (2 months ago)
DESCRIPTION

The work outlined below will start from Saturday 23rd November.

Background:

Our FireBrick team has been working on the 'hang' problem that we faced with the LNSs earlier in the year.

The nature of the problem has made investigating the problem very time consuming as it is extremely difficult to reproduce. However, we do believe that a plausible cause has been identified, and code changes have been made to mitigate the problem.

We have been testing this new code, both in our test lab and on a few select A&A routers, for over two months. During this time the new code has not caused the hardware to hang, where older versions of the code did.

Our next step is to run the new code on our LNSs, the ones our customers connect to for their broadband connections.

We plan to do this slowly, out of hours and in a couple of phases.

We believe the cause of the hang is related to how memory is initially allocated for the tasks the FireBrick will be performing, this means that if the hardware is going to hang then this will most likely happen over the first couple of days (or first couple of hours).

Stage one: (Completed)

We plan to upgrade only one of our LNSs at first. We will move broadband connections on to it in the early hours of the morning and then move them back off a few hours later. This means that during the day, customers will be on the normal set of LNSs.

Then, each night, over the course of two weeks, the LNS will be power cycled and we will move an increasing number of connections over, until it is at the point of taking twice the amount of connections that we'd normally run on an LNS. (We normally run LNSs at around 40% capacity, so twice the number of connections is not a problem.)

Stage two: (Completed)

Once we have confirmed that the hang is not happening, the second phase would be to run customer connections on the upgraded for a few days at a time.

We will go through a cycle of: move connections off, reboot the LNS, move connections on, wait a few days. Repeat. We will do this with an increasing number of connections until it's at the point of taking a normal amount of connections.

Stage Three:

As of the end of 2024, half our LNSs have been running the new software without any problems. From January 7th we will e doing overnight upgrades of the remaining LNSs.

More information:

So as to minimise impact to customers, the work of moving connections off and on will happen overnight between 1AM and 5AM.


Resolution: All of our FireBrick FB900 LNSs and BGP routers have been upgraded successfully.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Oct 23, 03:00 AM (5 months ago)
CLOSED
Oct 23, 03:51 AM (5 months ago)
DESCRIPTION
Due to the problems on Tuesday afternoon we have customers unevenly spread across our pool of LNSs. We will move some customers in the early hours of Wednesday morning. These customers will see a drop and reconnect of their service.
Resolution: This work has been completed.

MAINTENANCE Completed Control Pages
AFFECTING
Control Pages
STARTED
Oct 16, 02:00 PM (5¼ months ago)
CLOSED
Oct 16, 04:18 PM (5¼ months ago)
DESCRIPTION
We're performing a database migration which will mean the display of broadband usage stats, VoIP Call records (CDRs) and SMS logs will be unavailable on one control pages on the afternoon of 16 October.
Resolution: This work has been reverted.

MAINTENANCE Completed Routing
AFFECTING
Routing
STARTED
Oct 07, 05:00 PM (5½ months ago)
CLOSED
Oct 15, 02:00 PM (5¼ months ago)
DESCRIPTION
We will be enabling a new transit provider this week - Arelion, AS1299. This is in addition to our three existing transit providers: NTT, Colt (aka Lumen/Level3) and Cogent, as well as multiple peering links. This gives us increased capacity but more importantly it gives us new, diverse routes to the rest of the internet which makes us even better connected to the wider world.
Resolution:

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Oct 07, 01:00 AM (5¾ months ago)
CLOSED
Oct 07, 10:00 AM (5½ months ago)
DESCRIPTION
Due to various BT works over the past few weeks we have a small number of customers on our 'spare' LNS. We will be moving these customers off during the early hours of Monday 7th October. Customers on the i.Gormless LNS will have their line drop and reconnect.
Resolution: This work has been competed.