Last 10 posts, sorted by last updated
Order posts by limited to posts

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Jun 23, 10:56 AM (15¾ days ago)
CLOSED
Jun 23, 11:15 AM (15¾ days ago)
DESCRIPTION
We have noticed some broadband customers over our TalkTalk network lost service at 10:11 this morning, this has been raised with TT who are investigating. Update to follow.....
Resolution:

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Mar 22, 09:28 PM (3½ months ago)
CLOSED
Mar 28, 02:00 PM (3¼ months ago)
DESCRIPTION
We have been seeing an increase in broadband lines dropping and reconnecting. This started from 9PM Saturday. We're investigating the cause. Skip to the 'resolution' section below for fixes.
Resolution: Summary:
  • A problem with Draytek routers meant many around the country (or world?) have problems staying connected, this affects Drayteks on many ISPs, not just us. This seems to be triggered by some sort of 'attack' against Draytek routers which have vulnerable software.
  • We have sent SMS/Emails to many of the affected customers pointing them to this status post. We have also emailed Draytek for comment.
  • Possible fix #1: We expect a software upgrade to resolve the problem and strongly suggest customers with Draytek routers upgrade their software or try a different make of router.
  • Possible fix #2: If software upgrade does not help, then make sure remote access to the router is disabled, and any VPN service is disabled - especially the ssl vpn and ssh vpn. (In GUI go to: VPN and Remote Access > Remote access control)
  • Last resort fix #3: Replace the router with a different model (at least Temporarily). We have Technicolor routers in stock...
  • Firmware available from: https://fw.draytek.com.tw

Further details as sent out from Draytek:

If you are experiencing this issue, please follow the steps below to troubleshoot:

Disconnect the WAN cable.

Log into the router’s Web UI and check the system uptime. If the uptime is lower than the last known reboot, this indicates the router recently restarted.

Disable Remote Management by going to [System Maintenance] > [Remote Management].

Disable SSL VPN Service by going to [VPN and Remote Access] > [Remote Access Control].

Reboot the router and reconnect the WAN cable.

Monitor the connection to see if the WAN remains stable.

Firmware check and update:

Verify your router’s firmware version. If it is outdated, update it to the latest version.

Before updating, note your current firmware version. If you do not have a copy of the current firmware, download it first.

Take a configuration backup to avoid losing your settings.

If your WAN connection is stable:

Even if your device is not disconnecting, it is good practice to ensure you are on the latest firmware.

If your router is already on recent firmware and the newest version is not marked as Critical, an update may not be urgent but is still recommended for optimal performance and security.

For further assistance, visit our support page or contact our Customer Support team.

Thank you for your patience and cooperation.


MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Mar 15, 08:00 AM (3¾ months ago)
CLOSED
Mar 31, 09:22 AM (3¼ months ago)
DESCRIPTION
We'll be performing overnight software upgrades on our pool of LNSs, starting Sunday 16th March. One LNS will be done per night, and this involves moving lines between the LNSs and upgrading when lines have been moved off. Customers may see a PPP drop and reconnect in the early hours of the morning this week as this work is carried out.
Resolution: This work has been completed.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Feb 14, 12:00 PM (4¾ months ago)
CLOSED
Feb 18, 03:00 AM (4¾ months ago)
DESCRIPTION

Happening Monday and Tuesday early mornings:

Most of our LNSs have extra diagnostic and debugging hardware installed which has been used to help us track down the problems we were having with them in early 2024. We are now in a position where we can remove this. We have planned to do this work over two nights: the early hours of February 17th and 18th.

This work will involve us moving customers off the LNSs as we do the work. In practice this means customer connections will experience a drop and reconnect at around 1AM and 4AM on the 17th and 18th February.


Resolution: This work was completed at around 3AM pn Tuesday morning.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Nov 19, 09:30 AM (7½ months ago)
CLOSED
Jan 23, 07:00 PM (5½ months ago)
DESCRIPTION

The work outlined below will start from Saturday 23rd November.

Background:

Our FireBrick team has been working on the 'hang' problem that we faced with the LNSs earlier in the year.

The nature of the problem has made investigating the problem very time consuming as it is extremely difficult to reproduce. However, we do believe that a plausible cause has been identified, and code changes have been made to mitigate the problem.

We have been testing this new code, both in our test lab and on a few select A&A routers, for over two months. During this time the new code has not caused the hardware to hang, where older versions of the code did.

Our next step is to run the new code on our LNSs, the ones our customers connect to for their broadband connections.

We plan to do this slowly, out of hours and in a couple of phases.

We believe the cause of the hang is related to how memory is initially allocated for the tasks the FireBrick will be performing, this means that if the hardware is going to hang then this will most likely happen over the first couple of days (or first couple of hours).

Stage one: (Completed)

We plan to upgrade only one of our LNSs at first. We will move broadband connections on to it in the early hours of the morning and then move them back off a few hours later. This means that during the day, customers will be on the normal set of LNSs.

Then, each night, over the course of two weeks, the LNS will be power cycled and we will move an increasing number of connections over, until it is at the point of taking twice the amount of connections that we'd normally run on an LNS. (We normally run LNSs at around 40% capacity, so twice the number of connections is not a problem.)

Stage two: (Completed)

Once we have confirmed that the hang is not happening, the second phase would be to run customer connections on the upgraded for a few days at a time.

We will go through a cycle of: move connections off, reboot the LNS, move connections on, wait a few days. Repeat. We will do this with an increasing number of connections until it's at the point of taking a normal amount of connections.

Stage Three:

As of the end of 2024, half our LNSs have been running the new software without any problems. From January 7th we will e doing overnight upgrades of the remaining LNSs.

More information:

So as to minimise impact to customers, the work of moving connections off and on will happen overnight between 1AM and 5AM.


Resolution: All of our FireBrick FB900 LNSs and BGP routers have been upgraded successfully.

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 22, 02:46 PM (8½ months ago)
CLOSED
Oct 22, 03:20 PM (8½ months ago)
DESCRIPTION
We''re investigating disconnections of broadband lines.
Resolution: Three of our eight live LNSs lost power causing customers on those to drop their connection. Many customers reconnected fairly quickly, but due to the number of LNSs out of action normal fail-over was hampered and needed manual intervention to re-route traffic. Further investigations will take place.

NEWS Info Broadband
AFFECTING
Broadband
STARTED
Aug 22, 12:00 AM (10½ months ago)
DESCRIPTION
We are now on the ramp up process for one touch switching, allowing switching to/from a number of other providers using One Touch Switching. Currently:
  • Brighton Fibre
  • broadband for Surrey Hills Limited
  • BRSK Limited
  • BT
  • Cuckoo Limited
  • EE
  • G.Network Communications Limited
  • Hyperoptic Limited
  • Now
  • Plusnet
  • Sky
  • TalkTalk
  • Utility Warehouse Limited
  • Virgin Media (Virgin Media O2)
  • Vodafone UK
  • WightFibre
  • Zen Internet
  • YouFibre Limited
  • Zzoomm plc
  • More CPs added over next few days. Lots of work on daily calls to make this all work for 12th.

    NEWS Info Broadband
    AFFECTING
    Broadband
    STARTED
    Aug 09, 04:01 PM (11 months ago)
    DESCRIPTION

    This is a reminder than our "anti slamming" service will stop on 12th Sep 2024. Sorry.

    This is a result of OFCOM mandated One Touch Switching process, and consequential changes by BT.

    We do offer the option on broadband lines to tell us a new surname for your service, which will make a "match" using One Touch Switching fail if not correct. This may help residential customers, and OTS may also avoid some mistakes happening.

    However, migration of a service away from us will be possible without One Touch Switching, with no checks, no surname or account match, and no way for us to stop it, and at little or no notice, from 12th Sep 2024, for residential and business customers.

    If you suffer slamming we will try and work with you, and see if migrating back quickly is possible or the right thing for you. But there can be costs implications we have yet to work out depending on the specific service.

    More info on directors blog https://www.revk.uk/2024/08/broadband-slamming.html


    MAINTENANCE Completed Broadband
    AFFECTING
    Broadband
    STARTED
    Jun 14, 10:00 AM (1 year ago)
    CLOSED
    Jul 08, 10:45 AM (1 year ago)
    DESCRIPTION

    As you are probably aware A&A develops its own network hardware under the FireBrick brand. We have been, for some time now, using a mixture of the older "FB6000" and the newer "FB9000" models as BGP routers, L2TP routers and LNSs which terminate customer broadband connections; For the most part the FB9000 LNSs have been is use for customers with faster speed lines. We've been using FB9000s for our L2TP services and as BGP routers for over a year now.

    You might have seen that we did have some stability problems earlier in the roll out of the FB9000 for our LNS platform and therefore paused the deployment. For some months now, we have running firmware with well established stability and as such will be rolling out FB9000s in replacement of all older FB6000 LNS. This work will happen in small stages during June and July.

    This will be mildly customer impacting, causing a brief outage as each line moves across, but no worse than any common LNS firmware update like we have been doing consistently for years. We will do migrations in the early hours of the morning and according to your preferences in the control pages.

    We will post updates to this status post when parts of this wider project are scheduled. eg when we schedule moving a set of customers off a specific LNS on to a new one.


    Resolution: These migrations have completed. All broadband customers are now on FB9000 LNSs.

    MAJOR Closed Broadband
    AFFECTED
    Broadband
    STARTED
    Feb 27, 12:00 PM (1¼ years ago)
    CLOSED
    Feb 27, 12:31 PM (1¼ years ago)
    DESCRIPTION
    A number of lines dropped and reconnected at around noon. We're investigating.
    Resolution:

    The X.Witless LNS hung and restarted which caused customers to disconnect and reconnect.

    This incident is related to https://aastatus.net/42608 X.Witless had been running without incident for 104 days. However, it is not fitted with an NVMe drive and was running software that pre-dates our NVMe drive fixes. We suspect the hang was caused by these two factors.

    Further work on our LNSs is being planned and updates will be posted to the status page in due course.