We have been seeing an increase in broadband lines dropping and reconnecting. This started from 9PM Saturday. We're investigating the cause. Skip to the 'resolution' section below for fixes.
We are still investigating. It is worth clarifying that this is a relatively small number of lines. What is causing complications is that there is no obvious pattern as to why these lines are impacted.
A pattern is eventually emerging - thanks to other reports (thank you!) - it seems other ISPs are affected too - however, it seems related to Draytek routers having some sort of problem. We suggest customers upgrade the software on their Draytek routers.
This may be useful to customers using Draytek: https://www.draytek.com/about/security-advisory/ The problems we and other ISPs have been seeing could be related to the 'Buffer Overflow Vulnerabilities'.
Draytek also say:
1. Upgrade your firmware immediately to the version listed below for your device.
2. Before upgrading: Back up your current configuration (System Maintenance > Config Backup). Use the ".ALL" file for upgrading to preserve your settings. If upgrading from an older version, review the release notes for specific instructions.
3. If remote access is enabled: Disable it unless absolutely necessary. Use an access control list (ACL) and enable 2FA if possible. For unpatched routers, disable both remote access (admin) and SSL VPN. Note: ACL doesn't apply to SSL VPN (Port 443), so temporarily disable SSL VPN until upgraded.
Information if customers are still having problems after upgrading software: Some of our traffic captures this afternoon see the last packet being sent to a Draytek router before it goes offline/crashes as being a SYN to port 443 (https). Perhaps if latest software is still causing problems, check if the web interface is open to the world and disable it. If the ssl VPN is set for port 443, then try disabling that too. (This may not help, depending on the actual problem the Draytek routers have, but it's worth a try.)
It's a bit early to tell for sure... but it looks like the 'attack' against Draytek rotuers may have stopped at 19:11 Sunday evening... Time will tell. Customers should still upgrade the 'attack' could start up at any time... (Another ISP is reporting the same.)
....and from 19:34 it's started up again, and Draytek routers are being knocked offline again...
Due in 6 hours
Summary:
Further details as sent out from Draytek:
If you are experiencing this issue, please follow the steps below to troubleshoot:
Disconnect the WAN cable.
Log into the router’s Web UI and check the system uptime. If the uptime is lower than the last known reboot, this indicates the router recently restarted.
Disable Remote Management by going to [System Maintenance] > [Remote Management].
Disable SSL VPN Service by going to [VPN and Remote Access] > [Remote Access Control].
Reboot the router and reconnect the WAN cable.
Monitor the connection to see if the WAN remains stable.
Firmware check and update:
Verify your router’s firmware version. If it is outdated, update it to the latest version.
Before updating, note your current firmware version. If you do not have a copy of the current firmware, download it first.
Take a configuration backup to avoid losing your settings.
If your WAN connection is stable:
Even if your device is not disconnecting, it is good practice to ensure you are on the latest firmware.
If your router is already on recent firmware and the newest version is not marked as Critical, an update may not be urgent but is still recommended for optimal performance and security.
For further assistance, visit our support page or contact our Customer Support team.
Thank you for your patience and cooperation.
We'll be performing overnight software upgrades on our BGP routers. Starting Saturday 22nd March. Routing tables update before a restart so there should be minimal impact to traffic.
Due in 3¼ days
We'll be performing overnight software upgrades on our pool of LNSs, starting Sunday 16th March. One LNS will be done per night, and this involves moving lines between the LNSs and upgrading when lines have been moved off. Customers may see a PPP drop and reconnect in the early hours of the morning this week as this work is carried out.
This notification is for customers who run their own authoritative DNS servers and use our secondary-dns.co.uk as secondary (slave).
As part of our DNS infrastructure project we will start to initiate Zone Transfers from an additional set of IP addresses.
You will still send your NOTIFY to secondary-dns.co.uk but you will start to see AXFR requests from an additional set of IP addresses.
Therefore, please update your ACLs to allow the following addresses in addition to what you have at the moment:
Please update your ACLs by March 11th.
We will have an overlap of using the existing DNS servers and the new DNS servers. As part of our testing we will start to use the new IP addresses from February 17th
This is only relevant to customers who run their own authoritative DNS servers and use our secondary-dns.co.uk as an additional nameserver.
Overview: We run a "secondary" DNS service for customers where they run the master DNS server and we are secondary slaves. We have a project underway that involves migrating all our authoritative DNS services to a new platform. As part of this we are needing to disable some of the automation we do for adding and updating the customer's master IP address automatically.
The change: From June 17th, If you run your own master DNS server for your domain(s) and secondary-dns.co.uk is a slave, if you change the IP address of your master you will need to contact support@aa.net.uk to request us to update our side.We have more information about our Authoritative DNS project on our Support Site: https://support.aa.net.uk/New_Authoritive_DNS
This change has been made.
Due 9¼ months ago (overdue)
This will search through the various outages and maintenance reports we have received from our suppliers which may be affecting you.
You can also see a list of outages and maintenance work we receive from our suppliers that may affect your specific broadband service on the Control Pages.
The search box above should find posts affecting your service however, you can use a Keyword search.
This is the status page of Andrews & Arnold Ltd.
Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.
The last update was Yesterday 16:38:57