Last 10 minor outages posts, sorted by last updated
Order posts by limited to posts

MINOR Closed AA Services
AFFECTED
AA Services
STARTED
Apr 23, 05:30 PM (6¼ hours ago)
CLOSED
Apr 23, 08:49 PM (3 hours ago)
DESCRIPTION
There is a routing problem affecting access to some of our services, eg our website and L2TP service among others. We're investigating.
Resolution: This was caused by a third party internet provider, with whom we have been in talks with about them providing us some transit and had provisionally configured some of their routers to allow us to announce our IP blocks through them. We had not got to the point of actually setting up the service though. However, one of their routers malfunctioned and got in a state where it was re-announcing our IP blocks to some of the internet which meant some of the internet was sending traffic bound for us to them. We mitigated some of the problems by announcing more specific routes and also got in touch with the provider who promptly fixed their router.

Closed: 3 hours ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Apr 22, 12:14 PM (1¼ days ago)
CLOSED
Apr 22, 12:30 PM (1¼ days ago)
DESCRIPTION
Some customers are having problems with registering their VoIP phone. Investigations are underway. This will cause problems for some customers with receiving and making calls.
Resolution: There was a problem with us storing the port for some SIP registrations between 11:14 and 12:30 which was causing some registrations to fail.

Closed: 1¼ days ago
View full details

MINOR Closed Servers
AFFECTED
Servers
STARTED
Apr 15, 01:36 PM (8¼ days ago)
CLOSED
Apr 15, 03:07 PM (8¼ days ago)
DESCRIPTION
At around 13:30 one of our servers had a disk problem, and it needs to be rebooted and fixed. This server is hypervisor that runs some of our core services. As we run many redundant and spare servers which fail over to other servers when a problem occurs the customer impact is minimal.
Resolution:

Closed: 8¼ days ago
View full details

MINOR Closed DATA SIMs
AFFECTED
DATA SIMs
STARTED
Apr 11, 12:15 PM (12¼ days ago)
CLOSED
Apr 11, 01:30 PM (12¼ days ago)
DESCRIPTION
We've seen some Data SIMs drop and reconnect from 12:15 today - we suspect caused by something upstream, probably in the mobile network.
Resolution:

Closed: 12¼ days ago
View full details

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Apr 09, 02:00 PM (14¼ days ago)
CLOSED
Apr 09, 02:07 PM (14¼ days ago)
DESCRIPTION
At 2pm L2TP customers experienced a drop and reconnect of their service.
Resolution: Hardware replacement underway: https://aastatus.net/42656

Closed: 14¼ days ago
View full details

MINOR Closed SMS
AFFECTED
SMS
STARTED
Apr 08, 09:15 AM (15½ days ago)
CLOSED
Apr 08, 11:21 AM (15½ days ago)
DESCRIPTION
SMS delivery via HTTP POST was broken via one of our SMS relays for a while this morning. The symptom was that "da", the destination address, was being posted as the "target" rather than the destination number. This means if we post to your server on https://example.com/sms/, we could have posted the SMS with the destination number of literally "https://example.com/sms/". This would have broken anything depending on the "da" to make decisions on what to do with the message. This is fixed now, and the problem occurred between around 9:15 and 11:21. Apologies for any inconvenience.
Resolution:

Closed: 15½ days ago
View full details

MINOR Closed DNS, Email and Web Hosting
AFFECTED
DNS, Email and Web Hosting
STARTED
Apr 03, 10:54 AM (20½ days ago)
CLOSED
Apr 04, 10:54 AM (19½ days ago)
DESCRIPTION
Our DoH/DoT resolvers ( https://support.aa.net.uk/DoH_and_DoT ) were intermittently failing DNS lookups. It seemed to start over the Easter weekend. Our DoT/DoH front ends are DNS aware proxies (dnsdist) to back ends running unbound. dnsdist uses TLS to speak DNS to the back ends. Some of the back ends had failed to reload their TLS certificates after renewal, so although the certificates were valid unbound was still serving old certs and they eventually expired. This resulted in broken back ends in the pool, which dnsdist kept trying to bring back into service. The intermittent nature of the failures meant that it wasn't obvious to users, as clients generally retry silently in the background. Of course our monitoring should have caught this! We've fixed the underlying problem which caused unbound not to pick up the renewed certificates, and we've improved monitoring to catch similar problems should they occur in future.
Resolution:

Closed: 19½ days ago
View full details

MINOR Closed Hetzner
AFFECTED
Hetzner
STARTED
Feb 13, 07:00 PM (2¼ months ago)
CLOSED
Mar 23, 04:03 PM (1 month ago)
DESCRIPTION
Hetzner is a German based server hosting provider. We have seen intermittent problems in routing traffic to them in recent days.
Resolution:

Closed: 1 month ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Mar 19, 03:20 PM (1 month ago)
CLOSED
Mar 19, 03:33 PM (1 month ago)
DESCRIPTION
We're investigating reports if some incoming calls not arriving/
Resolution: A change to the database on one of our backends caused phone registration information to fail which caused some incoming calls to fail. The change was reverted at 15:32 and calls are now working. This was a change that has been running in test systems successfully for weeks, but further investigations will be carried out.

Closed: 1 month ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Mar 15, 11:10 AM (1¼ months ago)
CLOSED
Mar 15, 11:23 AM (1¼ months ago)
DESCRIPTION
We're investigating reports of call problems with one of our voice servers.
Resolution:

Closed: 1¼ months ago
View full details