Last 10 minor outages posts, sorted by last updated
Order posts by limited to posts

MINOR Closed Servers
AFFECTED
Servers
STARTED
Apr 15, 01:36 PM (2½ days ago)
CLOSED
Apr 15, 03:07 PM (2½ days ago)
DESCRIPTION
At around 13:30 one of our servers had a disk problem, and it needs to be rebooted and fixed. This server is hypervisor that runs some of our core services. As we run many redundant and spare servers which fail over to other servers when a problem occurs the customer impact is minimal.
Resolution:

Closed: 2½ days ago
View full details

MINOR Closed DATA SIMs
AFFECTED
DATA SIMs
STARTED
Apr 11, 12:15 PM (6½ days ago)
CLOSED
Apr 11, 01:30 PM (6½ days ago)
DESCRIPTION
We've seen some Data SIMs drop and reconnect from 12:15 today - we suspect caused by something upstream, probably in the mobile network.
Resolution:

Closed: 6½ days ago
View full details

MINOR Closed L2TP
AFFECTED
L2TP
STARTED
Apr 09, 02:00 PM (8½ days ago)
CLOSED
Apr 09, 02:07 PM (8½ days ago)
DESCRIPTION
At 2pm L2TP customers experienced a drop and reconnect of their service.
Resolution: Hardware replacement underway: https://aastatus.net/42656

Closed: 8½ days ago
View full details

MINOR Closed SMS
AFFECTED
SMS
STARTED
Apr 08, 09:15 AM (9¾ days ago)
CLOSED
Apr 08, 11:21 AM (9½ days ago)
DESCRIPTION
SMS delivery via HTTP POST was broken via one of our SMS relays for a while this morning. The symptom was that "da", the destination address, was being posted as the "target" rather than the destination number. This means if we post to your server on https://example.com/sms/, we could have posted the SMS with the destination number of literally "https://example.com/sms/". This would have broken anything depending on the "da" to make decisions on what to do with the message. This is fixed now, and the problem occurred between around 9:15 and 11:21. Apologies for any inconvenience.
Resolution:

Closed: 9½ days ago
View full details

MINOR Closed DNS, Email and Web Hosting
AFFECTED
DNS, Email and Web Hosting
STARTED
Apr 03, 10:54 AM (14½ days ago)
CLOSED
Apr 04, 10:54 AM (13½ days ago)
DESCRIPTION
Our DoH/DoT resolvers ( https://support.aa.net.uk/DoH_and_DoT ) were intermittently failing DNS lookups. It seemed to start over the Easter weekend. Our DoT/DoH front ends are DNS aware proxies (dnsdist) to back ends running unbound. dnsdist uses TLS to speak DNS to the back ends. Some of the back ends had failed to reload their TLS certificates after renewal, so although the certificates were valid unbound was still serving old certs and they eventually expired. This resulted in broken back ends in the pool, which dnsdist kept trying to bring back into service. The intermittent nature of the failures meant that it wasn't obvious to users, as clients generally retry silently in the background. Of course our monitoring should have caught this! We've fixed the underlying problem which caused unbound not to pick up the renewed certificates, and we've improved monitoring to catch similar problems should they occur in future.
Resolution:

Closed: 13½ days ago
View full details

MINOR Closed Hetzner
AFFECTED
Hetzner
STARTED
Feb 13, 07:00 PM (2 months ago)
CLOSED
Mar 23, 04:03 PM (25¼ days ago)
DESCRIPTION
Hetzner is a German based server hosting provider. We have seen intermittent problems in routing traffic to them in recent days.
Resolution:

Closed: 25¼ days ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Mar 19, 03:20 PM (29¼ days ago)
CLOSED
Mar 19, 03:33 PM (29¼ days ago)
DESCRIPTION
We're investigating reports if some incoming calls not arriving/
Resolution: A change to the database on one of our backends caused phone registration information to fail which caused some incoming calls to fail. The change was reverted at 15:32 and calls are now working. This was a change that has been running in test systems successfully for weeks, but further investigations will be carried out.

Closed: 29¼ days ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Mar 15, 11:10 AM (1 month ago)
CLOSED
Mar 15, 11:23 AM (1 month ago)
DESCRIPTION
We're investigating reports of call problems with one of our voice servers.
Resolution:

Closed: 1 month ago
View full details

MINOR Closed LNS
AFFECTED
LNS
STARTED
Mar 15, 07:00 AM (1 month ago)
CLOSED
Mar 15, 07:40 AM (1 month ago)
DESCRIPTION
A 7:30AM, the X.Witless restarted causing customers on it to drop and reconnect.
Resolution: This was related to https://aastatus.net/42608 This LNS is now out of service and will be analysed by our developers.

Closed: 1 month ago
View full details

MINOR Closed LNS
AFFECTED
LNS
STARTED
Mar 09, 11:35 AM (1¼ months ago)
CLOSED
Mar 09, 11:30 AM (1¼ months ago)
DESCRIPTION
Customers on the X.Witless LNS dropped and reconnected at 11:35 today.
Resolution: This is related to the ongoing LNS hangs we've been seeing: https://aastatus.net/42608. We do apologise to customers affected by this. This incident does help towards diagnosing and investigating the root cause.

Closed: 1¼ months ago
View full details