Last 10 posts, sorted by last updated
Limit to posts

MAINTENANCE Completed Core Network
AFFECTING
Core Network
STARTED
Oct 31, 02:00 AM (22½ hours ago)
CLOSED
Oct 31, 04:50 AM (19½ hours ago)
DESCRIPTION
Following on from https://aastatus.net/37744 we need to bring a switch back in to service as it is currently off. This is planned for Saturday 31st October between 2AM and 5AM. We expect the actual work to be completed earlier than 5AM though. We expect some disruption to services whilst we do this.
Resolution:

Closed: 19½ hours ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 29, 09:00 AM (2½ days ago)
CLOSED
Oct 29, 10:00 AM (2½ days ago)
DESCRIPTION
We had overnight problems with a faulty switch in London. details here: https://aastatus.net/37744 Bringing the replacement switch back in to service is causing some disruption to services though.
Resolution:

Closed: 2½ days ago
View full details

MAINTENANCE Completed Core Network
AFFECTING
Core Network
STARTED
Oct 29, 03:00 AM (2¾ days ago)
CLOSED
Oct 29, 02:00 PM (2¼ days ago)
DESCRIPTION
Starting 3AM Thursday 29th October: Following on from our planned work in September (https://aastatus.net/37312) we are wanting to reboot some of our other core switches as we still have some oddities with them which is preventing is from bring back one of our BT links in to service. We will minimise the the disruption as much as possible but there may be short periods where broadband connections will drop and reconnect. We expect the work to be completed by 5:30AM and will update this post during the work.
Resolution: We have an understanding of the problems that occurred this morning and we have planned to bring the switch back in to service on Saturday 31st October between 2AM and 5AM. This work will be tracked on: https://aastatus.net/37985

Closed: 2¼ days ago
View full details

MAINTENANCE Completed VoiP - 07 numbers
AFFECTING
VoiP - 07 numbers
STARTED
Oct 23, 10:00 PM (8 days ago)
CLOSED
Oct 23, 03:27 PM (8¼ days ago)
DESCRIPTION

Our carrier for incoming calls to 07 numbers has scheduled maintenance on their platform that will affect incoming calls. This only affects our customers who have an 07 number from us.

Start time: 2020-10-23 22:00
End time: 2020-10-23 22:30

Affected services: Whilst every effort will be made to reroute call traffic before this maintenance, some in-progress inbound and outbound calls may be dropped and will not automatically re-establish.

Resolution:

Closed: 8¼ days ago
View full details

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Oct 20, 11:55 AM (11½ days ago)
CLOSED
Oct 20, 12:10 PM (11½ days ago)
DESCRIPTION
There was a network problem at around noon which lasted for a few minutes which would have caused some destinations on the wider internet to be unreachable. This was resolved at the time
Resolution:

Closed: 11½ days ago
View full details

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Oct 09, 12:00 PM (22½ days ago)
CLOSED
Oct 13, 05:00 PM (18¼ days ago)
DESCRIPTION
A change on our VoIP platform as meant that we are sending the caller ID in international format, ie, +44... instead of national format (eg 0...) This may be causing some customers problems where they are expecting the CLI in a specific format. This was an unintended change and we're looking to reverse it and put the format back to what it was.
Resolution:

Closed: 18¼ days ago
View full details

MINOR Closed SMS Calls
AFFECTED
SMS Calls
STARTED
Oct 12, 12:00 PM (19½ days ago)
CLOSED
Oct 12, 01:10 PM (19½ days ago)
DESCRIPTION
Incoming SMS and phone calls to our 07 numbers are having problems. We're talking to the carrier about this at the moment. Outgoing calls and SMS are unaffected. Incoming calls and SMS to non-07 numbers are also unaffected.
Resolution:

Closed: 19½ days ago
View full details

MAINTENANCE Completed Status Page
AFFECTING
Status Page
STARTED
Sep 22, 06:01 PM (1¼ months ago)
CLOSED
Sep 23, 10:42 AM (1¼ months ago)
DESCRIPTION
On the evening of the 22nd of September our status pages and web IRC system will be shut down due to planned maintenance to power systems in the data centre they're hosted in. We don't yet know how long they will be down, but it may be for a few hours as the window is over night and ends 5am on the 23rd of September.
Resolution: This was has been completed.

Closed: 1¼ months ago
View full details

MAINTENANCE Completed General
AFFECTING
General
STARTED
Sep 17, 02:00 AM (1½ months ago)
CLOSED
Sep 17, 05:19 AM (1½ months ago)
DESCRIPTION
Due to the recent outages we are needing to investigate in more detail some of our network infrastructure. We are planning to do this from 2AM on Thursday 17th September, and plan to complete the work by 5AM. So as to fully understand and fix the cause of some of our recent problems we may cause some disruption to services. There may be line drops and short periods where there may be limited connectivity. We will attempt to keep this to a minimum, and will update this post throughout the period.
Resolution: This work has been completed. There are a couple of outstanding things we still need to work on, but on the whole this has been successful. The main thing being the reboot of a core switch which has fixed most of the problems we've had recently.

Closed: 1½ months ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (1½ months ago)
CLOSED
Sep 15, 04:00 PM (1½ months ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 1½ months ago
View full details