Last 10 posts, sorted by last updated
Limit to posts

MAINTENANCE Planned Status Page
AFFECTING
Status Page
STARTING
Sep 22, 06:01 PM (10 hours ) Starting soon
DESCRIPTION
On the evening of the 22nd of September our status pages and web IRC system will be shut down due to planned maintenance to power systems in the data centre they're hosted in. We don't yet know how long they will be down, but it may be for a few hours as the window is over night and ends 5am on the 23rd of September.
Last updated: 4¾ days ago
Update due: 1 day
View full details

MAINTENANCE Completed General
AFFECTING
General
STARTED
Sep 17, 02:00 AM (5 days ago)
CLOSED
Sep 17, 05:19 AM (5 days ago)
DESCRIPTION
Due to the recent outages we are needing to investigate in more detail some of our network infrastructure. We are planning to do this from 2AM on Thursday 17th September, and plan to complete the work by 5AM. So as to fully understand and fix the cause of some of our recent problems we may cause some disruption to services. There may be line drops and short periods where there may be limited connectivity. We will attempt to keep this to a minimum, and will update this post throughout the period.
Resolution: This work has been completed. There are a couple of outstanding things we still need to work on, but on the whole this has been successful. The main thing being the reboot of a core switch which has fixed most of the problems we've had recently.

Closed: 5 days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (7 days ago)
CLOSED
Sep 15, 04:00 PM (6½ days ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 6½ days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (15½ days ago)
CLOSED
Sep 06, 08:00 PM (15¼ days ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

Closed: 15¼ days ago
View full details

MAINTENANCE Completed Datacentre Power
AFFECTING
Datacentre Power
STARTED
Sep 06, 12:01 AM (16¼ days ago)
CLOSED
Aug 21, 10:00 PM (1 month ago)
DESCRIPTION

One of our London datacentres is planning maintenance on the power feeds that supply one of our racks. This work involves transferring our power feeds to new new electrical infrastructure and will mean we lose one of our power feeds at a time whilst they do that.

We have two power feeds and they will be moved over one at a time. There should be no customer impact as most of our equipment use both power feeds, but this is a period of "at risk".

The work will happen during Sunday 6th September.

Resolution: CANCELLED: Datacentre say: Due to the power outage encountered at LD8 on Tuesday 18th August, we migrated all remaining customer supplies on to the new infrastructure. As this task has now been completed, the CMR is no longer required and was therefore cancelled.

Closed: 1 month ago
View full details

MINOR Closed VoIP and SIMs
AFFECTED
VoIP and SIMs
STARTED
Aug 24, 04:00 PM (28½ days ago)
CLOSED
Aug 25, 09:46 AM (27¾ days ago)
DESCRIPTION
We're investigating reports of a small number of incoming calls failing. It looks like registration state isn't syncing correctly across our SIP platform.
Resolution:

Closed: 27¾ days ago
View full details

MAJOR Closed Power Outage
AFFECTED
Power Outage
STARTED
Aug 18, 04:30 AM (1 month ago)
CLOSED
Aug 18, 09:40 PM (1 month ago)
DESCRIPTION
The London LD8 datacentre suffered a loss of power at 4:30AM. Our status page was affected and this is a new post which will have some updates as services are being restored.
Resolution: Brief summary:
  • From 04:34 there was total loss of power at Equinix's LD8 datacentre in London. This affected our equipment until 19:34
  • TalkTalk Ethernet services were off line for the whole duration
  • Our status page, although physically in Amsterdam was unreachable for the whole duration, and our failover status page gave a brief message about the incident
  • broadband lines that were initially affected lines reconnected at around 7AM as we moved service to our Telehouse datacentre
  • A small number of broadband lines suffered packet loss due to one of our 19 LNSs being overloaded
  • The loss of power meant we lost about half of our links to the wider internet - however these are replicated at our Telehouse datacentre for resilience and connectivity was mostly unaffected but one of our 3 remaining transit links had a small amount of packet loss, which would have affected some services
  • Many of our other services (email, web hosting, DNS etc) are hosted in both Telehouse and Equinix's data centres for resilience and were mostly unaffected.
  • Once power was restored work was done to move lines back to their original LNS - this involved forcing some lines to reconnect
  • From 10:30PM, we consider our network stable.
Although many customers were unaffected by this incident others were affected in various way. We do apologise for this. We'll be posting more information about the actual cause of the power loss when we receive it.

Closed: 1 month ago
View full details

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Aug 18, 08:22 PM (1 month ago)
CLOSED
Aug 18, 10:30 PM (1 month ago)
DESCRIPTION

Due to the power outage at Harbour Exchange https://aastatus.net/36892; many broadband circuits were moved to other LNSs.

Now that service has been restored, we need to re-balance sessions over all available LNSs to reduce possible congestion issues.

Over the next 2 hours we will be clearing sessions that are currently on the wrong LNS.

Downtime should be minimal. We apologise for the inconvenience.

Resolution:

Closed: 1 month ago
View full details

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 17, 09:45 PM (1 month ago)
CLOSED
Aug 17, 11:05 PM (1 month ago)
DESCRIPTION
We've had a few lines drop their PPP connection this evening, and our blip graph is showing spikey activity from around 9:45PM, we're investigating.
Resolution: about 5% of broadband lines were affected by loss of service due to a suspected hardware failure (yet to be confirmed) of one of our LNS routers - this is the router that customer broadband lines connect to at our side. This device has been taken out of service and lines reconnected. Work will be scheduled to replace the faulty hardware with spares.

Closed: 1 month ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Aug 12, 04:30 AM (1¼ months ago)
CLOSED
Aug 12, 05:15 PM (1¼ months ago)
DESCRIPTION
We have a number of DSL lines down in and around Scotland. BT say: "Due to the severe weather in Scotland early this morning there was a major outage at Edinburgh Exchange causing loss of 20CN & 21CN / ETHERNET, MEAS & BROADBAND SERVICES. Engineers are currently on site recovering services. We're sorry for any inconvenience that this has caused."
Resolution:

Closed: 1¼ months ago
View full details