Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

...start date, newest first ...last updated first
    Maximise all events
MAINTENANCE Planned Status Page
AFFECTING
Status Page
STARTING
Sep 22, 06:01 PM (8 hours )
DESCRIPTION
On the evening of the 22nd of September our status pages and web IRC system will be shut down due to planned maintenance to power systems in the data centre they're hosted in. We don't yet know how long they will be down, but it may be for a few hours as the window is over night and ends 5am on the 23rd of September.
Last updated: 4¾ days ago
Update due: 23 hours
View full details

MAINTENANCE Completed General
AFFECTING
General
STARTED
Sep 17, 02:00 AM (5¼ days ago)
CLOSED
Sep 17, 05:19 AM (5 days ago)
DESCRIPTION
Due to the recent outages we are needing to investigate in more detail some of our network infrastructure. We are planning to do this from 2AM on Thursday 17th September, and plan to complete the work by 5AM. So as to fully understand and fix the cause of some of our recent problems we may cause some disruption to services. There may be line drops and short periods where there may be limited connectivity. We will attempt to keep this to a minimum, and will update this post throughout the period.
Resolution: This work has been completed. There are a couple of outstanding things we still need to work on, but on the whole this has been successful. The main thing being the reboot of a core switch which has fixed most of the problems we've had recently.

Closed: 5 days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (7¼ days ago)
CLOSED
Sep 15, 04:00 PM (6½ days ago)
DESCRIPTION
We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.
Resolution: Summary and further work

Here is a brief summary of this incident and what we are planning to do to improve matters.

  1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
  2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
  3. 01:52 about 30% of lines still had problems staying connected
  4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
  5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
  6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
  7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.


Closed: 6½ days ago
View full details

MAJOR Closed Broadband
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (15½ days ago)
CLOSED
Sep 06, 08:00 PM (15½ days ago)
DESCRIPTION
We're investigating line drops that started at around 4PM and is affecting various broadband lines.
Resolution: We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

Closed: 15½ days ago
View full details

MINOR Closed VoIP and SIMs
AFFECTED
VoIP and SIMs
STARTED
Aug 24, 04:00 PM (28½ days ago)
CLOSED
Aug 25, 09:46 AM (28 days ago)
DESCRIPTION
We're investigating reports of a small number of incoming calls failing. It looks like registration state isn't syncing correctly across our SIP platform.
Resolution:

Closed: 28 days ago
View full details