Outages
Maintenance
Our Network:
Major

0

Happening Now

1

Future Planned

0

Suppliers' Network:
Open Events:
STATUS
Open
AFFECTING
Broadband
STARTED
Dec 11, 09:00 PM (2 days ago)
REFERENCE
27109 / AA27109
INFORMATION
  • INITIAL
    2 days ago by Andrew

    We're seeing low levels (1-5%) of packet loss and low throughput (eg 16-60 variable on a 70M VDSL) in the evenings on some TalkTalk connected lines in the SO postcode area. We've passed the affected lines on to TalkTalk for them to investigate further.

  • UPDATE
    1½ days ago by Andrew

    TalkTalk are unable to see any congestion problems at the moment -we're investigating further.

  • UPDATE
    13¾ hours ago by Andrew

    Further investigation shows that although the examples of lines with packet loss in the evening are all the the SO postcode areas, they are all diversely connected through the TalkTalk network - different DSLAMs, Multiple switches and backhauls and also different BNGs across three exchanges.

    We'll investigate these lines as individual faults for the time being and will close this status event.

  • UPDATE
    8¾ hours ago by Andrew

    We've re-opened this as TalkTalk are now investigating a possible switch that is 'maxing out' at peak time.

  • NEXT UPDATE...

    Due 1¼ days ago (overdue)

STATUS
Open
AFFECTING
General
STARTED
Nov 22, 03:00 AM (21¾ days ago)
REFERENCE
23491 / AA23491
INFORMATION
  • INITIAL
    21¾ days ago by Adrian

    We are upgrading some routers over night again, and we then plan to do a rolling LNS upgrade. More information about LNS switchovers: https://support.aa.net.uk/LNS_Switches

  • UPDATE
    14¼ days ago by Adrian

    We'll be doing some more router upgrades as well over the weekend, and are continuing the rolling LNS updates over night.

  • UPDATE
    7½ days ago by Adrian

    We are starting a rolling update again tonight (Thursday), and this time it means one LNS will transfer PPP sessions just before midnight rather than in the middle of the night tonight.

  • UPDATE
    7¼ days ago by Adrian

    We do also expect to re-load the L2TP service endpoint early evening. As usual this should be a matter of a few seconds downtime only.

  • UPDATE
    6¾ days ago by Adrian

    A further blip is expected shortly (around 2:15am) on the "N" LNS, and L2TP services.

  • UPDATE
    6¾ days ago by Adrian

    It looks like all is well with the new rolling update now, so this will continue now nightly for the next two weeks right up until our change freeze. We'll update this status if we see any further issues, but please let us know right away if you see problems.

EXPECTED CLOSE
Dec 20, 12:00 AM ( In 5¾ days )
Recently Closed Events:
Broadband blip graph

The graph shows the last few hours of logins and logouts of ADSL, VDSL, SIMs and L2TP circuits.

The current time is on the left. Green is login, red is logout.

If there are spikes, then this shows a large number of logouts, which may indicate an outage or planned work happening.

You can click on a spike to search for incidents or maintenance that were open around that time.

About our status page

This is the status page of Andrews & Arnold Ltd.

Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.

Contacting us
Our support number is 033 33 400 999, or you can email support@aa.net.uk or text 01344 400 999 to raise a support ticket.

Spotted a Major Service Outage? (MSO)
A Major Service Outage disrupts the service of multiple customers simultaneously. If you believe that a problem affects multiple customers, and is not mentioned here already, text the number above. Begin the text with "MSO". This alerts multiple staff immediately, waking them if necessary. False alarms (i.e. raising MSO for a single line being down) may result in your number being prevented from raising MSO alerts in future. More info.

Regular Maintenance
Thursday evenings, from 10pm, are designated as a general maintenance window where we will perform non-service affecting updates.

Timeline view of events on our network and systems

Events from the AAISP network from the last six months through to two month's time on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

Congestion Reports

Every night we analyse the packet loss and latency from us to our carriers and also from us to each of our customers.

A&A to Backhaul Carriers

This table show un-errored seconds on our connections in to our carriers for the past 7 days. Our aim is for all of these to be at 100%.

Links to carriers

We have links to back-haul carriers BT and TT.

Our target is that we operate a uncongested links to the carriers. This means that if the general trend is that the link will be getting full we order more capacity from carriers. The capacity we order is expensive, takes time to change and has minimum terms. This means it can be difficult to perfectly manage the traffic. We allow some head room but not more than necessary. It is technically impossible to guarantee that the link is always uncongested (not without having a pipe the size of all internet links in the world added together) but by careful monitoring and allowing enough headroom we can aim for that target.

This table shows an analysis of how well we meet our target.

Unerrored seconds

The unerrored seconds report is the simplest statistic to understand. If we drop even a single packet in a second then this counts as an errored second. Given that we can be handling hundreds of thousands of packets every second, this can be a very sensitive measure of congestion. Bear in mind we will drop the larger packets first which will normally be TCP which re-sends the dropped plackets. This helps ensure VoIP and interactive uses of the internet are unaffected.

Daytime unerrored seconds are based only on 9am to 6pm Monday to Friday

Non-premium

We also consider 100 second samples. If we have reduced the throughput of non-premium lines at all (as a result of dropped packets for several seconds in a row) then the whole 100 seconds is considered to have affected non-premium customers. Because we are looking at 100 seconds at a time it is possible for very short busts of traffic to cause this to show a worse figure than the unerrored seconds. Even if traffic was reduced by 0.5% for part of the 100 second period the whole 100 seconds counts as errored.

Premium

Considering the 100 seconds samples, there is a limit to how much we will reduce the non-premium customers. If that limit is reached then we consider the whole 100 second sample is having an impact on premium lines. This is a sign of much more severe bursts of congestion for several minutes. Bear in mind, a single 100 second sample in a day is more than 0.1% of the day, and even if this does happen VoIP and interactive services should be unaffected.

Read more.
Link congestion stats
Link Week days 9-6 24 hour Non-premium Premium
TalkTalk 100% 100% 100% 100%
BT 100% 100% 100% 100%
Backhaul Carrier to Customers

This report can pinpoint congestion in exchanges, BRAS's or particular areas of the country.

Method of analysis

Every line is sent an LCP echo every second and loss and latency measured. This is summarised per 100 second sample and archived for every line for each whole day. The previous day is then analysed each morning and the above table is updated.

Each line is considered to pass, fail or be inconclusive. Inconclusive includes lines where samples have too much upload or download which could themselves cause higher loss or latency to be observed. Lines with less than 18 hours on-line are excluded totally.

Latency is considered against a reference base-line latency for the circuit, and this has to be present for several hours to be considered. As such, lines without clean latency monitoring (some makes of router) are excluded.

For an area to show in the above table there must be no lines that pass at all, and at least 50% of the lines must fail (the rest being inconclusive). Where we have very few lines we require 80% to fail (and none to pass) to show on this list. The area must also have enough lines, so some smaller exchanges will not be detected.

The size stated is what percentage of our total lines are in the affected area.

The block graph under each report for yesterday is based on an average of valid samples for each line for each hour, and then taking the lowest value of all lines in the affected area, so the loss and latency shown is something that is affecting every line in an area.

TypeAreaSizeNote
Recently detected, but has been OK since Tue 11 Dec:
BT 21CN L-WIL, all BRASs 0.11% Packet loss approx 8pm to 10pm

We are not able to automatically identify all congestion spots so this does not mean there are not problem areas, do contact Support if you think you are suffering from congestion.