Outages
Maintenance
Our Network:
Major Outages

0

Minor Outages

1

Happening Now

0

Future Planned

1

Open Events:
POST CREATED
Sep 30, 01:49 PM (2¼ days ago)
STATUS
Open
AFFECTING
TalkTalk
STARTED
Sep 30, 01:30 PM (2¼ days ago)
REFERENCE
42468 / AA42468
INFORMATION
  • INITIAL
    Post created: 2¼ days ago by Andrew

    We do apologise for the inconvenience these drops are having on our customers.

    There have been a number of problems affecting TalkTalk ADSL and VDSL circuits this week. They have been a mix of overnight planned work, overnight planned work going wrong and causing disruption in the mornings and drops during the day.

    These are less of a problem where customers have bonded lines with services from different back-haul providers (TalkTalk and BT) but they are disruptive to single-line customers.

    Earlier in the week we took our Telehouse interconnect out of service as it was connected to equipment at TalkTalk that was having serious problems. However, we have still seen a few drops during the day. These drops are not affecting all circuits but also don't seem to be localised to particular areas of the country.

    TalkTalk say they have no more work to do on the Telehouse side and so over this weekend we will bring back the downed interconnect which will restore resilience. This should not cause a blip, but will be done out of hours.

    We are still in dialogue with TalkTalk about the unexplained drops and will report here the findings and outcome.

  • UPDATE
    22¼ hours ago by Andrew

    There were two further drops Saturday - 00:06, 19:56 and 22:26. TalkTalk circuits either saw a some packetloss (about 15 seconds) or their PPP drop and reconnect. We have had email and telephone calls with TalkTalk on Saturday evening and they are investigating but are not yet finding a network issue. From our side, we see 15 seconds or so packet loss to the TalkTalk side of the L2TP tunnels from all of our LNSs. We see no errors on our physical interconnect to TalkTalk and our BGP sessions to TalkTalk stay up. It does seem as though the loss is on the TalkTalk side but investigations are still happening. As mentioned before we have an interconnect to TalkTalk in a different datacentre (Telehouse) which is currently down as TalkTalk had problems on their equipment earlier in the week and have asked us not to bring it in to service. Usually for a problem like this we would move traffic off one interconnect to the other to see if the problem follows of stays; but we're unable to do that at this moment.

  • UPDATE
    8 hours ago by Andrew

    We've seen 6 drops on Sunday so far - We are currently awaiting TalkTalk to confirm if we're able to bring up our Telehouse interconnect as so far, the cause of the packetloss on our Equinix interconnect has not been discovered.

  • UPDATE
    2 hours ago by Andrew

    We've seen further drops on Sunday. TalkTalk have advised us not to re-enable our Telehouse interconnect as they say that will cause an increase in problems. They are also have a meeting booked with their core engineers on Monday morning to go over the issues.

  • NEXT UPDATE...

    Due in 12¾ hours

Broadband blip graph

The graph shows the last few hours of logins and logouts of ADSL, VDSL, SIMs and L2TP circuits.

The current time is on the left. Green is login, red is logout.

If there are spikes, then this shows a large number of logouts, which may indicate an outage or planned work happening.

You can click on a spike to search for incidents or maintenance that were open around that time.

About our status page

This is the status page of Andrews & Arnold Ltd.

Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.

The last update was Today 19:02:22

Contacting us
Our support number is 033 33 400 999, or you can email support@aa.net.uk or text 01344 400 999 to raise a support ticket.

Spotted a Major Service Outage? (MSO)
A Major Service Outage disrupts the service of multiple customers simultaneously. If you believe that a problem affects multiple customers, and is not mentioned here already, text the number above. Begin the text with "MSO". This alerts multiple staff immediately, waking them if necessary. False alarms (i.e. raising MSO for a single line being down) may result in your number being prevented from raising MSO alerts in future. More info.

Regular Maintenance
Thursday evenings, from 10pm, are designated as a general maintenance window where we will perform non-service affecting updates.

Congestion Reports

Every night we analyse the packet loss and latency from us to our carriers and also from us to each of our customers.

A&A to Backhaul Carriers

This table show un-errored seconds on our connections in to our carriers for the past 7 days. Our aim is for all of these to be at 100%.

Links to carriers

We have links to back-haul carriers BT and TT.

Our target is that we operate a uncongested links to the carriers. This means that if the general trend is that the link will be getting full we order more capacity from carriers. The capacity we order is expensive, takes time to change and has minimum terms. This means it can be difficult to perfectly manage the traffic. We allow some head room but not more than necessary. It is technically impossible to guarantee that the link is always uncongested (not without having a pipe the size of all internet links in the world added together) but by careful monitoring and allowing enough headroom we can aim for that target.

This table shows an analysis of how well we meet our target.

Unerrored seconds

The unerrored seconds report is the simplest statistic to understand. If we drop even a single packet in a second then this counts as an errored second. Given that we can be handling hundreds of thousands of packets every second, this can be a very sensitive measure of congestion. Bear in mind we will drop the larger packets first which will normally be TCP which re-sends the dropped plackets. This helps ensure VoIP and interactive uses of the internet are unaffected.

Daytime unerrored seconds are based only on 9am to 6pm Monday to Friday

Non-premium

We also consider 100 second samples. If we have reduced the throughput of non-premium lines at all (as a result of dropped packets for several seconds in a row) then the whole 100 seconds is considered to have affected non-premium customers. Because we are looking at 100 seconds at a time it is possible for very short busts of traffic to cause this to show a worse figure than the unerrored seconds. Even if traffic was reduced by 0.5% for part of the 100 second period the whole 100 seconds counts as errored.

Premium

Considering the 100 seconds samples, there is a limit to how much we will reduce the non-premium customers. If that limit is reached then we consider the whole 100 second sample is having an impact on premium lines. This is a sign of much more severe bursts of congestion for several minutes. Bear in mind, a single 100 second sample in a day is more than 0.1% of the day, and even if this does happen VoIP and interactive services should be unaffected.

Read more.
Updated 1 year ago. Link congestion stats
Link Week days 9-6 24 hour Non-premium Premium
Backhaul Carrier to Customers

This report can pinpoint congestion in exchanges, BRAS's or particular areas of the country.

Method of analysis

Every line is sent an LCP echo every second and loss and latency measured. This is summarised per 100 second sample and archived for every line for each whole day. The previous day is then analysed each morning and the above table is updated.

Each line is considered to pass, fail or be inconclusive. Inconclusive includes lines where samples have too much upload or download which could themselves cause higher loss or latency to be observed. Lines with less than 18 hours on-line are excluded totally.

Latency is considered against a reference base-line latency for the circuit, and this has to be present for several hours to be considered. As such, lines without clean latency monitoring (some makes of router) are excluded.

For an area to show in the above table there must be no lines that pass at all, and at least 50% of the lines must fail (the rest being inconclusive). Where we have very few lines we require 80% to fail (and none to pass) to show on this list. The area must also have enough lines, so some smaller exchanges will not be detected.

The size stated is what percentage of our total lines are in the affected area.

The block graph under each report for yesterday is based on an average of valid samples for each line for each hour, and then taking the lowest value of all lines in the affected area, so the loss and latency shown is something that is affecting every line in an area.

Updated 1 year ago.

There are no currently identified congestion spots

We are not able to automatically identify all congestion spots so this does not mean there are not problem areas, do contact Support if you think you are suffering from congestion.