We're seeing low levels (1-5%) of packet loss and low throughput (eg 16-60 variable on a 70M VDSL) in the evenings on some TalkTalk connected lines in the SO postcode area. We've passed the affected lines on to TalkTalk for them to investigate further.
TalkTalk are unable to see any congestion problems at the moment -we're investigating further.
Further investigation shows that although the examples of lines with packet loss in the evening are all the the SO postcode areas, they are all diversely connected through the TalkTalk network - different DSLAMs, Multiple switches and backhauls and also different BNGs across three exchanges.
We'll investigate these lines as individual faults for the time being and will close this status event.
We've re-opened this as TalkTalk are now investigating a possible switch that is 'maxing out' at peak time.
Due 4½ days ago (overdue)
We are upgrading some routers over night again, and we then plan to do a rolling LNS upgrade. More information about LNS switchovers: https://support.aa.net.uk/LNS_Switches
We'll be doing some more router upgrades as well over the weekend, and are continuing the rolling LNS updates over night.
We are starting a rolling update again tonight (Thursday), and this time it means one LNS will transfer PPP sessions just before midnight rather than in the middle of the night tonight.
We do also expect to re-load the L2TP service endpoint early evening. As usual this should be a matter of a few seconds downtime only.
A further blip is expected shortly (around 2:15am) on the "N" LNS, and L2TP services.
It looks like all is well with the new rolling update now, so this will continue now nightly for the next two weeks right up until our change freeze. We'll update this status if we see any further issues, but please let us know right away if you see problems.
The graph shows the last few hours of logins and logouts of ADSL, VDSL, SIMs and L2TP circuits.
The current time is on the left. Green is login, red is logout.
If there are spikes, then this shows a large number of logouts, which may indicate an outage or planned work happening.
You can click on a spike to search for incidents or maintenance that were open around that time.
This will search through the various outages and maintenance reports we have received from our suppliers which may be affecting you.
You can also see a list of outages and maintenance work we receive from our suppliers that may affect your specific broadband service on the Control Pages.
The search box above should find posts affecting your service however, you can use a Keyword search.
This is the status page of Andrews & Arnold Ltd.
Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.
Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.
Every night we analyse the packet loss and latency from us to our carriers and also from us to each of our customers.
This table show un-errored seconds on our connections in to our carriers for the past 7 days. Our aim is for all of these to be at 100%.
We have links to back-haul carriers BT and TT.
Our target is that we operate a uncongested links to the carriers. This means that if the general trend is that the link will be getting full we order more capacity from carriers. The capacity we order is expensive, takes time to change and has minimum terms. This means it can be difficult to perfectly manage the traffic. We allow some head room but not more than necessary. It is technically impossible to guarantee that the link is always uncongested (not without having a pipe the size of all internet links in the world added together) but by careful monitoring and allowing enough headroom we can aim for that target.
This table shows an analysis of how well we meet our target.
The unerrored seconds report is the simplest statistic to understand. If we drop even a single packet in a second then this counts as an errored second. Given that we can be handling hundreds of thousands of packets every second, this can be a very sensitive measure of congestion. Bear in mind we will drop the larger packets first which will normally be TCP which re-sends the dropped plackets. This helps ensure VoIP and interactive uses of the internet are unaffected.
Daytime unerrored seconds are based only on 9am to 6pm Monday to Friday
We also consider 100 second samples. If we have reduced the throughput of non-premium lines at all (as a result of dropped packets for several seconds in a row) then the whole 100 seconds is considered to have affected non-premium customers. Because we are looking at 100 seconds at a time it is possible for very short busts of traffic to cause this to show a worse figure than the unerrored seconds. Even if traffic was reduced by 0.5% for part of the 100 second period the whole 100 seconds counts as errored.
Considering the 100 seconds samples, there is a limit to how much we will reduce the non-premium customers. If that limit is reached then we consider the whole 100 second sample is having an impact on premium lines. This is a sign of much more severe bursts of congestion for several minutes. Bear in mind, a single 100 second sample in a day is more than 0.1% of the day, and even if this does happen VoIP and interactive services should be unaffected.Read more.
|Link||Week days 9-6||24 hour||Non-premium||Premium|
This report can pinpoint congestion in exchanges, BRAS's or particular areas of the country.
Every line is sent an LCP echo every second and loss and latency measured. This is summarised per 100 second sample and archived for every line for each whole day. The previous day is then analysed each morning and the above table is updated.
Each line is considered to pass, fail or be inconclusive. Inconclusive includes lines where samples have too much upload or download which could themselves cause higher loss or latency to be observed. Lines with less than 18 hours on-line are excluded totally.
Latency is considered against a reference base-line latency for the circuit, and this has to be present for several hours to be considered. As such, lines without clean latency monitoring (some makes of router) are excluded.
For an area to show in the above table there must be no lines that pass at all, and at least 50% of the lines must fail (the rest being inconclusive). Where we have very few lines we require 80% to fail (and none to pass) to show on this list. The area must also have enough lines, so some smaller exchanges will not be detected.
The size stated is what percentage of our total lines are in the affected area.
The block graph under each report for yesterday is based on an average of valid samples for each line for each hour, and then taking the lowest value of all lines in the affected area, so the loss and latency shown is something that is affecting every line in an area.
|Recently detected, but has been OK since Tue 11 Dec:|
|BT 21CN||L-WIL, all BRASs||0.11%||Packet loss approx 8pm to 10pm|
We are not able to automatically identify all congestion spots so this does not mean there are not problem areas, do contact Support if you think you are suffering from congestion.