Order posts by limited to posts

13 Oct 16:48:23
13 Oct 15:52:41
We had a network incident starting from just after 15:30 today lasting about 15 minutes. This looks like a DDOS attack but we are still investigating.
13 Oct 15:53:35
Normal service resumed at around 15:41
Started 13 Oct 15:30:00
Closed 13 Oct 16:48:23

22 Sep 11:35:00
21 Sep 13:26:39
There is some sort of routing problem affecting customers. We are still investigating
21 Sep 13:35:40
Routing/traffic is back to normal, we continue to monitor...
22 Sep 10:24:49
The burst of traffic that caused this disruption has not returned. We do apologise for the inconvenience this caused.
Started 21 Sep 13:20:00
Closed 22 Sep 11:35:00

1 Jul 04:34:32
1 Jul 03:11:16
From 2:18 today, we are seeing packet loss and some lines dropping their connection.
Resolution We've identified the caused of this, and traffic is back to normal.
Started 1 Jul 02:18:00
Closed 1 Jul 04:34:32

18 Jun 15:00:00
[Broadband] - BT Blip - Closed
18 Jun 14:50:08
A number of BT lines dropped out at about 14:44, this appears to be an issue on the BT side, all lines have now recovered. We are chasing Bt for an explanation.
18 Jun 15:03:46
Some lines are still blipping, we are investigating further.
Resolution The blip seems to have gone away - we may move some lines that ended up on the wrong LNS back later. This is not the first time this link to BT had blipped and we are chasing with BT.
Started 18 Jun 14:47:59
Closed 18 Jun 15:00:00

1 Jun 23:30:51
1 Jun 23:11:35
We're seeing odd routing blips across our network. We're investigating.
1 Jun 23:31:30
Traffic is back to normal. We'll investigate this further and will update this post in the morning.
Resolution This looked like it was some sort of DOS that affected our routing, possibly other ISPs too.
Started 1 Jun 22:55:00
Closed 1 Jun 23:30:51

2 Jun 11:18:33
2 Jun 11:15:34
At 11:07 one of our links to BT blipped - this caused a number of BT DSL lines to drop. Lines are reconnecting at the moment.
2 Jun 11:18:56
Most lines are now back online.
Started 2 Jun 11:07:00
Closed 2 Jun 11:18:33

27 May 14:48:58
27 May 10:37:50
One of our routers is misbehaving - the one we planned to reload shortly as it happens, and this seems to be impacting mobile data SIMs.
27 May 10:46:58
This appears to be a LINX issue
27 May 10:53:20
We have managed to shut down the port to LINX while this is resolved.
27 May 11:01:41
This looks to be impacting lots of ISPs, but traffic will go via alternative peering and transit whilst LINX have issues and so this should have minimal impact on customers at present.
27 May 12:09:13
LINX are clearly having some major issues on one of the LANs, and we remain disconnected until this is resolved. Once it is resolved, we may reload one of our routers so as to add more logging and allow us to more carefully monitor further issues.
27 May 14:39:14
LINX have rectified the issue, so we will be re-enabling LINX ports shortly.
Resolution All working now.
Started 27 May 10:30:00
Closed 27 May 14:48:58

13 Apr 15:01:38
13 Apr 14:51:55
There was an issue with two of our routers - a few lines dropped, and are reconnecting. Routing was affected for a minute or two. We're investigating.
Resolution Service has recovered as expected. We'll see if we can find the underlying cause. Sorry for any inconvenience.
Started 13 Apr 14:46:46
Closed 13 Apr 15:01:38
Previously expected 13 Apr 14:50:00

13 Mar 22:57:53
13 Mar 22:44:00
TalkTalk lines lost connection at about 10:30, and are reconnecting at the moment.
13 Mar 22:47:49
About 80% of lines have now reconnected.
13 Mar 22:58:06
Most lines are back.
Resolution Confirmed as a fault within the TalkTalk network that affected us and other ISPs.
Started 13 Mar 22:30:00
Closed 13 Mar 22:57:53

10 Jan 15:35:00
10 Jan 10:13:09
We are investigating a problem affecting some wholesale customers. This seems to be a data centre connectivity fault. Updatea to follow.
10 Jan 10:23:40
This is related to a fault with Datahop,outside of our network. They are aware and investigating.
10 Jan 15:37:12
Our Datahop ports are now back online. (Ironically, they came on line whilst waiting on hold for their NOC!). Wholesales lines are starting to reconnect. We suspect a faulty switch is to blame within Datahop.
10 Jan 21:09:52
Datahop confirm this was caused by a faulty switch.
Started 10 Jan 09:20:00
Closed 10 Jan 15:35:00

03 Dec 2014 18:20:00
03 Dec 2014 10:45:55
We are seeing MTU issues on some 21CN lines this morning where lines are unable to pass more than 1462 byte IP packets. It isn't affecting all lines, and the common factor appears to be that they are on ACC-ALN1 BRASs in London, but not all lines on those BRASs are affected. We have reported the issue to BT Wholesale and they are investigating the issue.
03 Dec 2014 16:18:23
BT have now raised an incident and are investigating.
03 Dec 2014 18:57:49
This has been fixed. We've been speaking to some network guys at BT this evening and helping them find the fault.
Resolution BT fixed this last night. We believe BT had equipment in the network that was miss-configured.
Started 03 Dec 2014 09:00:00
Closed 03 Dec 2014 18:20:00

28 Sep 2014 19:20:54
28 Sep 2014 18:52:50
We are experiencing a network problem affecting our broadband customers. Staff are investigating.
28 Sep 2014 19:08:28
This is looking like some sort of Denial of Service attack. We're lookig at mitigating this.
28 Sep 2014 19:16:36
The traffic has died down, things are starting to look better.
28 Sep 2014 19:21:46
Traffic is now back to normal.
Started 28 Sep 2014 18:30:00
Closed 28 Sep 2014 19:20:54

19 Aug 2014 12:59:53
19 Aug 2014 00:36:05
Initial reports suggest one of our fibre links to TalkTalk is down. This is affecting broadband lines using TalkTalk backhaul.
19 Aug 2014 00:43:35
00:05 TT Lines drop, looked like we had a router blip and a TT fibre blip - reasons yet unknown
00:15 Lines start to log back in
However, we are getting reports in intermittent access to some sites on internet - possible MTU related.
19 Aug 2014 01:33:16
MTU is still a problem. A workaround for the moment, is to lower the MTU setting in your router to 1432. Ideally this should not be needed, but try this until the problem is resolved.
19 Aug 2014 01:58:30
Other wholesalers using TT are reporting the same problem. TT helpdesk is aware of planned work that may be causing this. We have requested that that pass this MTU report on to the team involved in the planned work.
19 Aug 2014 07:14:05
TT tell us they think the problem with MTU has been fixed. We're still unsure at this moment, and will work with customers who still have problems.
19 Aug 2014 07:55:02
This is still a problem affecting customers using TT backhaul. TT are aware and are investigating. This is a result of a router upgrade within TT which looks to have been given incorrect settings.
Where possible, customers can change the MTU on their routers to be 1432
19 Aug 2014 08:55:47
We have been in contact with the TT Service Director who will be chasing this up internally at TT.
19 Aug 2014 09:05:48
Customers with bonded lines using TT and BT can turn off their TT modem or router for the time being.
19 Aug 2014 09:20:11
We are looking at re-routing TT connections through our secondary connection to TT...
19 Aug 2014 09:30:55
Traffic is now routing via our secondary connection to TT, this looks like it is not being routed via the faulty TT router and it is looks as if lines are passing traffic as normal
19 Aug 2014 09:55:32
Some customers are working OK, some are not.
The reason being is that we have 2 interconnects to TT. We are still seeing connections from both of them, however, we have a 1600 byte path from one but only 1500 from the other. The 1500 one is the one that TT did an upgrade on last night. So it looks like TT forgot to configure jumbo frames on an interface after the upgrade.
Needless to say, we've passed this information on to people at various levels within TT
19 Aug 2014 09:57:02
We are working on only accepting connections from TT via the working interconnect.
19 Aug 2014 10:39:32
We are forcing TT lines to reconnect, this should mean they then reconnect over the working interconnect and not the one with the faulty TT router.
19 Aug 2014 11:21:53
We are blocking connections from the faulty TT router and only accepting from the working one. This means when customers connect they have a working connection. However, this does mean that logins are being rejected from customers until they are routed via the working interconnect. It may take a few attempts for customers to connect first time.
19 Aug 2014 11:24:09
Some lines are taking a long time to come back. This is because they are still coming in via the broken interconnect - that we're rejecting. Unfortunately, affected lines just have to be left until they attempt to log in via the working interconnect. So, if we appear to be rejecting your login please leave your router to keep trying and it should fix itself.
19 Aug 2014 11:32:11
TT are reverting their upgrade from last night. This looks like it's underway at the moment.
19 Aug 2014 11:35:00
Latest from TT: "The roll back has been completed and the associated equipment has been restarted. Our (TT) engineers are currently performing system checks and a retest before confirming resolution on this incident. Further information will be provided shortly. "
19 Aug 2014 11:43:32
TT have completed their downgrade. It looks like the faulty link is working OK again, we'll be testing this before we unblock the link our side.
19 Aug 2014 13:01:55
We've re-enabled the faulty link, we are now back to normality! We do apologise for this outage. We will be discussing this fault and future upgrades of these TT routers with TT staff.
Started 19 Aug 2014 00:05:00
Closed 19 Aug 2014 12:59:53

29 Jul 2014 01:17:44
28 Jul 2014 21:38:18
We are having reports this evening of some lines being unable to log in, but are in sync. We are investigating.
28 Jul 2014 22:00:52
We believe we have identified the problem and are working on a fix.
28 Jul 2014 22:17:51
Lines are logging in successfully now. If you are still off, please keep trying.
Resolution An issue with authentication on the "C" LNS, and then on the "D" LNS. We have found the issue, and lines are connecting to "D" cleanly now. The underlying issue causing this is being investigated.
Started 28 Jul 2014 21:37:18
Closed 29 Jul 2014 01:17:44
Cause BT

10 Jul 2014 20:10:00
10 Jul 2014 19:18:35
We are seeing a problem with BT 21CN ADSL and FTTC circuits being unable to log in since approximately 18:00 today. Existing sessions are working fine but are failing to reconnect when they drop. 20CN ADSL and TalkTalk backhaul circuits are working fine.

BT have raised incident IMT25152/14 which looks to be related, but just says they are investigating a problem.

10 Jul 2014 22:16:28
BT have reported that service should have been restored as of 20:10 this evening.

Customers who are still having problems should attempt to re-connect as they may be stuck on a BT holding session.

Anyone still having problems after doing that should contact tech support.

Started 10 Jul 2014 17:15:00
Closed 10 Jul 2014 20:10:00
Cause BT

19 Nov 2013 23:22:00
19 Nov 2013 21:51:40
It looks like some lines on the Wolverhampton BRAS are showing significant problems. BT have issued an incident report IMT45679/13 but there are no further details at this time.
Resolution Closure Details: Service was restored at 23:22 when redundancy switch over was performed to move the service to standby path. Service stability have been observed. BT regrets any inconvenience this may have caused.
Started 19 Nov 2013 20:30:00 by AAISP Staff
Closed 19 Nov 2013 23:22:00

21 Oct 2013 16:04:00
21 Oct 2013 15:57:52
TalkTalk Wholesale lines are currently off line. We are investigating.
21 Oct 2013 15:59:09
Some lines are coming back online..
21 Oct 2013 16:01:56
Other wholesalers are having similar problems
21 Oct 2013 16:04:56
Most lines are back online.
21 Oct 2013 16:20:42
We had notified TalkTalk at the start of this, they have replied saying that they have identified some network alarms. We'll update when we get more information. For now this fault is closed as lines are back online.
21 Oct 2013 16:30:43
TalkTalk confirmed that this affected many other of their wholesalers.
23 Oct 2013 12:01:23
We've not had the report from TT yet, but we have chased TT for it.
15 Nov 2013 16:50:54
This is the report from TalkTalk regarding this incident:
Problem Management have carried our further investigations. Please see the root cause below and preventative measures:
On Monday 21/10/2013, following some network device reboots, traffic was lost on two aggregate interconnects. We confirmed this using graphs on Cacti. Investigations were carried out by the B2B team and at 3:53pm on Monday 21st October a configuration addition was implemented on the ldn-vc1.thn device to try and load balance traffic over the two interconnects. Unfortunately this change affected the global routing of all traffic on the box (including management and customer data traffic). As part of the standard B2B procedure, the commit confirmed feature was used on the device when the change was made which resulted in the device returning back to normal service (management and data traffic) after 2 minutes. Following a review of this incident by B2B it was determined that the change was high risk. B2B processes have now been changed to always perform risk analysis on any future configuration changes. As well as this, any future configuration changes deemed necessary should be completed out of hours via the TTT PEW change process.
Sincere apologies for any inconvenience this has caused.
Resolution Summary of events:
Lines went off at 15:53
The started coming back online at 15:56
Most, if not all lines were back online by 16:04
We will have a report from TalkTalk as to the reason for this later today (22nd)
Started 21 Oct 2013 15:52:37
Closed 21 Oct 2013 16:04:00

14 Jul 2013 04:05:05
13 Jul 2013 15:00:49
All lines dropped - major connectivity issue affecting many ISPs
13 Jul 2013 15:05:16
Connections are re-establishing - we'll try and get an explanation out of BT.
13 Jul 2013 15:09:34
Seems clear this affected a lot of ISPs using BT backhaul, not just us. Customers with dual BT+TT or BT+BE services through us automatically fell back to just the non BT lines.
13 Jul 2013 15:14:35
It seems to be taking some time for lines to come back.
13 Jul 2013 15:24:45
We are seeing a second wave of connections. When BT have a major issue like this they "default accept" and give a dummy connection for a short period. As those lines drop and reconnect to us we get a second lot of connections. Even so we only have about half of our lines back yet which suggests that BT may be struggling at a RADIUS level.
13 Jul 2013 16:00:45
People have connected in several waves, and we suspect BT RADIUS servers were struggling. This looks pretty widespread. No more details yet. It looks like almost everyone is back on line though.
13 Jul 2013 16:41:55
Details from BT: " Retrospective report. There was a brief loss of service to 100,00 broadband customers. This was cause when the Boarder gateway protocols (BGP) dropped causing a loss of service to all WBMC customers in the Stepney Green area for 2 minutes. The root cause is under investigation by the operational team. Service was fully restored at 15:04 BT regrets any inconvenience this may have caused. "
13 Jul 2013 17:17:54
Since line dropped we are seeing issues with apparent latency spikes and packet loss.
13 Jul 2013 17:43:28
Lines have dropped again.
13 Jul 2013 17:50:59
We have spoken to BT, they are well aware of the drops today, and we have given them new information regarding the packetloss that we and other ISPs have been noticing since the initial drop at 3pm
13 Jul 2013 17:52:13
Since the drop at 17:53, many customers are getting logged in to a default BT service and getting BT IP addresses. We suggest customers wait a short while, and reboot their routers. This incident is still open with BT.
13 Jul 2013 18:29:13
Lines have dropped again.
13 Jul 2013 19:04:38
They are flapping still - all going offer, and then coming back over a period of time, rinse, repeat. Still waiting for BT to fix it.
13 Jul 2013 20:21:10
The last big blip was 18:27, but we are seeing smaller blips every 10 minutes or so. Up until 19:16 this was around 10% of lines, and since has been around 5% of lines. Clearly this is something BT are having to work on throughout there network somehow. The last small blip was 20:05 and it has been well over 10 minutes now, so looking hopeful that this may finally be sorted.
13 Jul 2013 20:21:54
Sorry, as I typed that, another small blip at 20:20
13 Jul 2013 20:50:38
Last blip was 20:26, so looking like things may finally be sorted.
14 Jul 2013 03:08:58
And all BT lines go again - chasing BT now
14 Jul 2013 03:18:32
BGP sessions still down, over 12 minutes now. BT think they are changing a card, but that it should not have caused a loss in service as they re-routed traffic via a different node while they do it. Hence they are now on conference calls.
14 Jul 2013 03:23:02
From BT: This is what happened yesterday: As promised here is the latest situation regarding the issues seen at Stephney. It had been identified that a ‘supervisor’ card is faulty and half the traffic from Stephney has been diverted to Faraday to lighten the strain. You will see a likelihood of a loss of resilience however hopefully traffic is flowing, albeit at a lowered rate, but customer should have some connection. There is currently a conference call regarding this issue on-going and there are multiple parts of the BT Group endeavouring to resolve this issue. There is a plan to change out the faulty card on an IMT with reference 30031/13 at or around 02:00 in the morning.
14 Jul 2013 03:30:38
So, basically, they found the dodgy card last night, and planned to change it. They diverted traffic, or so they thought, and then started the card change just after 3am. Then they noticed that all the traffic stopped. As usual we were the first to get in touch with BT but other ISPs are on to them as well now. They are continuing with the card change (understandable). I imagine there will be questions to answer as to why the diverting did not work, and once again how we have a "single point of failure" again. This has been upgraded to a major incident in BT now.
14 Jul 2013 03:42:35
BGP back, sessions coming up now.
14 Jul 2013 04:06:01
Looks like over 95% of lines back now - some on a default accept will try again and all should be normal shortly. Lets hope that is it sorted for good. We'll post if we get more details from BT - I expect a proper report in a few days.
Started 13 Jul 2013 14:57:30
Closed 14 Jul 2013 04:05:05

31 May 2013 20:20:00
31 May 2013 18:43:08

Looks like a lot of lines via BT dropped, and a few reconnected, but dropped again.

Looks like BT only, not specific to any one of our LNSs, started with most of Stepney based BRASs going off.

31 May 2013 18:43:59

What is odd is it seemed to be BRASs toppling one after another. Very strange.

31 May 2013 18:44:53

Looks to have stabalised with around half of BT lines being off.

31 May 2013 19:11:19

No update from BT yet.

31 May 2013 19:14:43

Some of our customers with access to other ISPs using BT backhaul have confirmed that this issue is not just A&A/BT lines but is more widespread.

31 May 2013 19:21:25

We still cannot get hold of BT, and still have no update from BT...

31 May 2013 19:32:26

Customers are getting a web page which suggests various possible reasons but does not list BT as a possible cause, not amused.

We also got "21CN Incident Report :21CN WBC : STEPNEY GREEN : LOSS OF SERVICE ( CANCELLED ) : Reference 23981 : Issue FINAL" which makes no sense as they seem to cancelled the "incident".

Still no reply trying to contact BT

31 May 2013 19:38:55

Finally. BT have confirmed they did get our report, sent within 60 seconds of the incident starting. And they are aware of a major issue. Trying to get more details.

So far "seems to be quite wide-spread, with multiple CPs, not yet got to root cause as we have also lost some management layers"

31 May 2013 20:05:33

It looks like around half of lines are down, but all of RADIUS is down, so anyone reconnecting now will not connect - if you are on-line and working then don't touch anything!

31 May 2013 20:10:59

Finally we get "INITIAL REPORT for IMT 23989/13 : Advanced notice of potential loss of Broadband services."

Yes, they say "advanced notice" for something taht started 90 minutes ago, and "potential" for something that has wiped out half our lines!

"Geographical area     : Nationwide - Geographical region unknown at present"

31 May 2013 20:53:38

We are seeing lines coming back.

31 May 2013 20:56:30

I'll leave the status post open for now and will update when we get explanation from BT. Thanks for your patience everyone.

31 May 2013 21:07:53

Please note, on an unrelated note, we are re-mapping LNSs. This was to be done over night but we changed the routing while lines were down to avoid a second blip over night. This means graphs are showing without any history earlier in the day for lots of lines that were off during the outage. The graphs will be re-instated over night as usual and you can then see the outage clearly.

11 Jun 2013 14:42:59

Report from BT regarding this outage:


Broadband Users Affected 50%
Closed 31 May 2013 20:20:00
Cause BT

31 May 2013 19:10:31
31 May 2013 18:41:03
Lines: 28% ALL and 33% BT and 39% 20CN and 31% 21CN and 35% FTTC and 100% 21CN-REGION-21CN-BRAS-RED10-LO0-PE and 80% 21CN-REGION-21CN-BRAS-RED11-MQD and 100% 21CN-REGION-21CN-BRAS-RED11-SL and 100% 21CN-REGION-21CN-BRAS-RED12-GI-B and 100% 21CN-REGION-21CN-BRAS-RED12-L-WAT dropped at 2013-05-31 18:39:47
We have advised BT
This is likely to have affected multiple internet providers using BT

Duplicate post

Broadband Users Affected 13%
Started 31 May 2013 18:39:47 by AAISP automated checking
Closed 31 May 2013 19:10:31
Cause BT

25 Mar 2013 04:41:50
25 Mar 2013 02:04:59

Since around 1am it looks liike custoemrs are having problems connecting. We have done an LNS upgrade which means around a third of customers are off line. This is being investigated now.

25 Mar 2013 02:12:12

This looks related to the upgrade and seems to be related to platform RADIUS, but we are still investigating now.

25 Mar 2013 02:21:37

Lines are coming back, but we are working to understand the issue as it appears to be RADIUS or BGP related.

25 Mar 2013 02:31:30

The fix to this makes no sense, and will be reviewed later in the morning. The issue appears to relate to platform RADIUS from BT. We changed the way the endpoints are announced to BT and it all sprung in to life, but that makes no sense. The LNS upgrade does not explain it either. There may be some planned work as a result of this to investigate further.

25 Mar 2013 02:50:21

The problem was not jus BT, but the fix we found was a result of changing BGP to BT which caused it to flip which RADIUS it was using. We have confirmed the issue is platform RADIUS and caused by LNS upgrades. The platform RADIUS is currently directed to a non-upgraded box and so BT, BE and data SIMs should all be able to connect now.

There will be s/w upgrades over the coming nights to correct this.

25 Mar 2013 02:56:04

It looks like some issues with BE lines still, which we are working on

25 Mar 2013 03:02:23

We have found the underlying issue, and will be re-jigging lines over night, sorry for any inconvenience.

This will me some loss of graphs.

25 Mar 2013 03:16:06

The error has been corrected, and lines are being moved between the two affected LNSs now. Graphs for around 1/3 of customers will be missing before around 3:30 this morning.

25 Mar 2013 03:34:20

Even with two working platform RADIUS servers, Be lines are still not connecting. We are not sure if their RADIUS have somehow blacklisted us because of lack of response, or what. Still working on that.

25 Mar 2013 03:47:15

We have asked fluidata (who manage the BE connection) to check their RADIUS servers for us. This may come back anyway over night if it is a timeout/blacklist issue on their RADIUS.

25 Mar 2013 04:42:55

We have just had a resonse from Fluidata and they seem to have given their end a kick and all is well now.

Graphs lost for some lines before around 03:30, and some other LNS updates over night during the week.

Started 25 Mar 2013 00:50:00
Closed 25 Mar 2013 04:41:50

02 Jan 2013 03:45:00
02 Jan 2013 03:44:08

Not sure what yet

02 Jan 2013 03:46:53

Routers and LNSs are all up and did not restart or anything, not clear what happened yet.

02 Jan 2013 03:49:42

The issue lasted a few minutes and seems to be over with services back to normal. So far we can see that a number of BGP sessions internal to the Telehouse rack restarted. It almost looks like a switch restart, but we are still checking.

02 Jan 2013 03:54:47

Checks confirm nothing rebooted, not even one of the network switches, so it is not clear what happened at this stage, we are still investigating, but the problem itself is over at present.

Started 02 Jan 2013 03:39:00
Closed 02 Jan 2013 03:45:00

13 Dec 2012 11:52:33
13 Dec 2012 09:07:30

We have just seen a lot of 20CN and 21CN lines around scotland drop.

We are reporting this to BT now. 

13 Dec 2012 09:15:24

Looks like the BT metro node in Scotland is down as it is affecting 20 and 21CN lines. 

13 Dec 2012 09:26:56

BT operate are aware and currently investingating. 
BT will update us when they have more news. 

13 Dec 2012 09:52:41

BT have raised a incident against Scotland, still awaiting an update as to what is happening.

13 Dec 2012 10:10:46

Bt say they have lost service to 2 10Gb circuits. Diagnostics still ongoing.

13 Dec 2012 10:31:53

BT engineers are on there way to site.

13 Dec 2012 10:35:05

BT have an engineer on site and a second engineer en route to assist.
Should have further news within the hour.

13 Dec 2012 11:04:46

BT are currently working to restoe service.

13 Dec 2012 11:09:54

Lines coming back up.

13 Dec 2012 11:22:32

From BT:

The affected router has had stopped process restarted and the controlled recovery of the device is being closely monitored. Service should be available within the next 30 minutes. The geographical area of impact is across Scotland, the North of England and Northern Ireland, on traffic that was terminating on this router. 

13 Dec 2012 11:53:29
Service fully restored at 11:15 after a process restart. If you are still down please reboot your router, if you are still unable to reconnect please contact support.
13 Dec 2012 18:40:00

Some more info about this outage on the ISP Review website: http://www.ispreview.co.uk/index.php/2012/12/bt-20cn-and-21cn-broadband-isp-lines-down-in-scotland-and-north-england.html

Started 13 Dec 2012 09:06:19 by AAISP Staff
Closed 13 Dec 2012 11:52:33
Cause BT

08 Dec 2012 08:49:00
08 Dec 2012 08:47:12

More issues, this time broadband lines.

To summarise - a routine upgrade this morning resulted in a problem, but only when the last box was upgraded. We had upgraded routers during the week, as we don't upgrade everything all at once. However the nature of the problem was such that only when the last box was upgraded did problems happen.

The effect was a few seconds with no routing, and then a period where all worked. This repeated for around 6 minutes until fixes were applied.

We can see the cause, but this effect was not foreseen, so we are looking in to how we can avoid such issues in futre.

Sorry for any inconvenience.

Note: This would not have causes broadband lines to drop sync or lose connection at all - this was entirely a routing issue to the wider internet.

08 Dec 2012 08:49:39

The issue is exactly the same as we had in Maidenhead so has been fixed quickly.

08 Dec 2012 09:33:26

There have been reports that some parts of the internet damped flapping routes for some minutes after we applied a fix. These seem to be relatively few and far bewteen as overall traffic levels returned to normal levels immediately. Obviously there is not a lot we can do over such policies in individual ISPs.

Started 08 Dec 2012 08:43:00
Closed 08 Dec 2012 08:49:00
Previously expected 08 Dec 2012 08:49:00

13 Aug 2012 17:30:28
13 Aug 2012 13:33:01

One of our core links to BT droped affecting a third of our customers.

Lines are recovering and re-logging in.

It may take a few minutes for all lines to reconnect.

More details to follow

13 Aug 2012 13:48:04

Most lines are back online. There are some which are not though and are 'flapping' (going up and down).

To restore service we are clearing lines off the LNS (core router) that the faulty BT link is on. This should then fix these remaining lines. - a side affect is that it will also mean that some BE lines will disconnect and reconnect. We apologise for this.

13 Aug 2012 14:02:24

We are still seeing some lines 'flapping' we're investigating the cause of this.

13 Aug 2012 14:11:41

More lines are now in this 'flapping' state, we are still working on this.

13 Aug 2012 14:13:39

We have restarted the 'A' LNS in an attempt to stabalise connections.

13 Aug 2012 14:15:36

Lines on the 'A' LNS are now looking stable. We're working on the other LNS's.

13 Aug 2012 14:17:08
We have restarted the 'C' LNS and lines are reconnecting.
13 Aug 2012 14:18:52
We have restarted the 'D' LNS and lines are reconnecting.
13 Aug 2012 14:20:11

Having restarted our LNs's lines are reconnecting and are remaing stable.

(Graphs for today up to ~2:15pm would have been lost though.)

13 Aug 2012 14:32:14

ADSL lines are looking stable now.

We are still in contact with BT about the link that is currently still down, and we will be reviewing how we can cope better with this type of outage in the future.

13 Aug 2012 14:33:13

It is not clear exactly why the loss of a BT link has caused things to be come unstable. Restarting all of the LNSs had resolved this, and we are investigating this. We suspect there is an issue with the routing to BT when one of the links is down, and so may be doing some planned work in due course to make changes that could improve matters.

In the mean time we are trying to get the failed link back up and working.

13 Aug 2012 14:35:35

BT have raised an incident and escalated it internally.
More update when we hear back from BT. 

13 Aug 2012 15:41:28

We have seen a couple of knock on effects with the flapping lines - the LNSs are very fast, and so handled lines flapping way faster than the RADIUS accounting database can keep up. As a result, things like colours of lines shown on clueless control pages, and text updates for lines flapping, are a tad behind. It is all catching up.

We also experienced a knock on effect with database updates causing some secondary servers to get busy. This is something we are fixing properly in the long term, but it meant some issues with telephony, which were a bit unexpected. That has now been resolved, and in the longer term these secondary servers should no longer struggle when there are a lot ofg database updated.

13 Aug 2012 16:33:29

Several people have reported some web pages being slow - it seems some of the sessios have come up with low MTUs and this is affecting a number of customers. We are clearing affected tunnels manually to rectify this so some people may see a PPP restart.

13 Aug 2012 16:50:15

We can see how the timing of the LNS restart we did could have resulted in the MTU settings being wrong on the first few tunnels, and have made config changes for the future. I have cleared the affected sessions and they have reconnected cleanly.

13 Aug 2012 16:54:28

A key thing here is ensuring that the niggles of today are permanently fixed - we are, of course, working on that. Most things are fixed for long term, and the one issue (the flapping) is still being investigated and we have an idea what to do for that as planned maintenance. Sorry for any inconvience.

13 Aug 2012 17:06:23

Looking at the overall usage graphs for transit links, we can see that even though some lines were flapping a lot, and most had some issues, overall people were still able to use the internet.

13 Aug 2012 17:25:43

We have normality, I repeat, we have normality. Anything you still can't cope with is therefore your own problem...

The accounting has all caught up at last, sorry for the delay, and the various delayed texts and emails.

Thank you all for your patience.

14 Aug 2012 13:45:51

BT have an engineer on site investigating the fault. Customers are conected via the other fibre links we have though.

14 Aug 2012 13:53:41

BT have fixed the broken fibre by replacing the service card.

This will be monitored.

15 Aug 2012 10:00:35

Our monitoring is reporting that the fibre is not stable, and has been dropping. This fibre is not in use by customers so it's not affecting any customers. This has been passed on to BT.

15 Aug 2012 16:05:27

BT have been testing this fibre today, they are not seeing a problem and the drops that we saw this morning have not happened since 9am. We'll continue to monitor it though.

16 Aug 2012 10:15:58

BT confirm that they are seeing alarms on equipment, the investigation continues.

18 Aug 2012 08:36:52

We are still working with BT to resolve this fibre issue. On of our staff was at the datacentre yesterday and BT have changed the service card yet again. However, we're still seeing the port flapping, and BT see alarms at their side. Further BT engineers are tasked to work on this again today.

In the meantime, we continue to use the other fibres that are part of this 'resilient' set so that service is unaffected.


BT replaced the service card on the NTE at our side of this circuit, and the fibre has remained stable.

Broadband Users Affected 33%
Started 13 Aug 2012 13:23:00
Closed 13 Aug 2012 17:30:28

18 Jul 2012 21:18:37
18 Jul 2012 21:13:41

We're seeing odd routing problems at the moment, this will affect internet access for DSL and Ethernet customers in London. Staff are investigating.

18 Jul 2012 21:18:37

This looks to have recovered. We're still investigating the cause though.

Started 18 Jul 2012 21:05:50
Closed 18 Jul 2012 21:18:37

31 May 2012 17:59:08
31 May 2012 16:14:16

We're investigating a packet loss / routing problem at the moemnt, this will be affecting broadband customers, more details to follow.

31 May 2012 16:54:57

Looks like this was something quite major and country wide affecting various interconnects that the internet uses.

Things are geting back to normal now.

ADSL lines which did go down are re-connecting.


Service has been restored. This looks to have been caused by a problem within the LINX peering network which affected other networks and interconnects accross the country.

We've re-enabled our LINX peering.

Started 31 May 2012 16:03:32
Closed 31 May 2012 17:59:08

12 May 2012 10:35:45
12 May 2012 08:48:06

At 08:06:58 all 20CN and 21CN lines routed via Manchester dropped and at the same time we stopped seeing any platform RADIUS requests from BT.

This suggests a major routing issue within BT. It means that nobody can re-connect any broadband line.

We are trying to get an answer out of BT on this.

12 May 2012 08:58:33

BT have a string of open issues covering Manchester area. No ETA yet.

12 May 2012 09:08:05

We have confirmation that this is not just affecting us (thanks Thomas). BT are investigating.

12 May 2012 09:42:39

Whilst BT are being a tad inconsistent in telling various ISPs what is happening, we have heard from another ISP that "Engineer is onsite at Manchester with a spare card. Changeout is about to commence with the assistance of vendor  support. Next update will be at 10:20"

12 May 2012 10:20:20

BT are not confirming the statement that a card is to be replaced, and in fact appear to be totally ignoring me now.

12 May 2012 10:42:58

Looks to be working.

Manchester lines reconnecting

Platform RADIUS working


We are waiting for more detail on BT on this, and how it is they appear to depend entirely on a single card in Manchester for the entire BTW platform.

Started 12 May 2012 08:06:58
Closed 12 May 2012 10:35:45

11 May 2012 09:08:46
[Broadband] - ADSL blip - Closed
10 May 2012 11:51:17

Lines on our "b.gormless" LNS dropped. We're investigating. Lines are coming back.

10 May 2012 11:57:04

It looks like BT. Still investigating.

10 May 2012 11:59:33

It looks like BT's link to b.gormless has gone down. We've taken b.gormless out of service.

10 May 2012 12:00:24

BT fibre link is down - lines are moving to the backup LNS automatically.

10 May 2012 12:35:49

BT have raised a fault and are investigating their end.
Update due by 13:15 

10 May 2012 13:00:50

Most lines swicthed very quickly to the alternative LNS automatically.

The status of lines on the control pages too some time to catch up on the RADIUS accounting, but this has now happened.

Graphs may have gaps which will be repaired on the over night status.

10 May 2012 19:55:22

Although broadband lines look fine this link is still down, meaning we have lost resilience on links to BT.

We are still chasing.

10 May 2012 20:11:47

BT are arranging an engineer to go to the Datacentre.

10 May 2012 22:25:45

Still not heard anything more.
Chasing again. 

11 May 2012 09:09:57

Came back around 00:09.
It appears that this was a problem with a card BT side.
Need to confirm though. 


Cleared fault on Card.

Closed 11 May 2012 09:08:46

16 Apr 2012 19:04:46
16 Apr 2012 18:58:02

One of the three live LNSs just restarted.

We are investigating.

16 Apr 2012 18:58:50
I looks like the backup systems kicked in within seconds as planned.
16 Apr 2012 19:00:41
Many lines have correctly fallen back to D.gormless. They will be moved over night back to B.
16 Apr 2012 19:05:14

Lines have reconnected cleanly as per the contingency plans. We have to find the cause, but I'll close this major issue for now.

(Graphs would have been lost for the day, sorry)

Broadband Users Affected 33%
Started 16 Apr 2012 18:53:14
Closed 16 Apr 2012 19:04:46

02 Apr 2012 09:03:24
29 Mar 2012 00:24:09
Lines: 100% 20CN-REGION-READING and 100% 21CN-REGION-SL dropped at 2012-03-29 00:22:50
We have advised BT
This is likely to have affected multiple internet providers using BT
30 Mar 2012 12:07:50

Lines came back shortly after, we'll chasing BT for information and reasons.

02 Apr 2012 09:03:33

This was caused by a BT PEW.

Broadband Users Affected 7%
Started 29 Mar 2012 00:22:50 by AAISP automated checking
Closed 02 Apr 2012 09:03:24
Cause BT

14 Mar 2012 16:19:53
14 Mar 2012 16:14:21

One of the three LNSs in live use just restarted and we are investigating.

Lines are moving automatiucally to the backup LNS within seconds.

14 Mar 2012 16:19:53

This is the first test of the three LNS plus fallback confiuguration.

By all accounts customers reconnected very quickly - within seconds.

We have a clue what triggered the problem and we are investigating now.

Broadband Users Affected 33%
Closed 14 Mar 2012 16:19:53

27 Feb 2012
[Broadband] - LNS issues - Closed
26 Feb 2012 09:38:55

Something else went wrong this morning and we are investigating.

26 Feb 2012 09:44:02

Another crash - this time somewhere completely different and equally impossible.

26 Feb 2012 09:46:52

It looks like lines reconnected quickly over both LNSs as you would expect from a crash. This is an excellent test but obviously this sort of thing simply should not happen.

26 Feb 2012 12:39:54

OK we are moving people that connected to D after it crashed over to C. It is about a 1/3 of customers that are being moved. The controlled LNS switch is deliberately slow taking one line at a time over and ensuring no load on BT or our RADIUS that could cause default accepts or other delays or problems.

26 Feb 2012 12:49:06

We are installing new code with additional debugging on D, and will move some lines back to D shortly.

26 Feb 2012 12:49:35

This means we will lose graphs for this morning.

26 Feb 2012 12:57:35

Looks like a few dozen lines got a PPP kill twice. We have updated the scripts now, and we are moving 1000 lines over from C back to D now. The plan is to leave this for this afternoon to confirm all is working and stable.

26 Feb 2012 14:20:34

We are moving some more lines over now.

26 Feb 2012 15:06:16

We are moving the last of the lines over now.

26 Feb 2012 15:50:43

All lines are on the new code on the "D" LNS.

The problem is we have not got to the underlying cause yet.

We'll leave service like this for the rest of the day.

27 Feb 2012 14:01:47

Having been stable for over 24 hours I am closing this major issue but we will continue to monitor carefully and try and find the underlying cause.

Started 26 Feb 2012 09:07:42
Closed 27 Feb 2012

23 Feb 2012 07:28:40
[Broadband] - LNS issue - Closed
23 Feb 2012 07:12:01

There seems to have been some major issue over night with one of the LNSs in the new rack. Something odd with RADIUS from around 1am, and then lines dropping and reconnecting around 04:52. Graphs before 04:52 on one of the LNSs are lost as well. It seems problems persist with some BE lines though.

We are investigating.

23 Feb 2012 07:15:23

We are switching lines to the right LNS now.

23 Feb 2012 07:25:50

Stuck BE tunnels have been cleared and BE lines are all reconnecting now.

23 Feb 2012 07:29:11

Looks like the issue here resulted in BT lines coming straight back, but BE lines not doing so. We now have all lines back.

Started 23 Feb 2012 01:00:00
Closed 23 Feb 2012 07:28:40

17 Feb 2012 12:06:00
17 Feb 2012 12:07:06

We've just had a blip on some BT ADSL/FTTC lines, most are back online already. We'll investigate the cause.

17 Feb 2012 12:11:53

BT lines on just one of the LNS's blipped - lines on C.Gormless blipped, lines on D.Gormless did not. 

Lines are almost all back online now.

17 Feb 2012 12:14:52

This was affecting BT lines. BE lines were not affected.

17 Feb 2012 12:24:32

The network switch port connected to one of our BT fibres was offline between 12:02:36 and 12:04:23.

We're investigating further.


Half of our BT circuits dropped at 12:02, and starting coming back online at 12:05, most lines were online within a few minutes.

This was caused by a drop in one of our links to BT. A fault has been logged to BT regarding this. We will move customers off this link over the weekend so as to enable us to investigate this with BT in more detail.

Started 17 Feb 2012 12:03:00
Closed 17 Feb 2012 12:06:00

10 Feb 2012 06:27:00
10 Feb 2012 03:04:22

It looks like around half od the BE lines went off around 02:43 and the rest at 02:58.

We were not aware of planned work this morning, though various work has been planned this week.

10 Feb 2012 03:27:35

We have tried routing traffic via our old rack as well, and no joy.

10 Feb 2012 03:30:16

Obviously customers using BE+BT lines as a fallback arrangement are fine, using BT.

Obviously customers using 3G backup are working.

If you only have a BE line you may want to consider some of the fallback arrangements we can offer. Though it is rare for it to be BE that is the carrier that has failed, it is sensible to consider contingencies.

That said, we would hope they have this fixed during the night.

10 Feb 2012 03:39:19

There were unplanned outtages on 14 exchanges at approximately those times, we're still waiting to find out why this occured.

10 Feb 2012 05:30:00

O2 have confirmed that changes made by them to the network overnight resulted in loss of service to large parts of the network, including all wholesale services.

Their engineers are currently rebuilding configurations on all aggregation nodes on their network to reverse the changes. We are beginning to see sessions establish again, about 20% have reconnected in the past few minutes.

10 Feb 2012 06:30:00

O2 engineers have restored configuration on about 40% of core devices. We are seeing roughly that % of sessions back up.

10 Feb 2012 08:03:43

It appears that O2 engineers have now finished restoring configurations, as we can see all ISAMs on our monitoring again and we are seeing the majority of customers back online.


This was due to BE engineering work that failed. An initial report of the problem is available at http://aa.net.uk/news-2012-02-10-be.html

Started 10 Feb 2012 02:43:00
Closed 10 Feb 2012 06:27:00

09 Feb 2012 14:50:00
09 Feb 2012 14:55:22

There was a routing problem affecting ADSL lines.

We initially thought this was just our office (hence the previous post)

Routing is now restored and we're investigating the cause.

Started 09 Feb 2012 14:30:00
Closed 09 Feb 2012 14:50:00

01 Feb 2012 02:11:00
01 Feb 2012 02:43:36

There was, quite separate to any other issues, a problem at 02:06 which resulted in the main LNS resetting. All customers quickly reconnected with no problems.

We have detailed logs of the actual problem, but it is baffling us slightly at present. We will be investigating this later in the day. This does, however, appear to be completely unrelated to everything else that has been happening.

We are rebalancing some lines between LNSs over night after this incident.

Started 01 Feb 2012 02:06:07
Closed 01 Feb 2012 02:11:00
Previously expected 01 Feb 2012 02:11:00

31 Jan 2012 19:18:42
31 Jan 2012 18:57:10


31 Jan 2012 19:12:55

Something major with our LNS - investigating

31 Jan 2012 19:16:09

No clear on cause but looks like our fault not BT this time.

31 Jan 2012 19:16:32

Graphs lost, sorry

31 Jan 2012 19:16:59

Looks like people coming back on line now

31 Jan 2012 19:18:42

Not good - lost graphs - and cause unclear...


We will be investigating this in the morning. More details on my blog http://revk.www.me.uk/2012/02/ooops.html

Started 31 Jan 2012 18:56:00
Closed 31 Jan 2012 19:18:42

13 Jan 2012 11:37:13
13 Jan 2012 11:07:52

We're looking in to a network problem affecting broadband customers - More details to follow...

Lines are coming back and we arte trying to find what happened.

13 Jan 2012 11:22:05

We have engineers onsite in telehouse looking in to this.

However, lines and routing are recovering now.

13 Jan 2012 11:22:07

Looks like some network issue in our core rack in telehouse - engineers on site investigating now. Lines mostly back up now.

13 Jan 2012 11:25:35

We are still somewhat at a loss as to what happened, but it has meant all lines being dropped of the live LNS. We have switched the the backup anyway.

However, and possibly unrelated, we are seeing issues with BT 20CN lines coming up - i.e. they are not. This was happening before this more major issue, and means 20CN lines are still down. We are chasing With BT.

13 Jan 2012 11:35:49

20CN lines now on - looks like BT did a config change early which resulted in problems. This would have been a non issue if there was not also some sort of major blip which we have yet to identify.


Two problems it seems.

1. BT did a configuration change earlier than expected. No idea why, and normally this would have affected a couple of customers at most. We were looking in to this already when...

2. Somehow there was a loss of contact between our live LNS and core network. We can see this must have happened and we are checking the switch logs to see if we can find why. Engineers were on site in the new rack (not connected, do not the cause!) and are checking this out now.

The effect was all lines went off. We have changed the system to be less sensitive in such cases now and not drop all sessions due to a loss of connectivity to the core network.

However, as all lines went off, the BT config change meant none of the 20CN lines came back until we discovered that what they had done.

Almost all lines on-line, and all should now be able to connect as normal.

All of the switches are being replaced as part of the upgrade anyway, but we obviously want to understand the underlying issue here.

Update: This is looking increasingly like a switch issue - the switch is working now, and new switches are in the new rack ready to be deployed over the next week or two.

Started 13 Jan 2012 11:03:30
Closed 13 Jan 2012 11:37:13

12 Jan 2012 02:42:51
12 Jan 2012 01:34:38

Seems a major problem affecting all broadband lines - service si restoring automatically, but taking several minutes to get people back on line.

12 Jan 2012 01:39:17

Lines are coming back but much more slowly than we would expect

12 Jan 2012 01:49:24

Both LNSs are seeing people connect constantly and then disconnect, which makes little sense. We are still investigating the issue now.

12 Jan 2012 02:00:00

We have reset equipment and switched LNSs, which has affected lines that were not previously affected. We are trying to see a pattern here.

12 Jan 2012 02:12:24

Logs confirm that at 2am. suddenly things started working a lot better and many more lines coming back. Still investigating what is going on here.

12 Jan 2012 02:19:29

We think, whatever it was, has magically fixed itself. We are going to clear all sessions to one LNS now, so a blip again for some people.

12 Jan 2012 02:33:27

We are not entirely sure of the cause of the original problem, but it is such a long time since we have had a major issue like this it appears that we currently have a problem with the speed with which our systems can recover.

It seems our RADIUS server is struggling and once overloaded it is too slow to handle the connections and so the connections timeout and re-try causing more load and more delay. The result is that while everyone is trying to connect, nobody can, and it basically took well over an hour for lines to connect.

More investigation to follow.

12 Jan 2012 02:43:21

I am closing the mahor incident on this now - but we are investigating the cause and the slow reconnect.

Started 12 Jan 2012 01:08:53
Closed 12 Jan 2012 02:42:51

15 Nov 2011 15:21:16
[Broadband] - Line blip - Closed
15 Nov 2011 14:18:51

A significant percentage of our broadband lines blipped briefly, but appear to be coming back.

We're investigating.

15 Nov 2011 14:30:42

A number of lines have ended up on the wrong LNS - we're rebalancing sessions now.


This appears to have been an over sensitive test on one of our systems that thought we had lost connectivity and shutdown sessions. The test is being changed and code changes to adjust timeouts to avoid this in future.

Started 15 Nov 2011 14:17:22
Closed 15 Nov 2011 15:21:16

21 Oct 2011 09:54:00
21 Oct 2011 08:53:16

BT ADSL lines are down, Ethernet circuits are down too.

21 Oct 2011 08:55:50

The seems to be a BT outage in Stepney Green where we (and others) interconnect in to BT.

21 Oct 2011 08:59:23

Lines are coming back

BT have raised an Incident and are aware of a problem

21 Oct 2011 09:01:25

Lines are still logging in.

If you're not up yet, please do wait a bit longer.

There should be no need to reboot ADSL routers, as they should reconnect. 

21 Oct 2011 09:08:05

Lines are still reconnecting now - it will take a bit more time before all lines are back on.

21 Oct 2011 09:09:53

This does not seem to have affected Ethernet Lines, just BT ADSL. (BE ADSL lines were not affected)

21 Oct 2011 09:16:47

If customers are getting BT 'Service information' webpage coming up then please reboot your router.

-This will be caused by BT accepting your login but not passing it on to us. Rebooting will cause the router to relogin, and should get through to us.

21 Oct 2011 09:32:27

It is worth noting that BT have had issues with Stepney all week. We have been relatively lucky up until now not to be affected, but we have been working with a number of ISPs connected to Stepney during the week who have suffered similar issues. We don't have any exact details from BT as to the cause of the problems as yet, but clearly they are working on them.

21 Oct 2011 11:41:33

This incident is now closed. 

We've had confirmation from BT that this was affecting other ISPs too.

BT clearred the fault at 08:54.

By 09:05 we had about 50% of ADSL lines back online.

It was also reported by ThinkBroadband

Started 21 Oct 2011 08:37:00
Closed 21 Oct 2011 09:54:00
Cause BT

21 Oct 2011 09:02:34
21 Oct 2011 08:39:02
Lines: 20% ALL and 22% BT and 28% 20CN and 19% 21CN and 100% 20CN-BRAS-ERX11-READING3 and 100% 20CN-BRAS-ERX2-FARADAY and 100% 20CN-BRAS-ERX7-READING2 and 100% 20CN-BRAS-ERX8-BIRMINGHAM2 and 100% 20CN-BRAS-ESR1-MILTONKEYNES3 and 100% 20CN-BRAS-ESR1-SHEFFIELD3 and 100% 20CN-BRAS-ESR10-MILTONKEYNES4 and 100% 20CN-BRAS-ESR6-KINGSTON4 and 100% 20CN-BRAS-ESR6-READING4 and 100% 20CN-BRAS-ESR7-KINGSTON4 and 82% 21CN-BRAS-RED1-NT-B and 100% 21CN-BRAS-RED2-PE dropped at 2011-10-21 08:37:06
We have advised BT
This is likely to have affected multiple internet providers using BT
21 Oct 2011 08:56:18

Updates to this incident have moved to: http://status.aa.nu/apost.cgi?incident=1271

21 Oct 2011 09:08:59

Lines coming back - more info onthe other status message posted above.

Broadband Users Affected 18%
Started 21 Oct 2011 08:37:06 by AAISP automated checking
Closed 21 Oct 2011 09:02:34
Cause BT

20 Oct 2011 19:31:00
20 Oct 2011 19:30:33

It looks like most/all BE lines are down. We're investigating...

20 Oct 2011 19:35:06

BE lines are reconnecting now.

20 Oct 2011 19:45:45

From the look of the logs, we lost all the L2TP tunnels to BE between 19:24 and 19:31.

We'll post again when we find out what the cuase was.

20 Oct 2011 20:01:04

The outage has been confirmed to be cuased by a power outage.

Started 20 Oct 2011 19:25:00
Closed 20 Oct 2011 19:31:00

19 Oct 2011 09:33:33
18 Oct 2011 11:33:36

BT have informed us that an:

Emergency PEW is planned to carry out urgent maintenance work at Stepney Green. From 00:01 AM -02:00AM there will be a possible short impact to service while traffic is being rerouted within BT core to wbmc shared service at Stepney Green.
From 02:00 AM to 05:00 AM The impact will be:
Total loss of service to wbmc shared end users who are connected to the Stepney Green 21cn Bras's during the Pew window of 02:00 to 04:00.
Total loss of service to IPSC/wbmc end users who are connected to the Ilford 20cn Bras's during the Pew window of 02:00 to 04:00.
The customers will experience will be that they will be unable to connect to their CP's will the planned work in being actioned.

18 Oct 2011 12:16:49

We think this will be little impact for most customers, but may be a period off line for people on Stepney BRASs directly.

19 Oct 2011 09:33:43

the planned works are now over.

Started 19 Oct 2011 by BT
Closed 19 Oct 2011 09:33:33
Previously expected 19 Oct 2011 06:00:00 (Last Estimated Resolution Time from BT)

12 Oct 2011 23:59:59
12 Oct 2011 20:48:25

We see usage have occasional spikes, and they often have a reason, but this evening we have seen silly high usage since around 18:40, not only on 21CN but also 20CN. Something is clearly "up" and there is some "internet event" happening. No idea what yet, but has caught us unawares.

The system does the usual to manange the traffic - give more to the "premium" paying customers, and to generally give small packets for VoIP, DNS, and interactive traffic priority.

But we'll try and work out why and if necessary get more capacity in shortly.

P.S. it is not just us cuaght out by this - logs show some lines seeing loss and latency which is within BT's network (as we ensure the LCP echos get sent anyway).

12 Oct 2011 20:53:51

This is worse than the world cup traffic!

12 Oct 2011 20:55:08

Only clue is new Apple IOS5 stuff - if that is the cause I am impressed.

12 Oct 2011 20:57:04

Usage has just reached unprecidented levels - we have not seen anothing like this...

12 Oct 2011 20:59:52

Customers are even reporting issues on Be lines, what is this?

12 Oct 2011 21:02:30

We have core links hitting 1Gb/s - something we have been planning for in 6 months time - this is really unprecidented usage levels.

12 Oct 2011 21:34:07

One good thing from this is that it has provided the FireBrick team with a good benchmark of what happens when you push both the LNS and BGP routers to the full gigabit throughput. The answer is that it copes really well and is using CPU levels that are actaully very good. But what on earth is causing this unprecidented levels of usage is not entirely clear.

So far we thing there is a windows update and apple IOS5 release, but is that all it is?

13 Oct 2011 08:45:44

We are guessing this was IOS5 release.

14 Oct 2011 18:15:07

Whilst Thursday night was also somewhat excessively busy it was nothing like the previous nigh, and caused some congestion mostly on 21CN.

Closed 12 Oct 2011 23:59:59

18 Sep 2011 21:35:00
19 Sep 2011 09:33:10

All our BE ADSL lines lost connection at 21:30 on Sunday night due to a power failure in our supplier's network. The outage was brief and most lines had reconnected by 21:35.

Started 18 Sep 2011 21:30:00
Closed 18 Sep 2011 21:35:00

09 Sep 2011 11:13:49
09 Sep 2011 11:00:09

We are investigating a problem with short periods (seconds) of routing problems which will be affecting BT and BE broadband lines. Customers may be seeing no connectivity for short periods at a time. 

09 Sep 2011 11:13:49

This has been now resolved. 

Started 09 Sep 2011 10:50:00
Closed 09 Sep 2011 11:13:49

23 Aug 2011 03:30:00
22 Aug 2011 11:05:20

As the planned works notice is rather short, we are posting this as a major issue now. We expect all BT lines to go out of service over night from around 2am to 3am. This will not affect BE lines.


20CN lines had a short outage (half hour) and 21CN lines longer (90 minutes)

Started 22 Aug 2011 11:00:00
Closed 23 Aug 2011 03:30:00