There appears to be a major issue with fibre links in to our node in Maidenhead. This is also affecting out office which is why there was a delay posting this.
Staff are on the way to site now to investigate.
Staff are on the way. As ever, backup equipment was not set up right in our offices which had led to a delay updating customers on this. Incoming calls are being handle by very few staff at present which is also causing delays taking calls. We expect staff to be back on it in a few minutes.
From what we can see most things in the data centre are working, e.g phones. But the fact we are unable to call the data centre suggests they have something major going on. More to follow. Staff still not managed to get on site yet.
Ok still no proper news, but the data centre is apparently swamped with openreach, suggesting something major, but also suggesting that thy are on the case.
We can confirm that all the fibre links to our rack in Maidenhead are showing alarms at present, but that power and everything else is fine. BT vans are on site and they have a big hole in the ground, but the BT engineers claim nothing wrong and they did not touch anything.
We are reporting multiple fibre failures to BT now.
Oh, and no clue why data cent were not answering the phone. We'll worry about that later.
We have confirmation that other customers in the data centre have also lost fibre links. BT are on the case but no ETA yet.
Ok. Clarifying rumours here. BT vans on site, apparently on unrelated things. Hole in ground is near site and some sort of construction work. May be unrelated. Maybe not. We are waiting up updates from BT now.
It appears that BT had internal system problems meaning they have not actually logged the fault internally until just before 4pm!!
Whilst delays in BT are frustrating, the fact this appears to be a fibre break with other people affected means they should already be working on it. We hope. We are still awaiting an update.
Further checks confirm that our link from our office to Maidenhead is only half broken, which is surprising. I.e. one of the two fibres is apparently working. This suggests a damaged rather than cut fibre.
So far, all we have, is that BT are running diagnostics. No more information yet.
Now waiting for a call from the Openreach engineer who is running diagnostics...
There are two Openreach engineers on site investigating. They have said that they're investigating someone elses's fault on site so sounds like they might have found a common problem.
BT engineer turned up on site to find two other engineers already working on it. So it is in hand. No ETA.
BT found what looks to be a physical fibre fault (so a break somewhere - notes added at 19:24) and confirmed at 19:37 that they are still working on it.
BT are still working on the break. No ETA yet.
Still no news. BT should call back, hopefully with more information, soon.
Whilst we expect BT to sort this over night, we may not have status updates until early morning. All systems are set up to come back on line as soon as BT fix the fibre.
BT are trying to put in a temporary fix to the problem. No details as to the root cause, and no time scale for a fix yet, but they've said that if the temporary fix fails they'll have to replace the broken fibre. If that happens it is unlikely to be fixed tonight.
Most affected customers seem to have come back online in the last few minutes.
BT have confirmed that this is a "permanent fix", by which they mean that any further disruption should be a part of planned engineering works.
Our links for Ethernet services have been restored and our office is back on line. We're waiting for a full explanation from BT.
More details: http://aa.net.uk/news-2011-fibre-break.html
BT fixed broken fibre.