We're seeing low levels of continuous packet loss on CityFibre circuits in Maidenhead (Around 5%). This started at 2AM on Monday.
The timing of the loss correlates with a CityFibre planned OLT software upgrade in the Maidenhead area (Their reference for this work was CHG0073887)... this has been reported to CityFibre.
CF have found a config issue and are taking steps to fix, update to follow ........
CityFibre have scheduled work to to fix their OLT configuration on 18th and 19th April! This is under their reference of CHG0076369. This is not acceptable and we are escalating. We only have a couple of circuits in Maidenhead, but we'd expect all CityFibre circuits in Maidenhead which are connected to the OLTs to all be experiencing 5% packetloss which will be disrupting their internet connections.
Due in 3 hours
We're investigating connectivity problems.
Network is recovering
We are seeing some packet loss on our network again, we're still investigating this.
The Network is stable at the moment but the cause of this is still under investigation.
For information, the problems have occurred for minutes around: 08:55, 10:00, 10:10, 10:30. Customers would have had 'routing' problems during these times - ie problems reaching websites and places on the internet. More information to follow.
The disruption today has been caused by problems with our 'route servers'. These are BGP routers that manage all our internal and external IP address announcements and routing information for our network - they exchange IP address information between all our LNS routers and our edge transit/peering routers. For resilience, we run two route servers, in separate data-centres. They both exchange routes with all the other routers on our network. Each route server is also physically connected to two network switches.
The disruptions today were a result of both of our route dropping their Ethernet connections to their switches at a similar time, which caused a BGP sessions to drop which cascaded to the routing problems that were experienced at the time.We are still not 100% sure of the cause and investigations are continue. However, later software does manage the Ethernet ports slightly different and at 11:30 we upgraded the software on one of the router servers.
We are still monitoring the networkNetwork has remained stable since 10:30AM
The graph shows the last few hours of logins and logouts of ADSL, VDSL, SIMs and L2TP circuits.
The current time is on the left. Green is login, red is logout.
If there are spikes, then this shows a large number of logouts, which may indicate an outage or planned work happening.
You can click on a spike to search for incidents or maintenance that were open around that time.
This will search through the various outages and maintenance reports we have received from our suppliers which may be affecting you.
You can also see a list of outages and maintenance work we receive from our suppliers that may affect your specific broadband service on the Control Pages.
The search box above should find posts affecting your service however, you can use a Keyword search.
This is the status page of Andrews & Arnold Ltd.
Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.
The last update was Yesterday 21:08:16