Arounds half of our lines went off line just after midnight.
Investigating now.
The problem appears io be that BT have dropped several of their BGP routes causing lots of their BRASs to become inaccessable.
It looks like BT are not announcing their 21CN network to us. The affected lines appear to be all 21CN lines.
Some 20CN appear to be affected as well.
BT are working through their planned network upgrades and this morning they are doing Birmingham. This should only affect Birmingham connected exchanges (20CN and 21CN).
However, just like last time it has had an impact on BGP announcements to us (off Stepney Green, not Birmingham) such that BT drop *all* routers to 21CN BRAS.
The work will be completed in a few hours and all should be well.
BT staff have captured logs of the BGP announcements their end that confirm they are not announcing the routes to us via either of the links. Oddly they are announcing to other ISPs off the same node (Stepney Green). It looks like it is just us that have this unexpected dependancy on Birmingham
Sorry, to clarify, planned work in Birmingham will affect other ISPs. It is the unexpected linking of us to Birmingham making it a much bigger impact that is "just us"...
20CN Birmingham back
BGP routes came back when 20CN Birmingham lines came back, but 21CN lines not back on yet...
Looks like we are getting 21CN platform RADIUS but not L2TP yet
BT are investigating why we are not seeing connections from 21CN yet.
Lines finally starting to come back
Lines back up
BT now understand the issue and changes have been made to avoid this problem in future. It seems we helped uncover some configuration issues that were not clear. It seems the side effects of Birmingham PEW were much wider than just us and caused problems for various ISPs in different ways. It is always nice to actually get to the bottom of a problem like this and fix the root cause - well done to D&K at BT.