RESOLUTION

I wanted to try and explain more about what happened today, but it is kind of tricky without saying "Something crazy in the routing to/from BT". We did, in fact make a change - something was not working with our test LNS and a customer needed to connect. We spotted that, for some unknown reason, the routing used a static route internally instead of one announced by BGP, for just one of the four LNSs, and that on top of that the static route was wrong, hence the test LNS not working via that LNS. It made no sense, and as all three other LNSs were configured sensibly we changed the "A" LNS to be the same, after all, this is clearly a config that just worked and was no problem, or so it seemed. Things went flappy, but we could not see why. It looks like BGP in to BT was flapping, so people connected and disconnected rather a lot. We returned the config and things seemed to be fixed for most people, but not quite all. This made no sense. Some people are connecting and going on line, and then falling off line. The "fix" to that was to change the endpoint LNS IP address used by BT to an alias on the same LNS. We have done this in the past where BT have had a faulty link in a LAG. We wonder if this issue was "lurking" and the problem we created showed it up. This shows that there was definitely an issue in BT somehow as the fix should not have made any difference otherwise. What is extra special is that this looks like it has happened before - the logs suggest the bodge of a static route was set up in 2008, and I have this vague recollection of a mystery flappiness like this which was never solved. Obviously I do apologise for this, and having corrected the out of data static route this should not need touching again, but damn strange.