Posted at 2009-10-21 20:32 BST by RevK
Update #8: 2009-10-22 08:59 BST

  Something major happened and dropped a lot of lines.
  
  That appears to only have affected 20CN lines. Lines are coming back
  now. be lines were unaffected.
  
  We are trying to find the exact cause. We had made a very minor change
  and restarted BGP to BT on the backup LNS at the same time but if that
  was the cause then BT have something seriously screwed in their BGP.
  The work was not even put on as a "at risk "planned work because work
  on the backup LNS is done quite regularly with no issues.
  
  We'll endeavour to find the cause. We're waiting for BT.
  
  Update: We are running a clear on stuck sessions. Yes, this was meant
  to be fixed. It is being very slow to reconnect everyone. Looks like
  over half an hour to get most people back on line, so not good enough.
  
  We can't say for sure but we can't see how this was us. We'll see what
  BT say.
  We know slow reconnects is down to BT, and that is being chased.
  
  Update: 01:20, log analysis suggests that the timing was too perfect
  and this looks like BT had a major wobbly as a result of a simple BGP
  reconnect on a backup router. We are investigating this with BT.
  
  Update: 7am, BT have sent BGP logs from there end which confirm BGP did
  not drop at all on the live LNS so they have no clue why 20CN dropped
  when it did. We're not keen to try tests on this given the
  consequences, but will keep an eye on anything like this happening
| again.
> 
> Update: Interestingly other ISPs reported issues last night as well.
> Logs show the issue was BT not seeing traffic from us but us seeing
> traffic from BT but only on 20CN lines.[IMAGE]

URL: http://aaisp.blogspot.com/2009/10/open-major-blip-2030.html

-- 
AAISP Status Blog
URL:http://aaisp.blogspot.com/