Posted at 2009-10-06 09:31 BST by RevK
Update #0: 2009-10-06 09:39 BST

The congestion issues we have been chasing on 21CN are getting worse
and worse now.

We are continuing to work closely with BT technical staff to help them
diagnose and resolve this. Tests already done indicate this is not
specific to us or WBMC (the specific way we link to BT).

We are currently seeing 30% less peak traffic through 21CN than last
month which we can only assume is down to congestion within BT's 21CN
network. This just shows how much of a problem this is now. We have
loads of spare capacity our end!

We can only assume other users of 21CN such as BT Retail have the same
issue dispute their adverts apparently suggesting no slow down at peak
times on their new 21CN lines. We can't be sure this is the case, but
if not then that is a separate issue to address! We'll be interested to
hear from people that have access to such lines to make a comparison.

Obviously we'll update you with any progress. A number of our customers
have moved to an ISP still using 20CN or asked us to move their line to
20CN. At present it is not simple to move lines to 20CN, but we'll have
to get BT to sort that if congestion on 21CN continues.

The current technical update on this is that it is believed to be an
issue on the 21CN MSANs, and BT are working with the MSAN vendors to
resolve this. The MSAN is the equipment at the telephone exchange that
connects to your line. I'd like to thank those customers that are
helping with the testing.

This issue is just the latest in a string of issues with BT failing to
scale up 21CN effectively. Most of the other issues are resolved now,
and hopfully this one will be too.

Thank you all for your patience.[IMAGE]

URL: http://aaisp.blogspot.com/2009/10/info-21cn-congestion.html

-- 
AAISP Status Blog
URL:http://aaisp.blogspot.com/

Posted at 2009-10-06 09:31 BST by RevK
Update #1: 2009-10-06 17:33 BST

  The congestion issues we have been chasing on 21CN are getting worse
  and worse now.
  
  We are continuing to work closely with BT technical staff to help them
  diagnose and resolve this. Tests already done indicate this is not
  specific to us or WBMC (the specific way we link to BT).
  
  We are currently seeing 30% less peak traffic through 21CN than last
  month which we can only assume is down to congestion within BT's 21CN
  network. This just shows how much of a problem this is now. We have
  loads of spare capacity our end!
  
  We can only assume other users of 21CN such as BT Retail have the same
  issue dispute their adverts apparently suggesting no slow down at peak
  times on their new 21CN lines. We can't be sure this is the case, but
  if not then that is a separate issue to address! We'll be interested to
  hear from people that have access to such lines to make a comparison.
  
  Obviously we'll update you with any progress. A number of our customers
  have moved to an ISP still using 20CN or asked us to move their line to
  20CN. At present it is not simple to move lines to 20CN, but we'll have
  to get BT to sort that if congestion on 21CN continues.
  
  The current technical update on this is that it is believed to be an
  issue on the 21CN MSANs, and BT are working with the MSAN vendors to
  resolve this. The MSAN is the equipment at the telephone exchange that
  connects to your line. I'd like to thank those customers that are
  helping with the testing.
  
  This issue is just the latest in a string of issues with BT failing to
  scale up 21CN effectively. Most of the other issues are resolved now,
  and hopfully this one will be too.
  
| Thank you all for your patience.
> 
> Update: BT are doing tests on specific lines to test a theory and we
> hope to hear answers soon.
> 
> FAQ:-
> 
> Q: Are we sure this is BT and not A&A LNS? After all the A&A LNS is new
> isn't it?
> A: We are sure. One of the affected lines is in our office and we have
> tested BT and BE side by side which go to the same LNS our end and come
> from the same laptop in our office and use the same test sites to
> confirm throughput. We have also checked the LNS load is low as
> expected. We have also checked that we definitely have loads of
> headroom on the link to BT at our end.
> 
> Q: Are we sure it is all BT 21CN ISPs not just A&A or the way A&A
> connect.
> A: Yes. BT have done tests connecting a line to a different endpoint in
> BT bypassing the link to A&A and not using a WMBC link at all (WMBC is
> the specific way we connect to BT). They saw the problem.
> 
> Q: Do we know how widespread it is?
> A: Not exactly. Unlike other bank-haul congestion issues we have found
> and had fixed by BT this is not as obvious and only shows when tested.
> We have been trying to make quantative tests for this but have not yet
> managed something that is unobtrustive or totally reliable. The biggest
> indicator of the scale of the issue is that peak usage one 21CN has
> gone down by over 30%. This suggests a lot of exchanges are not
> affected.[IMAGE]

URL: http://aaisp.blogspot.com/2009/10/info-21cn-congestion.html

-- 
AAISP Status Blog
URL:http://aaisp.blogspot.com/

What symptoms should we look out for to see whether we are affected by this 
problem? Should it affect maximum download rate? Should it happen more at 
certain times of day?

I still seem to get pretty much my full BRAS rate (5.5M) for a download from a 
fast source, but web pages that require a lot of connections with small amounts 
of data each seem to take longer than when I first went onto 21CN. This might 
simply mean the sites I'm contacting have more traffic.

Alfred E Neuman  wrote:
> What symptoms should we look out for to see whether we are affected by this 
> problem? Should it affect maximum download rate? Should it happen more at 
> certain times of day?
> 
> I still seem to get pretty much my full BRAS rate (5.5M) for a download from a 
> fast source, but web pages that require a lot of connections with small amounts 
> of data each seem to take longer than when I first went onto 21CN. This might 
> simply mean the sites I'm contacting have more traffic.

I'm seeing what I believe is this problem intermittently, at what I
would call peak home user times (certainly weekday early evenings).

When the problem occurs it results in degraded download performance,
increased latency, and sometimes frequent modem resets.

-- 
Dominic Hargreaves | http://www.larted.org.uk/~dom/
PGP key 5178E2A5 from the.earth.li (keyserver,web,email)

On Tue, 6 Oct 2009 22:30:55 +0000 (UTC), Dominic Hargreaves wrote:

>Alfred E Neuman  wrote:
>> What symptoms should we look out for to see whether we are affected by this 
>> problem? Should it affect maximum download rate? Should it happen more at 
>> certain times of day?
>> 
>> I still seem to get pretty much my full BRAS rate (5.5M) for a download from a 
>> fast source, but web pages that require a lot of connections with small amounts 
>> of data each seem to take longer than when I first went onto 21CN. This might 
>> simply mean the sites I'm contacting have more traffic.
>
>I'm seeing what I believe is this problem intermittently, at what I
>would call peak home user times (certainly weekday early evenings).
>
>When the problem occurs it results in degraded download performance,
>increased latency, and sometimes frequent modem resets.

Congestion shouldn't cause modem resets, that's more likely to be a
modem/local problem.

Does the modem actually reset or simply reconnect?

What are your modem stats (SNR margin, sync speed and attenuation) both
when you don't have a problem and when you do?

Have you tried plugging into the test socket behind the cover (if you
have an NTE5 master socket)?

What do the AAISP logs say about it (https://clueless.aaisp.net.uk/
login and select the line link at the top of the graph)?

-- 
Regards - Rodney Pont
The from address exists but is mostly dumped,
please send any emails to the address below
e-mail	ngpsm4 (at) infohitsystems (dot) ltd (dot) uk

RevK wrote:
> Posted at 2009-10-06 09:31 BST by RevK
> Update #2: 2009-10-07 15:40 BST
> 
>   The congestion issues we have been chasing on 21CN are getting worse
>   and worse now.
>   
>   We are continuing to work closely with BT technical staff to help them
>   diagnose and resolve this. Tests already done indicate this is not
>   specific to us or WBMC (the specific way we link to BT).
>   
>   We are currently seeing 30% less peak traffic through 21CN than last
>   month which we can only assume is down to congestion within BT's 21CN
>   network. This just shows how much of a problem this is now. We have
>   loads of spare capacity our end!
>   
>   We can only assume other users of 21CN such as BT Retail have the same
>   issue dispute their adverts apparently suggesting no slow down at peak
>   times on their new 21CN lines. We can't be sure this is the case, but
>   if not then that is a separate issue to address! We'll be interested to
>   hear from people that have access to such lines to make a comparison.
>   
>   Obviously we'll update you with any progress. A number of our customers
>   have moved to an ISP still using 20CN or asked us to move their line to
>   20CN. At present it is not simple to move lines to 20CN, but we'll have
>   to get BT to sort that if congestion on 21CN continues.
>   
>   The current technical update on this is that it is believed to be an
>   issue on the 21CN MSANs, and BT are working with the MSAN vendors to
>   resolve this. The MSAN is the equipment at the telephone exchange that
>   connects to your line. I'd like to thank those customers that are
>   helping with the testing.
>   
>   This issue is just the latest in a string of issues with BT failing to
>   scale up 21CN effectively. Most of the other issues are resolved now,
>   and hopfully this one will be too.
>   
>   Thank you all for your patience.
>   
>   Update: BT are doing tests on specific lines to test a theory and we
>   hope to hear answers soon.
>> Update: BT still working on it and trying to find cause.
>   
>   FAQ:-
>   
>   Q: Are we sure this is BT and not A&A LNS? After all the A&A LNS is new
>   isn't it?
>   A: We are sure. One of the affected lines is in our office and we have
>   tested BT and BE side by side which go to the same LNS our end and come
>   from the same laptop in our office and use the same test sites to
>   confirm throughput. We have also checked the LNS load is low as
>   expected. We have also checked that we definitely have loads of
>   headroom on the link to BT at our end.
>   
>   Q: Are we sure it is all BT 21CN ISPs not just A&A or the way A&A
>   connect.
>   A: Yes. BT have done tests connecting a line to a different endpoint in
>   BT bypassing the link to A&A and not using a WMBC link at all (WMBC is
>   the specific way we connect to BT). They saw the problem.
>> Update: After several calls with other ISPs we can confirm this is
>> definitely not just us.
>   
>   Q: Do we know how widespread it is?
>   A: Not exactly. Unlike other bank-haul congestion issues we have found
>   and had fixed by BT this is not as obvious and only shows when tested.
>   We have been trying to make quantative tests for this but have not yet
>   managed something that is unobtrustive or totally reliable. The biggest
>   indicator of the scale of the issue is that peak usage one 21CN has
>   gone down by over 30%. This suggests a lot of exchanges are not
>   affected.[IMAGE]
> 
> URL: http://aaisp.blogspot.com/2009/10/info-21cn-congestion.html
> 

I've been seeing increasing problems on 20CN. My ntp servers are
struggling to keep up with rapid temperature changes at this
time of year, then add to that quite large spikes in latency and
ntpd is struggling. It's not that I've had long periods of heavy
usage, and normally that doesn't cause such large changes as
I've had recently. Is this likely same congestion issue?

http://www.lordynet.me.uk/pub/incoming/81-187-61-69.png
http://www.lordynet.me.uk/pub/incoming/81-187-61-74.png

David

David Lord wrote:

> I've been seeing increasing problems on 20CN. My ntp servers are
> struggling to keep up with rapid temperature changes at this
> time of year, then add to that quite large spikes in latency and
> ntpd is struggling. It's not that I've had long periods of heavy
> usage, and normally that doesn't cause such large changes as
> I've had recently. Is this likely same congestion issue?
> 
> http://www.lordynet.me.uk/pub/incoming/81-187-61-69.png
> http://www.lordynet.me.uk/pub/incoming/81-187-61-74.png

I should have mentioned those graphs are obtained by sntp so
effects of network delay are larger than when using using
normal ntp.

I've today reconnected so assume swapped LNS and tonight
since 19:00 seeing slight packet loss on cqm graph.

dmldl@a

David

Dominic Hargreaves wrote:
> Alfred E Neuman  wrote:
>> What symptoms should we look out for to see whether we are affected by this 
>> problem? Should it affect maximum download rate? Should it happen more at 
>> certain times of day?
>>
>> I still seem to get pretty much my full BRAS rate (5.5M) for a download from a 
>> fast source, but web pages that require a lot of connections with small amounts 
>> of data each seem to take longer than when I first went onto 21CN. This might 
>> simply mean the sites I'm contacting have more traffic.
> 
> I'm seeing what I believe is this problem intermittently, at what I
> would call peak home user times (certainly weekday early evenings).
> 
> When the problem occurs it results in degraded download performance,
> increased latency, and sometimes frequent modem resets.

The modem reset is more likely a loss of ppp due to non response on LCP
echos. Heavily loaded lines where the congestion is apparent appear to
have this happen to.