On Sat, 31 Oct 2009 15:11:08 +0000 (UTC), Tim Woodall put finger to
keyboard and typed:

>Assuming that the gradwell mailservers are working correctly, the only
>people who can tell if chiark is refusing mail sent via
>uk-rec-cycling-moderated-request (at) usenet.org.uk are people with
>access to the logs at gradwell or chiark.

That has been verified by people with access to those logs.

>If gradwell is returning a bounce when attempting to forward to chiark
>then that is backscatter, exactly what hotmail complains about chiark
>for. (Of course, it's possible that gradwell is connecting to chiark
>before accepting the mail - but that would be a very unusual setup)

Backscatter is what happens when an intermediate server accepts mail
for an invalid destination and then returns the bounce when it tries
and fails to forward it to that destination. But that's not what's
happening here: the destination address is valid (and can be verified
as such from gradwell's servers); chiark is rejecting mail based on
the sender address after gradwell's servers make a successful
connection to chiark. In such situations, returning the bounce to the
sender is the correct response, as otherwise they'll have no way of
knowing that their mail wasn't delivered.

Mark
-- 
Blog: http://mark.goodge.co.uk
Stuff: http://www.good-stuff.co.uk

In  Mark Goodge  writes:

>On Sat, 31 Oct 2009 15:11:08 +0000 (UTC), Tim Woodall put finger to
>keyboard and typed:

>>Assuming that the gradwell mailservers are working correctly, the only
>>people who can tell if chiark is refusing mail sent via
>>uk-rec-cycling-moderated-request (at) usenet.org.uk are people with
>>access to the logs at gradwell or chiark.

>That has been verified by people with access to those logs.

And that is indeed so. I have spent the past week examining those logs and
performing experiments in conjunction with a Hotmail/Live user in order to
discover exactly what is happening.

In view of all the misunderstandings being expressed in these threads, I
think I should set out all the technicalities of the issue, as I see them.

When an email is sent from site A to site B, what happens (behind the
scenes so far as most users can see) is:

1. A makes a TCP/IP connection with B on Port 25.  Some sites B blacklist
   some IP addresses straight off, and refuse the connection.
   That does not apply to any of the parties in the present issue, except
   that if A is Chiark and B is Hotmail, then Hotmail has indeed
   blacklisted Chiark.

2. A sends B an EHLO command, to identify itself.
   Some site B reject bogus identifications at this stage.
   That does not apply to any of the parties in the present issue.

3. A sends B a MAIL FROM command, to say who it is from (or, rather, the
   address to be informed if delivery fails - known as the "Return Path",
   and not necessarily the same as the From: header in the message).
   If B is Chiark, it spots that this resolves to a Hotmail/Live
   address, but takes no action just yet.

4. A sends B a RCPT TO command to say to whom the mail is to be sent
   (usually, but not necessarily, the same as the To: header).  At this
   point B checks whether it is ready, able and willing to deliver the
   email to that address (e.g. if the address does not exist in B's
   domain, or if it is for a domain that B does not act for, then it is
   not "able"). There may be further reasons why it might not be
   "willing".

   In the case of Chiark, they operate a facility called 'SAUCE', which
   each chiark addressee may configure. It this particular addressee is
   configured as "normal", it goes back to verify the MAIL FROM address
   (see step 3 - let us call it 'C'); otherwise it proceeds to step 5.

   The verification consists of pretending to send an email to C (it sends
   an EHLO, MAIL-FROM B and RCPT-TO C). If C says "Yes, I am perfectly
   happy to accept this mail", then Chiark says (in effect) "OK, that's
   fine; I have decided not to bother sending you an email after all" (by
   sending a QUIT); it then knows that C is a genuine verified address and
   proceeds to Step 5. But, in the meantime it has caused extreme
   irritation at C, because lots of spammers use exactly this same
   technique to test out zillions of randomly chosen addressees at C to
   see if they work, and this consumes huge resources at C. So if C knows
   that B regularly uses this behaviour, then they are likely to refuse
   all such connections from B (see step 1), unless B is prepared to pay
   money to cover those wasted resources.

   BUT, if C responds that it is unable to deliver this email, then B
   (Chiark) knows that the MAIL FROM address is bogus and sends a 5xx
   rejection back to A, and thus saves bothering the addressee at B with what
   is evidently going to be a Spam.

   OR, if C refuses to accept any connection from B (see step 1) or, which
   comes to the same thing, B knows that C operates this policy and does not
   bother to try the verification, it assumes the worst and sends a rejection
   back to A in the following form:

      550 SC-002 Mail rejected by Windows Live Hotmail for policy reasons.
      The mail server IP connecting to Windows Live Hotmail has exhibited
      namespace mining behavior. If you are not an email/network admin
      please contact your E-mail/Internet Service Provider for help.
      Email/network admins, please visit http://postmaster.live.com for
      email delivery information and support

   A recives that, and takes such action as it sees fit.

5. You only arrive here if, as a result of all the foregoing, B responds
   to A that it is ready, willing and able to accept this email, whereupon
   A sends a DATA command, following by the actual email message (headers
   plus body).

6. If A has further messages for B, go back to step 3. Otherwise A sends a
   QUIT and we are done.


Now let us apply all that to an email sent by foo@live.co.uk to
uk-rec-cycling-moderated-request@usenet.org.uk. This will involve three
trips through the whole process described above, with different identities
for A, B amd C at each stage.

1. live.co.uk (aka Hotmail) determines that the MX record for
   usenet.org.uk is mail-in-1.lb.gradwell.net, so that is where it sends
   the message. A is live.co.uk, B is mail-in-1.lb.gradwell.net, MAIL-FROM
   is foo@live.co.uk (because that is who needs to be told it if bounces)
   and RCP-TO is uk-rec-cycling-moderated-request@usenet.org.uk.

   Usenet.org.uk has instructed gradwell that such mail is to be
   redirected to urcm-moderators@chiark.greenend.org.uk. So that sets up a
   second transaction:

2. Gradwell send the message to Chiark. A is now
   some-machine@.gradwell.net, B is chiark.greenend.org.uk, MAIL-FROM is
   still foo@live.co.uk and RCPT-TO is
   urcm-moderators@chiark.greenend.org.uk.

   Since urcm-moderators is SAUCEd as "normal" (it needn't be, but that is
   the moderator's choice) Chiark proceeds to verify it as above. Or,
   rather, since it knows that live.co.uk (as C) will not listen to it),
   it immediately sends the 550 rejection response, as quoted at length
   above.

   some-machine@.gradwell.net immediately sees this response, and sends a
   "bounce message", which brings us to the third transaction:

3. Gradwell contacts live.co.uk (aka Hotmail). A is now
   some-machine@.gradwell.net, B is live.co.uk, MAIL-FROM is empty
   (because no-one wants to see bounces to bounces or, worse, an infinite
   loop of bounce messages) and RCPT-TO is foo@live.co.uk (who was in the
   MAIL-FROM at transaction 1). B is perfectly willing to accept this
   message, and so it is duly sent on to foo. The body of the bounce
   message contains the full 550 response, as quoted above, so foo will
   know (if (s)he understands it) what has happened. I have verified that
   Hotmail does indeed accept these bounce messages, and pass them on to
   foo, the original sender.

So there you have it. The whole mess could easily be avoided if only the
address urcm-moderators@chiark.greenend.org.uk were not SAUCEd to do that
verification process.

That particular verification process seems, on the face of it, to be a
sensible precaution. And when it first began to be used it worked and
caught much spam. But all competent spammers know how to get around it
(e.g. by using botnets whose unwilling members' ISPs will happily confirm
that the address is genuine), whilst incompetent spammers still try to use
the same technique for dictionary attacks, for which a large juicy site
like Hotmail is eminently suitable (from their POV).

The general consensus amongst email server administrators (not Chiark
obviously) is now that this technique is counter productive and should not
be used.

And so we seem to be in a pissing contest between Hotmail and Chiark, which
Chiark is most unlikely to win, and in which the uk.* hierarchy is just
the Piggy in the Middle :-( .

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131            Web: http://www.cs.man.ac.uk/~chl
Email: chl@clerew.man.ac.uk      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

"Charles Lindsey"  wrote in
news:KsICDq.Lsu@clerew.man.ac.uk: 

> In  Mark
> Goodge  writes: 
> 
>>On Sat, 31 Oct 2009 15:11:08 +0000 (UTC), Tim Woodall put finger
>>to keyboard and typed:
> 
>>>Assuming that the gradwell mailservers are working correctly, the
>>>only people who can tell if chiark is refusing mail sent via
>>>uk-rec-cycling-moderated-request (at) usenet.org.uk are people
>>>with access to the logs at gradwell or chiark.
> 
>>That has been verified by people with access to those logs.
> 
> And that is indeed so. I have spent the past week examining those
> logs and performing experiments in conjunction with a Hotmail/Live
> user in order to discover exactly what is happening.
> 
> In view of all the misunderstandings being expressed in these
> threads, I think I should set out all the technicalities of the
> issue, as I see them. 
> 
> When an email is sent from site A to site B, what happens (behind
> the scenes so far as most users can see) is:
> 
> 1. A makes a TCP/IP connection with B on Port 25.  Some sites B
> blacklist 
>    some IP addresses straight off, and refuse the connection.
>    That does not apply to any of the parties in the present issue,
>    except that if A is Chiark and B is Hotmail, then Hotmail has
>    indeed blacklisted Chiark.
> 
> 2. A sends B an EHLO command, to identify itself.
>    Some site B reject bogus identifications at this stage.
>    That does not apply to any of the parties in the present issue.
> 
> 3. A sends B a MAIL FROM command, to say who it is from (or,
> rather, the 
>    address to be informed if delivery fails - known as the "Return
>    Path", and not necessarily the same as the From: header in the
>    message). If B is Chiark, it spots that this resolves to a
>    Hotmail/Live address, but takes no action just yet.
> 
> 4. A sends B a RCPT TO command to say to whom the mail is to be
> sent 
>    (usually, but not necessarily, the same as the To: header).  At
>    this point B checks whether it is ready, able and willing to
>    deliver the email to that address (e.g. if the address does not
>    exist in B's domain, or if it is for a domain that B does not
>    act for, then it is not "able"). There may be further reasons
>    why it might not be "willing".
> 
>    In the case of Chiark, they operate a facility called 'SAUCE',
>    which each chiark addressee may configure. It this particular
>    addressee is configured as "normal", it goes back to verify the
>    MAIL FROM address (see step 3 - let us call it 'C'); otherwise
>    it proceeds to step 5. 
> 
>    The verification consists of pretending to send an email to C
>    (it sends an EHLO, MAIL-FROM B and RCPT-TO C). If C says "Yes,
>    I am perfectly happy to accept this mail", then Chiark says (in
>    effect) "OK, that's fine; I have decided not to bother sending
>    you an email after all" (by sending a QUIT); it then knows that
>    C is a genuine verified address and proceeds to Step 5. But, in
>    the meantime it has caused extreme irritation at C, because
>    lots of spammers use exactly this same technique to test out
>    zillions of randomly chosen addressees at C to see if they
>    work, and this consumes huge resources at C. So if C knows 
>    that B regularly uses this behaviour, then they are likely to
>    refuse all such connections from B (see step 1), unless B is
>    prepared to pay money to cover those wasted resources.
> 

I think it would be fairer to say that Microsoft demand payment from 
B for a certificate to say B is not the sort of entity which would 
waste resources.  After all, B is not wasting resources and there is 
no reason why B should pay for wasted resources.  
However, in the particular case of this form of verification, it is 
hard to fault Microsoft's policy as you discuss in the rest of the 
post.


-- 
Percy Picacity

Charles Lindsey  said:



>

>
>      550 SC-002 Mail rejected by Windows Live Hotmail for policy
>      reasons. The mail server IP connecting to Windows Live Hotmail
>      has exhibited namespace mining behavior. If you are not an
>      email/network admin please contact your E-mail/Internet Service
>      Provider for help. Email/network admins, please visit
>      http://postmaster.live.com for email delivery information and
> support
>

Thankyou Charles for the full explanation.  I knew what was happeneing, now 
I see exactly how.

> 3. Gradwell contacts live.co.uk (aka Hotmail). A is now
>   some-machine@.gradwell.net, B is live.co.uk, MAIL-FROM is empty
>   (because no-one wants to see bounces to bounces or, worse, an
>   infinite loop of bounce messages) and RCPT-TO is foo@live.co.uk
>   (who was in the MAIL-FROM at transaction 1). B is perfectly willing
>   to accept this message, and so it is duly sent on to foo. The body
>   of the bounce message contains the full 550 response, as quoted
>   above, so *foo will know (if (s)he understands it)* what has
>   happened. I have verified that Hotmail does indeed accept these
>   bounce messages, and pass them on to foo, the original sender.

And there's the huge problem.

If foo is a hotmail user with little knowledge of all this - rather like 
me - and gets this message, will they understand it?  foo connected to Live 
when they sent the email, so if I was foo I would think they were telling me 
that /I/ was the one exhibiting this behaviour.  The bounce message tells me 
to contact live.  They no doubt would come back with a technical message I 
struggled to understand, and then I have to figure out, assuming by then I 
care, what to do next.

Sure, there may be some sort of explanation somewhere but just now all I 
know is what is on the charter page.


-- 
kat
   >^..^<

On Tue, 3 Nov 2009 08:45:49 +0000 (UTC), Percy Picacity put finger to
keyboard and typed:

>
>I think it would be fairer to say that Microsoft demand payment from 
>B for a certificate to say B is not the sort of entity which would 
>waste resources.  After all, B is not wasting resources and there is 
>no reason why B should pay for wasted resources.  
>However, in the particular case of this form of verification, it is 
>hard to fault Microsoft's policy as you discuss in the rest of the 
>post.

In this case, B is wasting resources. Or, more to the point, it is
demonstrating behaviour typical of a spammer. It is, therefore,
blocked from Hotmail, and Microsoft offers two options for removing
the block:

1. Stop acting like a spammer.

2. Pay us money to prove you are not a spammer.

Microsoft's policy may appear rather harsh, but it's not at all
unreasonable in context.

Mark
-- 
Blog: http://mark.goodge.co.uk
Stuff: http://www.good-stuff.co.uk