gmane.ietf.rfc822

draft-kucherawy-received-state

Murray S. Kucherawy — 2012-01-09T22:44:07

Hi all,

Over on ietf-smtp, https://datatracker.ietf.org/doc/draft-kucherawy-received-state/ was developed late last year.  This is an optional tweak to Received fields allowing annotation of entry into special states of mail handling, such as quarantines or hold-for-moderation MLM actions that would help to explain large gaps in timestamp sequences.

I'm looking for a wider audience of reviewers and (hopefully supporters).  If this sort of thing seems like a good (or terrible) idea, please do follow up and comment.

How much and what kind of support is evident will guide the choice of what its next steps are (if any).

Thanks,
-MSK

draft-kucherawy-received-state

Murray S. Kucherawy — 2012-01-09T22:44:07

Hi all,

Over on ietf-smtp, https://datatracker.ietf.org/doc/draft-kucherawy-received-state/ was developed late last year.  This is an optional tweak to Received fields allowing annotation of entry into special states of mail handling, such as quarantines or hold-for-moderation MLM actions that would help to explain large gaps in timestamp sequences.

I'm looking for a wider audience of reviewers and (hopefully supporters).  If this sort of thing seems like a good (or terrible) idea, please do follow up and comment.

How much and what kind of support is evident will guide the choice of what its next steps are (if any).

Thanks,
-MSK

Re: Do S/MIME and OpenPGP protect message headers?

Dave CROCKER — 2011-12-28T16:12:33


On 12/28/2011 7:27 AM, Rolf E. Sonneveld wrote:


However, of course, the contained message then ceases to be treated as a normal 
message by the receiving user agent.  It's just content.

Although not it's primary task, data integrity protection /is/ provided for 
listed header fields by DKIM.  (The signature lists the fields it covers. This 
typically does include From:, To:, and cc: and Subject.)

d/

Do S/MIME and OpenPGP protect message headers?

Rolf E. Sonneveld — 2011-12-28T15:27:42

Hi,

just to check: S/MIME (http://tools.ietf.org/html/rfc5751) does not  
does include/cover the 5322.From and the 5322.To fields as part of the 
cryptographic payload. Protection of headers can be achieved by wrapping 
up a the complete message into a message/rfc822 bodypart and 
sign/encrypt that (par. 3.1 of that RFC). How is that with OpenPGP 
(http://tools.ietf.org/html/rfc4880)? OpenPGP does not protect 5322.From 
and 5322.To either, does it?

Par. 5.11 of RFC4880:

    A User ID packet consists of UTF-8 text that is intended to represent
    the name and email address of the key holder.  By convention, it
    includes anRFC 2822  <http://tools.ietf.org/html/rfc2822>  [RFC2822  <http://tools.ietf.org/html/rfc2822>] mail name-addr, but there are no
    restrictions on its content.  The packet length in the header
    specifies the length of the User ID.


The recipient address is not mentioned anywhere. So is the following 
statement correct?:

Neither S/MIME nor OpenPGP protects the '5322.headers' of a message, 
unless that message itself is wrapped up and used as body of an 
enclosing S/MIME or PGP message

/rolf

Privicons and redaction

Alessandro Vesely — 2011-12-09T12:18:22

In abuse reporting, an established albeit discouraged practice
consists of redacting reported messages.  In a nutshell, users may
have the ability to signal some received messages as being abusive.
Their mailbox providers should then wrap those messages according to
abuse reporting format (ARF, RFC 5965) and send them back to the
senders if they had subscribed to the relevant feedback loop.  See
http://en.wikipedia.org/wiki/Feedback_loop_%28email%29

Redaction conceals the recipient, rather than the sender.  In this
case, authors should markup the parts of the messages that contain
sensible data, in case the recipients report those messages as spam,
possibly inadvertently.  A document describing how to carry out
redaction is now in LC, so it can hardly mention privicons.  However,
privicons could consider this possible use.  The redaction doc is at
http://datatracker.ietf.org/doc/draft-ietf-marf-redaction/

For casual readers, privicons are defined here:
http://tools.ietf.org/html/draft-koenig-privicons

draft-koenig-privicons-03.txt

Bill McQuillan — 2011-12-05T20:54:06

My comments on "Privacy Preferences for E-Mail Messages"

   .
   .
   .

It would be more correct to use the term "header field" rather 
than "header" here and in the rest of the document.
   .
   .
   .

   .
   .
   .

The definition of Sending User needs to be clarified: is it the 
email-address from the Sender: header field? Or one or more of 
the email-address from the From: header field?
   .
   .
   .
   .
   .
   .

This "MUST" seems to conflict with the MAY later in this spec.
   .
   .
   .

These definitions need to be formatted to more clearly 
distinguish the term from its definition. Perhaps like the 
"privicon" definition.


The definition of "first line" is not at all clear except for a 
TEXT/PLAIN body.

In the case of a TEXT/HTML body, the intent seems to be that the 
first line is the topmost "line" visible to the Receiving User. 
However, since the "first line" privicon is the determining 
specification in the case of disagreements and the MUA is 
expected to "enforce" this, the MUA must be able to *find* the 
privicon. This could be difficult for the MUA with sophisticated 
HTML code.

For bodies such as IMAGE/*, AUDIO/*, VIDEO/*, MODEL/*, or 
APPLICATION/* how is the "first line" to be found?

In the case of a MESSAGE/* the first line would be part of the 
embedded message's header and would probably cause a parse error.

In the case of MULTIPART/MIXED the first line would presumably be 
the "first line" of whatever the first embedded part is. And in 
the case of MULTIPART/ALTERNATIVE the first line would have to be 
the "first line" of each embedded version with a method of 
resolving conflicts there, too.

The remaining MULTIPART/* type bodies would have similar issues.


The use of MUST here without specifying the required text is 
unenforceable and is better expressed with a SHOULD.
   .
   .
   .

As mentioned above this MAY conflicts with the MUST earlier in the spec.


I find the requirement of a single space character following the 
colon to be objectionable, this is not Usenet! There should also 
be provision for a comment or explanatory text following the 
privicon.
   .
   .
   .

The preceding paragraph seems out of place.


This example seems to imply that the "firstLine" is the first 
non-blank line, does this also apply to the body "first line"?


By "Footnote" here do you mean "Footer"?
   .
   .
   .
   .
   .
   .

It is not clear what privacy issue is being addressed with the 
"Don't print" privicon. If the Receiving User can more easily 
keep a message private by printing it, locking it in a safe, and 
deleting the electonic copy, why would the Sending User object?

What danger does the Sending User prevent with this option?
   .
   .
   .

The preceding paragraph is not clear. Do the privicons "OR" 
together or replace the first with the second under some 
conditions?


This table is not symmetrical, so which is the first privicon and 
which is the second? Otherwise, explain why [/] and [o] cannot be in 
either order.
   .
   .
   .

The first use of the word "header" in the preceding paragraph is 
correct, but the second is not!
   .
   .
   .

This example places the (presumably first body line) privicon as 
part of the message header and thus will cause a syntax error and 
likely make the privicon invisible to the Receiving User.
   .
   .
   .

If the following instructions are OPTIONAL, the use of 12 "MUST"s 
in them is hard to justify.
   .
   .
   .

It seems to me that the list of *all* of the From: addresses, the 
Sender: address, and the To: addresses should be considered as 
non-third party addresses since they presumably are all aware of 
the message already.
   .
   .
   .

The preceding paragraph is not clear.
   .
   .
   .

Where is the option to tell the MUA "Do NOT delete, and never 
bother me again."?


Where is the option to tell the MUA "Do NOT delete, and never 
bother me again."?


Where is the option to tell the MUA "Do NOT delete, and never 
bother me again."?


Esactly how can the MUA determine in the *body* that the Sending 
User identity has been removed?
   .
   .
   .

Should the Sending User and all Receiving Users be considered as 
added to "internal" automatically for the purposed of this reply 
of forward?
   .
   .
   .

It may be appropriate to forward the privicons of the original 
message, optionally.

[Fwd: [apps-discuss] Call for adoption of draft-kucherawy-mta-malformed-00.txt as an APPSAWG document]

Alexey Melnikov — 2011-04-20T21:45:29

FYI.

Re: Comments on Malformed Message BCP draft

Hector Santos — 2011-04-20T14:38:21

Keith Moore wrote:



I don't think I was Keith.  I am firm believer all transactions should 
be treated equally when it comes to following standards with a "wait 
for someone to complain" exception.    I was pointing out dealing with 
compromising user issues is a different problem with a different set 
of solutions and I believe relaxing standards does not help with the 
problem.  So even if you wanted to use a negative MUA behavior change 
red-flag as part of a local policy compromised user monitor, you 
couldn't.


For offline MUAs, I generally agree, at least for the things we often 
are concern about. For Online MUAs, there are improvements, including 
support for List-IDs as done with gmail.com

Re: Comments on Malformed Message BCP draft

Keith Moore — 2011-04-20T13:19:19

On Apr 20, 2011, at 5:55 AM, Hector Santos wrote:


Please don't propose to penalize users for switching MUAs.   It's difficult enough for people to do as it is, and that is part of why MUAs are so dysfunctional and have evolved so little over the past 10+ years.   

Keith

Re: Comments on Malformed Message BCP draft

Hector Santos — 2011-04-20T09:55:48

Murray S. Kucherawy wrote:

Isn't this a different problem - compromised users?

When there is a compromised MSA user, there are other more serious 
issues to address - like detecting by whatever means and lock the 
user/machine out immediately.   It could be stolen user credentials or 
machine compromised for users who are still IP based authorized by 
their ISP network.  I seriously doubt the compromised user/machine is 
going to use the same user MUA to blast spam, it could, but then the 
user can see that something is wrong.

This gives more weight to the idea to not tolerate standards 
violations. Relaxing on standards makes it more difficult to use a 
delta in MUA compliance behavior as good or bad.

Re: Comments on Malformed Message BCP draft

Keith Moore — 2011-04-20T04:35:59


On Apr 20, 2011, at 12:29 AM, Murray S. Kucherawy wrote:


The only way I see to deal with these cases is for downstream relays or MXes to bounce or even drop malformed messages.  That will weed out those (hopefully few) MSAs.  

(Bouncing such messages won't get many users to change their MUAs.   But if their mail doesn't get through, they will change MSPs.  Many users have multiple email accounts anyway, and "resending from a different email address" is a common retry strategy.)

Keith

RE: Comments on Malformed Message BCP draft

Murray S. Kucherawy — 2011-04-20T04:29:10


Where the MSA is under the control of someone that wants to exploit browser or MUA weaknesses, or doesn't care about standards as long as his/her crap gets to the inbox, doesn't that view of the world means there's no hope for improvement?

Re: Comments on Malformed Message BCP draft

Keith Moore — 2011-04-20T04:24:41

I strongly suspect that the sender's response would be to:
a) find another mail service provider (even at the expense of getting another email address), and/or
b) stop sending mail to any recipients for whom such bounces happened with any regularity
...neither of which actually fixes the problem (except for providers that provider their own user agents)

People these days are so used to dealing with spam and error messages that they don't understand, that the vast majority of them won't even bother to read such messages. As soon as they realize it's not intended for them _personally_ , the trigger finger reflexively hits the delete key. The user may indeed be annoyed, but the chance that he'll take useful corrective action is almost zero.

If we want such measures to work, we need to find a way to either
a) send the notifications to the vendors/providers of such user agents, or
b) arrange for them to get triggered only when the user starts using a new user agent, so he'll immediately associate the user agent with the problem whether or not he reads the message.

a) can perhaps be accomplished more efficiently in other ways than by annoying users, e.g. by providing a mail message test suite, advertising it widely, and inviting vendors to test their products with it.

b) can perhaps be accomplished by having the MSA notice when the user starts using a new user agent, and only then, notifying the user that his messages from said new user agent are nonconforming.

That's not to say that I want MSAs to forward thoroughly broken messages; I don't. But once a message leaves the MSA, I think the opportunity to provide any kind of useful feedback to the submitter has probably passed.

Keith

Re: FW: Comments on Malformed Message BCP draft

Charles Lindsey — 2011-04-19T20:25:07





Yes, but there are other possibilities (yes, 'fixing' is to be avoided).
But for suitable messages (for some meaning of "suitable") you could send
a bounce back up the return path of the following form:

"Your message is non-compliants for reasons <blah>. Although we have
attempted to deliver it, you need to be aware that it may fail to reach
its intended recipient, and we would suggest that you endeavour to make
future messages compliant with the relevant internet standards <blah list
of standards blah>."

That way, the message (probably) gets through, but the sender gets pissed
off sufficiently to complain to his provider, threatening to vote with his
feet, etc. etc.

So this piles pressure on providers to adhere to standards (a Good Thing)
but still allows the mail to get through (also a Good Thing).

Re: [apps-discuss] Comments on Malformed Message BCP draft

Douglas Otis — 2011-04-18T22:56:08

On 4/18/11 5:48 AM, John C Klensin wrote:
Fully Agree.

-Doug

Re: Comments on Malformed Message BCP draft

Hector Santos — 2011-04-18T18:00:09

Keith Moore wrote:


+1

The principle applies in many areas, but overall, it is most 
usefulness is when there existed greater uncertainty.


+1

and I think intent is also related to trust or being non-anonymous.

Exceptions are a fact of life.  If you are unknown, then you get on 
the back of line like everyone else. But if you are known ("on the 
special party list"), you get special treatment and can avoid standing 
on a line.  Its always possible to learn that you are not a 
troublesome, a regular party dude to get a pass, but its all still 
about be recognized.

After 25+ years, the uncertainly principle for relaxation protocols 
should no longer be a quick out. There is simply no excuse today to 
continue tolerating violations of protocols or relaxing it for all. 
Exceptions to the rule of thumb has to be about trusted exceptions - 
not risky exceptions.

Re: Comments on Malformed Message BCP draft

Hector Santos — 2011-04-18T15:44:50

Dave CROCKER wrote:


Paraphrasing Shaw, telling the truth is the funniest joke in the world.

I would like to suggest a reality which it is closer to:

     Receive-side workaround exceptions

and not the rule of (protocol) thumb.

The reason is because most systems are compliant today, especially 
among actively supported software and when there is an issue raised, 
only good guys complain/report, not the bad guys. Hence most problems 
are generally fixed. The historical reality has been the anonymous bad 
guy exploitation of protocol relaxations which has proven to the cusp 
of the highly expensive mail abuse problem.

What we are really talking about here with workaround exceptions, is 
sender trust, in which case, it is not longer anonymous and the 
protocol relaxations begin to make more sense.   But anonymous abuse 
SHOULD NOT be tolerated and continue to be an excuse to violate 
protocol. There are some strong compliancy parts of protocol which has 
certainly shown a practical reality for a reject first high payoff, 
feasible efficiency to help against anonymous unsolicited abuse.

Of course, as always, the devil is in the details.

Re: [apps-discuss] Comments on Malformed Message BCP draft

Hector Santos — 2011-04-18T15:05:24

John C Klensin wrote:

+1

You can see one common issue the malformed message I-D shows as an 
example with the field-name having a space:

    "field-name :"

This would be akin to the SMTP space syntax debates, with the example:

    Mail From:space<address>

Recognizing this long time issue, this note was added to RC5322 
(Section 3.3)

    Since it has been a common source of errors, it is worth noting that
    spaces are not permitted on either side of the colon following FROM
    in the MAIL command or TO in the RCPT command.  The syntax is exactly
    as given above.

Implementators are going to decide on whether they will accept the 
illegal form with the space all depending on the severity of 
frequency.  In the past, the #1 MUA was the offending problem so while 
we didn't encourage relaxation, it had to be noted.

IMO, I don't think this a similar consideration applies to similar 
5322 header syntax errors.  It was reasonable to see why the SMTP 
systems had to consider relaxing it due to a wide world major 
deployment of the offending software.   We don't have that here with 
RFC 5322 headers, do we?

Re: [apps-discuss] Comments on Malformed Message BCP draft

Hector Santos — 2011-04-18T14:44:16

Tony Finch wrote:

+1.

My view is that since only Good Guys complain, Bad Guys do not, there 
are some well established areas for strong compliancy checks that has 
proven to have a high efficiency bad mail filtering rate with little 
false positives for good guys. This is because most good guys are 
compliant, especially for these particular areas of compliancy. 
Others are more subjective, so it generally it becomes a decision to 
how its enabled - on/off by default.

Traditionally, the transport is a passive payload transfer system but 
has have options for decisions than it does in applying subjective 
decisions related to the payload.

I think good examples are:

SMTP LEVEL:

   - anal strong syntax checks, i.e, "MAIL FROM:space"

   - Checking for explicit EHLO [ip literal] matching connection IP
     requirement. Exceptions used for growing NATS related issue
     in SOHO market with more than 1 PC.  PORT 587 can be used
     to skip this check.

   - Return path verification - methods in debate, but it is
     increasingly recognized that the validity requirement
     has become more important in the area of bounce avoidance.

822/2822/5322 PAYLOAD LEVEL

The first thing here is making sure we have a valid message per RFC 
822/2822/5322.  I listed all three because it depends on the receiver 
on how it  determine a valid message.

First, many systems don't need any RFC headers at all.  Just the body 
and will automatically fill in required RFC fields:

    From:          <--- SMTP Return Path
    To:            <--- SMTP Forwarding path
    Date:          <--- Reception Timestamp

822 systems require From:, To/Cc:, Date:, 2822 systems relaxed it to 
From:, Date and it remains this way with 5322 systems.  I recall 
reading long ago an I-D by Keith Moore to even further relax it to 
remove the From: requirement.  That would of been terrible.

Anyway, when some of these are missing or none, the design question 
becomes whether the missing headers should be filled in or rejected, 
always or sometimes.  For some, the answer was straight forward, for 
others it was not.

I will note that with RFC2821bis/RFC2822bis, we explored enabling 
stronger compliance (default on) and almost immediately, we got a few, 
not a lot, reports and that was enable the default to a FILL IN to 
avoid any future support overhead.   I believe it was 2-3 reports 
only, and I recall one report because the issue here was scenarios 
where two identical list messages (by content and context) for two 
users at a host and one fail and the other did not was found to be 
because both took different mail processing distribution paths.

I honestly don't recall the details of the incorrect headers, but the 
main thing was how the differences occurred.  Following the trace of 
both, it was clear the sender was using two path distributions, where 
one path was using  the original non-compliance software path and the 
the second was a new beta software path (I recall the "Beta" term in 
Received lines).  So it appeared the sender was aware of the 
distribution problem and was migrating to more RFC5322 compliant 
software.  Of course, while this is all interesting to the customer, 
he could care less and wanted a solution now.  I forget what was done 
here.

Anything beyond this has not really been a big issue for us. Some of 
the examples listed in the report are interesting to note, but I can't 
see a real justification to add time and energy to automate 
corrections when the most practical reality by far is majority of the 
good guys are compliant.  If an issue does occur, and the issue is 
related to broken legacy software, then you have no choice by to help 
solve the problem.  But if the offending software is active, then you 
have to weigh all the factors on how to feasibly serve the customer. 
Severity and frequency of problem. It is more than one, is the 
offending active vendor aware and have solution you can research. Does 
the customer have access to the offending senders?  Can you provide a 
work around, etc, etc, etc.

Just consider the report malformed message scenarios with the space 
for colon, this will require you to change YOUR software, not trivia 
for everyone, when parsing RFC headers to one that checks for two 
conditions:

     "field-name:"
     "field-name :"

I am not sure I really want to do that.  It will fall into the same 
engineering compliance debates with SMTP space issues:

     Mail From:<address>
     Mail From: <address>

The difference here is that it was well known that the #1 MUA Outlook 
was doing this and most systems could not ignore that fact.   Today, I 
believe these MUAs were corrected and you don't expect it, so you can 
probably get away (to a higher degree) with enforcing the syntax.

But do we have such a high level of the malformed 5322 header with a 
space in it, in particular from any big brand vendor?  I'm not seeing 
this.

Re: Comments on Malformed Message BCP draft

Dave CROCKER — 2011-04-18T14:36:58



On 4/15/2011 7:09 PM, ned+ietf-822< at >mrochek.com wrote:


Receive-side workarounds, rather than rejections of protocol/format violations, 
seems to be an inherent state of affairs for apps protocols.  No one likes that 
fact, but no one has come up with a way to change the reality.

d/

RE: Comments on Malformed Message BCP draft

2011-04-16T02:09:26


More like "almost always". I estimate we average one complaint that turns out
to be message maformation every week or two, and it's been this way for almost
two decades. But despite the fact we as a matter of policy *always* push back
saying "fix the actual problem", I can't recall more than a handful of cases
where this resulted in the actual problem being tracked down and fixed.
Workarounds are a lot easier than, say, getting a printer manufacturer
to update their firmware so it doesn't produce this content-type header:

    context-type: 'image/tiff'

(Yes, that's an actual example from a few weeks back, and also note that while
it is clearly maformed, it's syntactically legal.)

Additionally, the various options we provide for stricter message checking are
among the few options we've implemented that AFAIK are essentially never used
by anyone, ever.


I agree, although I suspect getting consensus on this is going to be difficult.

Ned