gmane.ietf.imapext

qresync WG adoption call

Dave Cridland — 2013-05-20T18:20:26

Folks,

We now have two drafts by Alexey published:

 - draft-melnikov-rfc4551-bis-00 (CONDSTORE)
 - draft-melnikov-5162bis-00 (QRESYNC)

I'd like to propose that the working group adopt these as our start point.

Therefore this constitutes a call for adoption of both documents; reviews
would be very useful at this stage and I'll prod people if there's no
volunteers.

Thanks,

Dave. (As qresync co-chairman)
_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

GMail's extensions: X-GM-THRID immutability

Jan Kundrát — 2013-05-09T16:27:43

Hi,
this might be a bit OT for this mailing list, but given that GMail is a rather dominant player among the IMAP service providers and due to the usefulness of having data available to other, I'm posting here anyway.

- It isn't clear to me what the immutability guarantees of the returned data of the X-GM-THRID [1] extension are. It would be terrific if it was safe to cache these values in the same manner as any other immutable per-message data, but I suspect this is not possible -- if the threading is "right", delivery of any new message could potentially join independent threads together, thereby changing the thread IDs. How is GMail handling this? How is the threading computed?

- Any plans to support the THREAD=REFS [2] in GMail?

- Any plans for support of the SORT extensions?

- Any plans for CONTEXT=SORT and CONTEXT=SEARCH? :)

The reason why I'm asking is because I'd like to enable threading for Trojita's users who are accessing GMail. Any threading algorithm apart from THREAD=REFS has significant limitations without a server-side sorting simply because old threads with new arrivals are not listed as recent. The only remedy is giving up and hoping that UIDs somehow correlate with the age of a any message, which is rather crude heuristic.

With kind regards,
Jan

[1] https://developers.google.com/google-apps/gmail/imap_extensions#access_to_the_gmail_thread_id_x-gm-thrid
[2] http://tools.ietf.org/html/draft-ietf-morg-inthread-01

qresync WG kick-off

Dave Cridland — 2013-05-08T15:29:59

As chairman.

Just so everyone's on the same page, the plan for the working group is
essentially as follows:

- Alexey's putting together starting drafts, which we'll be looking to
adopt.
- Once those drafts are out, we'll need reviewers on them to gauge
what milestones we should have - we currently just have a publication
request milestone of December; Eliot and I suspect this could be moved
much sooner if we feel that the issues are well-known.
- We're not intending meeting. This includes online meetings - all
discussion will therefore be solely on this list.


Just to refresh your memories, all discussion on this list (working-
group related or not) is covered by the Note Well; if you're unclear
what your responsibilities are under this, please ask.


Sent with [inky: <http://inky.com?kme=signature>]

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

WG Action: Formed IMAP QRESYNC Extension (qresync)

The IESG — 2013-04-29T17:39:20

A new IETF working group has been formed in the Applications Area. For
additional information please contact the Area Directors or the WG
Chairs.

IMAP QRESYNC Extension (qresync)
------------------------------------------------
Current Status: Proposed Working Group

Chairs:
  Dave Cridland <dave< at >cridland.net>
  Eliot Lear <lear< at >cisco.com>

Assigned Area Director:
  Barry Leiba <barryleiba< at >computer.org>

Mailing list
  Address: imapext< at >ietf.org
  To Subscribe: https://www.ietf.org/mailman/listinfo/imapext
  Archive: http://www.ietf.org/mail-archive/web/imapext/

Charter of Working Group:

The Internet Message Access Protocol (IMAP), defined in RFC 3501,
specifies a protocol for accessing email messages on a server that
implements a message store from a client. It also includes commands for
manipulating the message store -- creating, deleting, and renaming
mailboxes, adding a message to a mailbox, and copying messages from one
mailbox to another.

Base IMAP as described in RFC 3501 requires that in order to discover
flag changes and expunged messages in a mailbox, the client has to fetch
flags for all messages it knows in the mailbox and compare returned
results with its own state.

This can generate a significant amount of traffic for big mailboxes. The
IMAP CONDSTORE extension (RFC 4551) provides a facility for IMAP clients
to quickly resynchronize mailbox flag changes in a mailbox. The IMAP
QRESYNC extension (RFC 5162) extends CONDSTORE to also cover expunged
messages, and reduces the number of round trips needed to resynchronize
by extending the SELECT/EXAMINE command.

The CONDSTORE and QRESYNC extensions are deployed in both clients and
servers.  These deployments have exposed errors and clarity issues in
the specifications, and they need correcting.  The IMAP QRESYNC
Extension working group has the task of updating CONDSTORE and QRESYNC
extensions on the Standards Track.

The working group might produce one (combined) or two separate documents
(as now) updating these extensions. The working group will review errata
and update the documents as needed to incorporate those, and will
correct significant errors and inconsistencies, but will keep changes at
this stage to a minimum and avoid incompatible changes.

No other IMAP extension work is in scope for this working group.

Milestones:
  Dec 2013 - Request publication on updated qresync/condstore spec(s)


_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Biggest Fake Conference in Computer Science

2013-04-27T16:51:04

Biggest Fake Conference in Computer Science


We are researchers from different parts of the world and conducted a study on  
the world’s biggest bogus computer science conference WORLDCOMP 
( http://sites.google.com/site/worlddump1 ) organized by Prof. Hamid Arabnia 
from University of Georgia, USA.


We submitted a fake paper to WORLDCOMP 2011 and again (the same paper 
with a modified title) to WORLDCOMP 2012. This paper had numerous 
fundamental mistakes. Sample statements from that paper include: 

(1). Binary logic is fuzzy logic and vice versa
(2). Pascal developed fuzzy logic
(3). Object oriented languages do not exhibit any polymorphism or inheritance
(4). TCP and IP are synonyms and are part of OSI model 
(5). Distributed systems deal with only one computer
(6). Laptop is an example for a super computer
(7). Operating system is an example for computer hardware


Also, our paper did not express any conceptual meaning.  However, it 
was accepted both the times without any modifications (and without 
any reviews) and we were invited to submit the final paper and a 
payment of $500+ fee to present the paper. We decided to use the 
fee for better purposes than making Prof. Hamid Arabnia (Chairman 
of WORLDCOMP) rich. After that, we received few reminders from 
WORLDCOMP to pay the fee but we never responded. 


We MUST say that you should look at the above website if you have any thoughts 
to submit a paper to WORLDCOMP.  DBLP and other indexing agencies have stopped 
indexing WORLDCOMP’s proceedings since 2011 due to its fakeness. See 
http://www.informatik.uni-trier.de/~ley/db/conf/icai/index.html for of one of the 
conferences of WORLDCOMP and notice that there is no listing after 2010. See Section 2 of
http://sites.google.com/site/dumpconf for comments from well-known researchers 
about WORLDCOMP. 


The status of your WORLDCOMP papers can be changed from scientific
to other (i.e., junk or non-technical) at any time. Better not to have a paper than 
having it in WORLDCOMP and spoil the resume and peace of mind forever!


Our study revealed that WORLDCOMP is a money making business, 
using University of Georgia mask, for Prof. Hamid Arabnia. He is throwing 
out a small chunk of that money (around 20 dollars per paper published 
in WORLDCOMP’s proceedings) to his puppet (Mr. Ashu Solo or A.M.G. Solo) 
who publicizes WORLDCOMP and also defends it at various forums, using 
fake/anonymous names. The puppet uses fake names and defames other conferences
to divert traffic to WORLDCOMP. He also makes anonymous phone calls and tries to 
threaten the critiques of WORLDCOMP (See Item 7 of Section 5 of above website). 
That is, the puppet does all his best to get a maximum number of papers published 
at WORLDCOMP to get more money into his (and Prof. Hamid Arabnia’s) pockets. 


Monte Carlo Resort (the venue of WORLDCOMP for more than 10 years, until 2012) has 
refused to provide the venue for WORLDCOMP’13 because of the fears of their image 
being tarnished due to WORLDCOMP’s fraudulent activities. That is why WORLDCOMP’13 
is taking place at a different resort. WORLDCOMP will not be held after 2013. 


The draft paper submission deadline is over but still there are no committee 
members, no reviewers, and there is no conference Chairman. The only contact 
details available on WORLDCOMP’s website is just an email address! 

Let us make a direct request to Prof. Hamid arabnia: publish all reviews for 
all the papers (after blocking identifiable details) since 2000 conference. Reveal 
the names and affiliations of all the reviewers (for each year) and how many 
papers each reviewer had reviewed on average. We also request him to look at 
the Open Challenge (Section 6) at https://sites.google.com/site/moneycomp1 


Sorry for posting to multiple lists. Spreading the word is the only way to stop 
this bogus conference. Please forward this message to other mailing lists and people. 


We are shocked with Prof. Hamid Arabnia and his puppet’s activities 
http://worldcomp-fake-bogus.blogspot.com   Search Google using the 
keyword worldcomp fake for additional links.

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

GENURLAUTH pipelining

Jan Kundrát — 2013-04-17T15:03:11

Hi,
I've noticed that my code for GENURLAUTH is rather dumb and also quite buggy -- it breaks when there are multiple GENURLAUTH commands in flight (yes, it's a serious bug, users sending multiple messages at once could send wrong ones to wrong senders). What I need to know is a reliable way of mapping the returned URLs into the commands which issued them. I suspect this might not be an easy problem to solve, though. Unlike ESEARCH, the GENURLAUTH contains no tag information to "tie" it back to the issuing command, so I think that one should either disable pipelining of GENURLAUTH altogether, or somehow guess the correct bidning based on the received URLs.

One could rely on the strict wording of RFC 4467's Chapter 5:

An initial URL is built, ending with ";URLAUTH=<access>" but without
the ":<mech>:<token>" components. An authorization mechanism is
selected and used to calculate the authorization token, with the
initial URL as the data and a secret known to the IMAP server as the
key. The URLAUTH-authorized URL is generated by taking the initial
URL and appending ":", the URL authorization mechanism name, ":", and
the ASCII-encoded hexadecimal representation of the authorization
token.

This means "ignore any URLs which do not begin with the string I sent". However, I can imagine all sorts of transformations a server might want to perform on the URLs they hand out -- the most trivial is case mangling for usernames/hostnames/mailboxes, but there are even harder-to-detect, yet legitimate (?) transformations like using a real FQDN for a server's hostname.

Apart from that, there's also the possibility of some semi-evil mangling like providing a different UID/UIDVALIDITY/mailbox triplet which might make a certain level of sense when e.g. dealing with virtual mailboxes like "[All messages]".

What is your advise here?

With kind regards,
Jan

rfc4551bis conversion from text to XML

Alexey Melnikov — 2013-03-07T10:51:00

Is there a kind soul that can convert my textual copy of rfc4551bis to 
XML? I can guarantee an acknowledgment and a beer (or an alternative 
drink of your choice) at one of IETF meetings.

Also, if people can point me to a tool that might help with the 
conversion process, that would be awesome as well.

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Proposal for a new IMAP Working Group to revise CONDSTORE& QRESYNC

Alexey Melnikov — 2013-03-05T14:57:25

Hi,
I've noticed some surge in activity related to implementing CONDSTORE 
(RFC 4551)/QRESYNC (RFC 5162) in both clients and servers, so I drafted 
a new WG charter proposal to update/clarify these:

-----------
The Internet Message Access Protocol (IMAP), defined in RFC 3501,
specifies a protocol for accessing email messages on a server
that implements a message store from a client. It also includes
commands for manipulating the message store -- creating, deleting, and
renaming mailboxes, adding a message to a mailbox, and copying
messages from one mailbox to another.

Base IMAP as described in RFC 3501 requires that in order to discover
flag changes and expunged messages in a mailbox, the client has to
fetch flags for all messages it knows in the mailbox and compare returned
results with its own state. This can generate a significant amount of
traffic for big mailboxes.

The IMAP CONDSTORE extension (RFC 4551) provides a facility for IMAP
clients to quickly resynchronize mailbox flag changes in a mailbox.
The IMAP QRESYNC extension (RFC 5162) extended CONDSTORE to also
cover expunged messages and reduced the number of round trips needed
to resynchronize by extending the SELECT/EXAMINE command.
Both extensions seen deployment in both clients and servers.
These deployments has exposed errors and clarity issues in the
specifications, which need correcting.

The IMAP QRESYNC extension (imapqresync) working group has the task
of updating CONDSTORE and QRESYNC extensions as Proposed Standards.
The working group might produce one (combined) or two separate
documents (as now) updating these extensions. The working group will
review errata and update the documents as needed to incorporate those,
and will correct significant errors and inconsistencies, but will keep
changes at this stage to a minimum. In the event that implementation
experience would dictate an incompatible change to one (or both) of
these extensions, the corresponding IMAP capability identifiers would
be changed.

No other IMAP extension work is in scope for this working group.
---------

So please comment on whether you think this work is worth doing and on 
the proposed charter text.

I've also posted a new version of RFC 5162bis 
(http://tools.ietf.org/html/draft-melnikov-5162bis-01), which addressed 
most (but not all) of existing errata on the document. If a new WG is to 
be formed, I propose that the document is used as one of the initial 
documents.

Thank you,
Alexey

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Accepting an "ESORT" response instead of an ESEARCH

Jan Kundrát — 2013-02-25T14:59:58

Hi,
it turned out that a certain IMAP server sends out non-standard "ESORT" responses in response to the ESORT command (the correct way is to respond with an ESEARCH). I've filed a bugreport with an attached patch which hopefully fixes the issue, but even if they accept it soon, there's the usual problem with releases taking time and sysadmins delaying updates etc, and in the end, it's always the client which gets the blame "because other MUAs work just fine". The server the reporter is using was released roughly 10 months ago; I have unfortunately no idea about when they started advertising the ESORT capability, the publicly available git mirror doesn't go that far back in time.

Do you guys see any problem with a client simply accepting "* ESORT ..." in the same way and using the same syntax as an ESEARCH response? Would you leave such a workaround in your codebase indefinitely, or remove it after some time period?

With kind regards,
Jan

IMAPMOVE wrapping up

Barry Leiba — 2013-01-16T23:42:54

Alexey has put the current information about implementations into the
working group wiki:  http://tools.ietf.org/wg/imapmove/trac/wiki

This completes the last chartered milestone (a few weeks early,
actually), and the Secretariat has marked that milestone as done:
http://datatracker.ietf.org/wg/imapmove/charter/

Additionally, the MOVE specification, which is in the RFC Editor
queue, is now in AUTH48 state -- the final editing and author review
before it's published, which means that it should be published as an
RFC within the next week or two.

As soon as the RFC is published, I will ask the Secretariat to
formally close down the working group.  I want to take this
opportunity to send out some thanks to

* Arnt for taking the lead on the document,

* everyone who reviewed, commented, and implemented,

* Alexey and Ned for chairing (and Ned for doing editing while Arnt
was under the weather).

I consider this working group a smashing success: we chartered it
quickly and gave it a focused, short-term task, it met its goals and
its schedule, and we have a new, standard IMAP extension for something
people have been asking for for years.

This is also the first IMAP-related RFC that will be published after
Mark Crispin's untimely passing.  We wouldn't be here doing this
without his earlier work.

Barry, Applications AD
_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Implementation experience -- final charter item

Barry Leiba — 2012-12-03T23:13:10

The imapmove charter requires an analysis of implementation
experience.  I know that Alexey has an implementation, and rumour has
it that there are several others.

There's a wiki page for collecting and saving this:
http://trac.tools.ietf.org/wg/imapmove/trac/wiki

If you have an implementation of some version of this spec (whether or
not it's the latest), please post any information you can about it.
Let us know whether you have a server implementation, a client
implementation, or both.  Let us know what version of the spec you
implemented, or whether you have an implementation of the initial idea
from before the WG was formed.  Let us know what experience you had in
implementing it, and any experience you have with deployment.  Let us
know any interoperability testing you've done.  Let us know whether
it's in shipping code, beta test, lab test, or whatnot, and when-ish
it's likely to be in shipping code if you're able to make an estimate.

Please post messages about all this to the mailing list.  The chairs
can decide how they want to deal with putting it into the wiki.

Thanks,
Barry
_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Protocol Action: 'Internet Message Access Protocol (IMAP)- MOVEExtension' to Proposed Standard(draft-ietf-imapmove-command-05.txt)

The IESG — 2012-12-03T22:19:57

The IESG has approved the following document:
- 'Internet Message Access Protocol (IMAP) - MOVE Extension'
  (draft-ietf-imapmove-command-05.txt) as Proposed Standard

This document is the product of the IMAP MOVE extension Working Group.

The IESG contact persons are Barry Leiba and Pete Resnick.

A URL of this Internet Draft is:
http://datatracker.ietf.org/doc/draft-ietf-imapmove-command/




Technical Summary

This document defines an IMAP extension to move messages from one 
mailbox to another. This function (very common in MUA UIs) is not 
provided by stock IMAP, and clients have to use a combination of 
UID STORE, UID COPY and EXPUNGE, and cope with partial failures 
and side effects. The document is targeted to become as 
Standards Track document, which is appropriate for this type of 
document. 


Working Group Summary

There was some discussion in the WG about whether MOVE command 
(as opposed to just UID MOVE) should be defined in the document 
or not. Consensus was that there was no reason not to define it 
(this preserves symmetry with other IMAP commands that operate on 
a group of messages.) 

There was also a discussion on whether multiple pipelined MOVE 
commands should be allowed or not. 

None of these discussions were particularly heated or long lived.


Document Quality

The document was extensively reviewed by IMAP experts and that is 
the most important type of review for it. 

There are multiple implementations of this IMAP extension in both 
clients and servers. More than 5 different servers implement it 
already. 


Personnel

Alexey Melnikov is the document shepherd. Barry Leiba is the 
responsible AD. 
_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

After MOVE - I propose DUMP/RESTORE format

Bron Gondwana — 2012-12-02T23:03:07

Now that MOVE is settling down, I really want to explore a vendor-neutral
DUMP and RESTORE format for IMAP mail stores.

I haven't given up on the IMAP5 concept - though in discussion with Arnt
recently we have both come to the conclusion that another stateful long-lived
protocol is probably not right.  What would be more useful to those of us
trying to build large clustered solutions is a protocol which is more batch
based, so it could be used in a connectionless way, without needing a complex
stateful proxy in the middle.

I'll come back to that.


Anyway - DUMP/RESTORE and Replication.

David (cc'd) did the initial work for replication in Cyrus, and I've built
on that with with new protocol in Cyrus 2.4.  It has some warts though - it
exposes far too much Cyrus implementation specifics.

I want to use the same concepts for a backup format which is durable,
incremental, and can be restored in such a way that a client which
connected again later can not tell that it's connecting to a restored
copy of the mailbox rather than the original mailbox.

I would also change our replication protocol for Cyrus to be based on
making synthetic incremental backups and applying them to the replica -
so that in theory any server which supported the DUMP/RESTORE format
would be able to be a replication source or destination.

Why do I think this is useful?
==============================

For users:
* ability to move between mail providers more easily (just import and
  export mailboxes)

For providers:
* ability to switch server software.
* ability to take efficient incremental backups which can be restored
  exactly as taken (at least in the case of Cyrus, this doesn't currently
  exist - you can't backup an entire mailbox at a point in time without
  having to reconstruct afterwards to clean up the mess - it doesn't do
  snapshots)

For us at FastMail/Opera:
* replace our custom-built Cyrus backup tool which does under-the-hood
  locking magic with something standard.


How Cyrus Replication works:
============================

We take advantage of the MODSEQ values from CONDSTORE/QRESYNC to work
out what needs to be sent to bring the replica into date.  At FastMail
we test our replicas once per week by running a set of IMAP queries
against both the master and replica (repeating multiple times in case
of mismatch so we don't get too many false positives).  These queries
are designed to exercise all the interesting fields, so that we know
the replication protocol is working correctly.

To ensure integrity, we also calculate a "SYNC_CRC" value.  This is the
XOR of a CRC32 for each UID in the mailbox.  The CRC32 for the individual
UIDs is calculated on a string formatted from all the mutable fields of
the message.  This allows efficient updates, because you can just calculate
the old and new values on each change, and XOR them together to get the
new SYNC_CRC value.

In the case of a SYNC_CRC mismatch, we download all the metadata (similar
to FETCH 1:* (UID FLAGS INTERNALDATE MODSEQ)) for every record and
calculate what changes need to be made on the replica to bring it back into
sync.  This is where UID promotion happens if the content of the message
is different.  We do mismatch detection with a GUID field (actually
DIGEST.SHA1 on the RFC822 body)

Here's an example from the wire:

<1354488670<COMPRESS DEFLATE
<1354488670<SET OPTIONS %(CRC_VERSION 2)

Creating two mailboxes:

<1354488685<GET MAILBOXES (user.foo user.foo.subdir)
<1354488685<APPLY MAILBOX %(UNIQUEID fa8673d7-471e-4d5b-9c7f-f3ee64106447 MBOXNAME user.foo LAST_UID 0 HIGHESTMODSEQ 2 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1354488684 PARTITION default ACL "foo lrswipkxtecdan  admin   lrswipkxtecd    " OPTIONS P SYNC_CRC 0 RECORD ())
<1354488685<APPLY MAILBOX %(UNIQUEID b2381db2-2e89-4a35-92d5-5ce55bf9fc4d MBOXNAME user.foo.subdir LAST_UID 0 HIGHESTMODSEQ 1 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 0 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1354488684 PARTITION default ACL "foo  lrswipkxtecdan  " OPTIONS P SYNC_CRC 0 RECORD ())

Appending two messages.  First we query the remote end to
see what's there now:

<1354488687<GET MAILBOXES (user.foo)
OK success

Then we try to reserve those messages (this is a waste of
time really, it shouldn't bother trying, that's a bug):

<1354488687<APPLY RESERVE %(PARTITION default MBOXNAME (user.foo) GUID (196922b6d822b618c665874fb523b9058a0adb56 ec4a76ae5e5f772dee837494134c75069286623a))
OK success
<1354488687<APPLY MESSAGE (%{default 196922b6d822b618c665874fb523b9058a0adb56 92}
From: test <test< at >example.com>
To: test <test< at >example.com>

Some stuff in the body...
.
 %{default ec4a76ae5e5f772dee837494134c75069286623a 372}
Return-Path: <brong< at >brong.net>
Received: from local (slot2 [127.0.0.52])
         by test_slot2_4092 (Cyrus git2.5+0) with LMTPA;
         Sun, 02 Dec 2012 23:51:26 +0100
X-Sieve: CMU Sieve 2.4
From: test <test< at >example.com>
To: test <test< at >example.com>
Message-ID: <cmu-lmtpd-4251-1354488686-0< at >test_slot2_4092>
Date: Sun, 02 Dec 2012 23:51:26 +0100

Some stuff in the body...
)

And finally, now that the messages are spooled, we update the
mailbox view.

<1354488687<APPLY MAILBOX %(UNIQUEID fa8673d7-471e-4d5b-9c7f-f3ee64106447 MBOXNAME user.foo LAST_UID 2 HIGHESTMODSEQ 5 RECENTUID 0 RECENTTIME 0 LAST_APPENDDATE 1354488687 POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1354488684 PARTITION default ACL "foo        lrswipkxtecdan  admin   lrswipkxtecd    hello   lrswipkxtecd    " OPTIONS P SYNC_CRC 31431471 RECORD (%(UID 1 MODSEQ 4 LAST_UPDATED 1354488686 FLAGS (\Flagged) INTERNALDATE 1268029091 SIZE 92 GUID 196922b6d822b618c665874fb523b9058a0adb56) %(UID 2 MODSEQ 5 LAST_UPDATED 1354488687 FLAGS () INTERNALDATE 1354488686 SIZE 372 GUID ec4a76ae5e5f772dee837494134c75069286623a)))

This is somewhat wasteful - it should be able to cache the remote
state and speculatively calculate a diff against what it expects to
be there and upload that as a single incremental dump.   In the usual
case that the other end hasn't changed, it could just apply that dump
and return the successful result.

NOTE though: SYNC_CRC 31431471.  The remote end calculates that after
applying these changes, and returns "OK" because it matched.

You can also see plenty of the warts there - both the %() syntax and things
like POP3_LAST_LOGIN and OPTIONS which are horribly vendor-specific.

Interesting bits to consider:
=============================

When backing up an entire user, you definitely want message de-duplication.
For FastMail, we delay EXPUNGE for a week, and also back up the EXPUNGEd
messages so that we our "restore from backup" feature can always find all
messages EXPUNGEd in the last week, even if we lose a server.

So this means there needs to be a way to cross-refererence a message in
another mailbox.  Cyrus uses SHA1 as the GUID, and uploads messages with
that identifier first before applying the rest of the changes.  For a
standard I believ we don't want to use a hash that's already on its way out
as the default, so we should look at an alternative to this.

There needs to be a way to read just the essential metadata from a
backup file quickly (that is uidvalidity,lastuid,highestmodseq) to
calculate which UIDs need their data included in the new backup, and
which also need their message bodies or XREFs included - similar to
the "GET MAILBOXES ()" query in the example above.

It also makes sense to have the format allow including an entire user's
mailboxes rather than doing each mailbox individually, since the XREFs
would otherwise be across backup files.

The format needs to both support every piece of data needed for every
current extension, and be extensible enough that new extensions' data
can be added.  Things I can think of immediately are METADATA/ANNOTATION
information, and what we in Cyrus call DELETEDMODSEQ - strictly, the
MODSEQ of the last EXPUNGEd message for which you have forgotten the
metadata.  Without this, you can't efficiently reply to QRESYNC queries,
because you need to tell about every gap in the UID sequence in case
the a message in there went away.

Thanks you:
===========

If you've read this far, thank you!  My goal is a format which captures
every piece of data which is required for any client connecting to the
server to be unable to see that it's a different server than previously
after a DUMP/RESTORE (assuming the server supports the same extensions
of course).  Anything which can be re-parsed from the message RFC822
doesn't belong in this format, only fields which are mutable (like FLAGS),
set externally (like INTERNALDATE), or necessary metadata about the past
(like MODSEQ and friends).

The biggest question facing me up front - what does it look like on the
disk/wire?  The Cyrus protocol at the moment looks almost like IMAP, and
parses almost like IMAP - with the added warts that it uses %() to
designate a list with key/value pairs rather than a list of items, and
it uses %{partition sha1 size} rather than {size+} to designate rfc822
messages.  That is clearly bogus for a generally applicable protocol.

The list of fields to include is quite clear - the only real consideration
is whether to support backups without MODSEQ information in them.  They
make incremental backups a lot harder, since you have to read all the
UID records from the old backup and compare them to the current values to
determine if anything has changed (like naive clients doing FETCH 1:*).
I would like a backup format that can support ANY server though, and be
built over regular IMAP by a standalone tool.

Regards,

Bron.

I-D Action: draft-ietf-imapmove-command-05.txt

2012-11-29T16:26:37

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the IMAP MOVE extension Working Group of the IETF.

Title           : Internet Message Access Protocol (IMAP) - MOVE Extension
Author(s)       : Arnt Gulbrandsen
                          Ned Freed
Filename        : draft-ietf-imapmove-command-05.txt
Pages           : 11
Date            : 2012-11-29

Abstract:
   This document defines an IMAP extension consisting of two new
   commands, MOVE and UID MOVE, that are used to move messages from one
   mailbox to another.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-imapmove-command

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-imapmove-command-05

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-imapmove-command-05


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Improved introduction to imapmove-command?

Ned Freed — 2012-11-28T17:49:06

Benoit Claise entered a comment (attached below) saying that the current
introduction is a little thin (which it is) and suggesting the addition
of some text from the charter.

The current introduction text is very short:

   This document defines an IMAP [RFC3501] extension to facilitate
   moving messages from one mailbox to another.  This is accomplished by
   defining a new MOVE command and extending the UID command to allow
   UID MOVE.

   A move function is not provided in the base IMAP specification, so
   clients have instead had to use a combination of the COPY, STORE, and
   EXPUNGE commands to perform this very common operation.  This has
   meant needing to cope with partial failures and side effects that can
   occur when multiple commands are involved.

   The MOVE extension is present in any IMAP4 implementation which
   returns "MOVE" as one of the supported capabilities to the CAPABILITY
   command.

My view is that the first and last paragraphs are fine, but the middle one
could do with some elaboration. I wish using the charter text had occurred to
me earlier; if it had I would have already done it.

Anyway, how about the following as a replacement?

    A move function is not provided in the base IMAP specification, so
    clients have instead had to use a combination of the COPY, STORE, and
    EXPUNGE commands to perform this very common operation.

    Implementors have long pointed out some shortcomings with this
    approach. Because the moving of a message is not an atomic process,
    interruptions can leave messages in intermediate states. Because
    multiple clients can be accessing the mailboxes at the same time,
    clients can see messages in intermediate states even without
    interruptions. If the source mailbox contains other messages that are
    flagged for deletion, the third step can have the side effect of
    expunging more than just the set of moved messages. And servers with
    certain types of back-end message stores might have efficient ways of
    moving messages, which don't involve actual copying of data. Such
    efficiencies are often not available to the COPY/STORE/EXPUNGE
    process.

I'm pretty sure this changes nothing about the protocol, so it's a safe
change to make at this point. 

Thoughts?
_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Additional text about interaction between MOVE and QRESYNC

Alexey Melnikov — 2012-11-28T17:04:41

I've decided to implement MOVE in Isode's server and I think something 
like the following should be added to section 4.4. While this can be 
deducted from the fact that MOVE is sort of COPY+STORE+EXPUNGE, RFC 5162 
explicitly lists all commands affected by it. So I propose adding:

    If the server is capable of storing modification sequences for the
    selected mailbox, it MUST increment the per-mailbox mod-sequence if
    at least one message was permanently moved due to the execution of
    the MOVE/UID MOVE command.  For each permanently removed message,
    the server MUST remember the incremented mod-sequence and corresponding
    UID.  If at least one message got moved, the server MUST send the
    updated per-mailbox modification sequence using the HIGHESTMODSEQ
    response code (defined in [RFC4551]) in the tagged OK response.

    When a message is moved to a target mailbox, if the server is capable
    of storing modification sequences for the mailbox, the server MUST
    generate a new modification sequence that is higher than the highest
    modification sequence of all messages in the mailbox and assigns it
    to the moved message.

And add RFC 4551 to references (Informative is Ok, but Normative would 
do as well).
I think this also makes RFC 5162 Normative.

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

I-D Action: draft-ietf-imapmove-command-04.txt

2012-11-27T04:17:55

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the IMAP MOVE extension Working Group of the IETF.

Title           : Internet Message Access Protocol (IMAP) - MOVE Extension
Author(s)       : Arnt Gulbrandsen
                          Ned Freed
Filename        : draft-ietf-imapmove-command-04.txt
Pages           : 10
Date            : 2012-11-26

Abstract:
   This document defines an IMAP extension consisting of two new
   commands, MOVE and UID MOVE, that are used to move messages from one
   mailbox to another.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-imapmove-command

There's also a htmlized version available at:
http://tools.ietf.org/html/draft-ietf-imapmove-command-04

A diff from the previous version is available at:
http://www.ietf.org/rfcdiff?url2=draft-ietf-imapmove-command-04


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Barry Leiba's Discuss on draft-ietf-imapmove-command-03:(withDISCUSS)

Barry Leiba — 2012-11-26T15:03:07

Barry Leiba has entered the following ballot position for
draft-ietf-imapmove-command-03: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)


Please refer to http://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.




----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

The last paragraph of Section 3.3 says this:

   Both MOVE and UID MOVE can be pipelined.  However, pipelined MOVE and
   UID MOVE commands MUST NOT specify overlapping sets of messages.

I'm not sure the "overlapping sets" is a sufficient restriction for
(non-UID) MOVE.  Consider a mailbox with messages having UIDs 101, 102,
103, 104, 105, 106, and 107, and this sequence:

C: t1 MOVE 2:3 Apple
C: t2 MOVE 4:5 Banana
S: * EXPUNGE 2
S: t1 NO [SERVERBUG] the move for message 3 failed

The command tagged t1 is meant to move UIDs 102 and 103.  That move will
cause the messages to be renumbered (leaving 101, 104, 105, 106, and
107), so that messages 4 and 5 represent UIDs 106 and 107.  But because
the move for message 3 (UID 103) failed, messages 4 and 5 will afterward
actually be UIDs 105 and 106.

I think that what this actually means is that the last two paragraphs are
contradictory:

   The server may send EXPUNGE (or VANISHED) messages before the tagged
   response, so the client cannot safely send more commands with message
   sequence number arguments while the server is processing MOVE.  The
   UID MOVE command does not have this limitation.

   Both MOVE and UID MOVE can be pipelined.  However, pipelined MOVE and
   UID MOVE commands MUST NOT specify overlapping sets of messages.

We can't have "the client cannot safely send more commands with message
sequence number arguments while the server is processing MOVE," and at
the same time allow the version of MOVE that uses sequence numbers to be
pipelined.  We might also consider a citation to RFC 3501, Section 5.5
here.




_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

News about Mark Crispin

Barry Leiba — 2012-11-20T00:44:51

Everyone here knows Mark Crispin -- or at least knows who he is: Mark is the author of the original 
IMAP specification, and has taken it through its different versions to the present IMAP4rev1.  He's 
written reference implementations of both server and client, and has been a vocal participant on 
all the mailing lists I'm posting this to.

I'm sad to have to report that Mark is now terminally ill, and is in hospice care.

For now, at least, I'm told that Mark is at least somewhat aware.  If anyone has brief well-wishing 
messages they'd like to send him, please post them to the <imap5< at >ietf.org> mailing list, and I'll 
forward them to Mark's long-term companion, Annie.  I will also post updates to that list as I get 
them.

[The Reply-To for this message is set to <imap5< at >ietf.org>, so that replies will go there.  You will 
have to subscribe to that mailing list in order to post to it.  You can do that here:
http://www.ietf.org/mailman/listinfo/imap5 ]

Barry Leiba

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

News about Mark Crispin

Barry Leiba — 2012-11-20T00:44:51

Everyone here knows Mark Crispin -- or at least knows who he is: Mark is the author of the original 
IMAP specification, and has taken it through its different versions to the present IMAP4rev1.  He's 
written reference implementations of both server and client, and has been a vocal participant on 
all the mailing lists I'm posting this to.

I'm sad to have to report that Mark is now terminally ill, and is in hospice care.

For now, at least, I'm told that Mark is at least somewhat aware.  If anyone has brief well-wishing 
messages they'd like to send him, please post them to the <imap5< at >ietf.org> mailing list, and I'll 
forward them to Mark's long-term companion, Annie.  I will also post updates to that list as I get 
them.

[The Reply-To for this message is set to <imap5< at >ietf.org>, so that replies will go there.  You will 
have to subscribe to that mailing list in order to post to it.  You can do that here:
http://www.ietf.org/mailman/listinfo/imap5 ]

Barry Leiba

_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext

Last Call: (The IMAPMoveExtension) to Proposed Standard

The IESG — 2012-11-07T23:05:47

The IESG has received a request from the IMAP MOVE extension WG
(imapmove) to consider the following document:
- 'The IMAP Move Extension'
  <draft-ietf-imapmove-command-02.txt> as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf< at >ietf.org mailing lists by 2012-11-21. Exceptionally, comments may be
sent to iesg< at >ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


   This document defines an IMAP extension consisting of two new
   commands, MOVE and UID MOVE, that are used to move messages from one
   mailbox to another.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-imapmove-command/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-imapmove-command/ballot/


No IPR declarations have been submitted directly on this I-D.


_______________________________________________
imapext mailing list
imapext< at >ietf.org
https://www.ietf.org/mailman/listinfo/imapext