gmane.mail.mime.gmime.devel

Re: [gmime-devel] "From" detection in FilterBest

Robert Schroll — 2013-05-06T00:12:55

Thanks.  I knew I was missing something there.

Robert


On Sun, May 5, 2013 at 7:40 PM, Jeffrey Stedfast <fejj-rDKQcyrBJuzYtjvyW6yDsg< at >public.gmane.org> wrote:

Re: [gmime-devel] "From" detection in FilterBest

Jeffrey Stedfast — 2013-05-05T23:40:15

Hi Robert,

It's actually the GMimeFilterFrom filter that QP-encodes the From-lines.

The idea was that if you were going to qp-encode any textual part, you'd 
attach the From-filter in "ARMOR" mode to qp-encode the F in From at the 
beginning of any lines.

Hope that helps,

Jeff

On 5/5/2013 7:11 PM, Robert Schroll wrote:

[gmime-devel] "From" detection in FilterBest

Robert Schroll — 2013-05-05T23:11:31

Hi all,

I've been looking at the source of FilterBest, and I note that one of 
the things it checks for is if any line begins with "From ".  If so, it 
will select (at least) the quoted-printable encoding.

I assume this is there to avoid confusing mbox files with lines that 
begin "From ".  But the quoted-printable encoding doesn't seem to alter 
this at all; that line will still begin "From ".  So this seems a bit 
pointless.

Am I missing something here?  Should the quoted-printable encoder be 
doing something to protect this line?  Or should this test just be 
removed?

Thanks,
Robert

Re: [gmime-devel] controling charset used ing_mime_utils_header_encode_text

Jeffrey Stedfast — 2013-05-04T13:43:09

Hi Paul,

US-ASCII and ISO-8859-1 are currently the exceptions to the 
user-charsets rule when encoding headers. US-ASCII because it is 
completely safe for transport and iso-8859-1 because it is expected that 
every mail client supports this charset.

I just added a g_mime_init() flag (GMIME_ENABLE_USE_ONLY_USER_CHARSETS?) 
to disable the current behavior and make it only use the user-charsets 
list to git master.

Jeff

On 5/3/2013 10:02 AM, Paul J Stevens wrote:

_______________________________________________
gmime-devel-list mailing list
gmime-devel-list< at >gnome.org
https://mail.gnome.org/mailman/listinfo/gmime-devel-list

[gmime-devel] controling charset used ing_mime_utils_header_encode_text

Paul J Stevens — 2013-05-03T14:02:06

Jeff,

is there a way to control the encoding used by mentioned call - and it's
sibling g_mime_utils_header_encode_phrase?

Even when I specifically set a preference to i.e. UTF-8 it keeps
encoding in ISO8859-1:

const char *sets[2] = {
 "UTF-8", "ISO8859-1"
};
g_mime_init(0);
g_mime_set_user_charsets(sets);
char *out = g_mime_utils_header_encode_text("jöst testing");
printf("[%s]\n", out);


still encodes this as

paul< at >shiko:~/test/gmime$ make
gcc -g -Wall -o test test.c `pkg-config --cflags --libs gmime-2.6`
paul< at >shiko:~/test/gmime$ ./test
[=?iso-8859-1?b?avZzdA==?= testing]

Re: [gmime-devel] messages with "> From" headers

Jeffrey Stedfast — 2013-04-25T14:23:15

Hi Dirk,

On 4/22/2013 3:18 PM, Dirk-Jan C. Binnema wrote:

There have been some changes to that area of code in the past 9 months 
or so, so it's possible that it used to work.

Maybe I can add a workaround for this particular case...

Jeff

Re: [gmime-devel] messages with "> From" headers

Paul — 2013-04-24T12:32:48



Hey, that's called a Postmark. Usually mail transports, such as Exim,
use it as indication of who actually sends this message.
It is not a part of any RFC so Gmime's behaviour is correct (Invalid
header).
You must be able to configure mail transport this ways so it will not
send Postmark to the end client.

We are using 2.4.24 and observe same behaviour, so that's not a recent
change i guess.

Hi,

It seems that recent gmime (2.6.12 maybe?) has gotten a bit pickier
regarding header parsing in g_mime_parser_construct_message; messages
that contain a pseudo-header starting with


(For example: see   https://github.com/djcb/mu/issues/189  )

make g_mime_parser_construct_message return NULL. I'm not sure what
causes such headers, but I think some tools generate those.

Would it be possible to simply ignore such headers?

Best wishes,
Dirk.

[gmime-devel] messages with "> From" headers

Dirk-Jan C. Binnema — 2013-04-22T19:18:55

Hi,

It seems that recent gmime (2.6.12 maybe?) has gotten a bit pickier
regarding header parsing in g_mime_parser_construct_message; messages
that contain a pseudo-header starting with


(For example: see https://github.com/djcb/mu/issues/189)

make g_mime_parser_construct_message return NULL. I'm not sure what
causes such headers, but I think some tools generate those.

Would it be possible to simply ignore such headers?

Best wishes,
Dirk.

[gmime-devel] unexpected rewrapping in long lines

Paul J Stevens — 2013-04-05T13:16:10

Jeff,

Below was confirmed on gmime-2.4 and gmime-2.6

When a message with a long utf7 encoded lines passes through gmime every
works as expected, except when a header is added to the message along
the line.

Consider below message. The quotation style was used to preserve indenting.

----
----

when it gets parsed as-is, everything is alright -
g_mime_object_to_string returns the message exactly as fed in.

However, when a header is added after parsing the message, the subject
line in this case gets re-wrapped in what appears to be an incorrect
fashion. At least, thunderbird and other mail clients don't grok it any
more.

Say a header 'X-Test' is set on above message:

g_mime_object_set_header(message->content, "X-Test", "testing");
g_mime_stream_reset(message->stream);
g_mime_object_write_to_stream(message->content, message->stream);

Now if I do g_mime_object_get_headers on the message->content
GMimeMessage object I get:

----
----

and the top-level mime-part also gets encoded as:


Does this ring any bells for you?

Re: [gmime-devel] unexpected rewrapping in long lines

Lou Picciano — 2013-04-05T13:44:38

Wow - This is really ringing some bells over here, given issues we'd seen with certain emails - usually at the point of loading headers, and most of the offending email being very UTF-8-ish...

Paul, tks for this confirmation.   

Lou Picciano

----- Original Message -----
From: Paul J Stevens <paul-oMN9YZNO2BY< at >public.gmane.org>
To: gmime-devel-list-rDKQcyrBJuzYtjvyW6yDsg< at >public.gmane.org
Sent: Fri, 05 Apr 2013 13:16:10 -0000 (UTC)
Subject: [gmime-devel] unexpected rewrapping in long lines

Jeff,

Below was confirmed on gmime-2.4 and gmime-2.6

When a message with a long utf7 encoded lines passes through gmime every
works as expected, except when a header is added to the message along
the line.

Consider below message. The quotation style was used to preserve indenting.

----
----

when it gets parsed as-is, everything is alright -
g_mime_object_to_string returns the message exactly as fed in.

However, when a header is added after parsing the message, the subject
line in this case gets re-wrapped in what appears to be an incorrect
fashion. At least, thunderbird and other mail clients don't grok it any
more.

Say a header 'X-Test' is set on above message:

g_mime_object_set_header(message->content, "X-Test", "testing");
g_mime_stream_reset(message->stream);
g_mime_object_write_to_stream(message->content, message->stream);

Now if I do g_mime_object_get_headers on the message->content
GMimeMessage object I get:

----
----

and the top-level mime-part also gets encoded as:

Does this ring any bells for you?

Re: [gmime-devel] unexpected rewrapping in long lines

Jeffrey Stedfast — 2013-04-06T03:34:42

Hi Paul,

I just filed the header folding bug: 
https://bugzilla.gnome.org/show_bug.cgi?id=697407

I've also got a fix I'm about to commit to git which seems to work.

I don't know about the other issue, though. When you file a bug about 
that one, I'll dig into it.

Jeff

On 4/5/2013 11:37 AM, Paul J Stevens wrote:

Re: [gmime-devel] unexpected rewrapping in long lines

Paul J Stevens — 2013-04-05T15:37:24


Ah. Ok.


I'll wip up a working proof-of-concept that exercises this bug and file
a report.


I didn't. But I will!

thanks

Re: [gmime-devel] unexpected rewrapping in long lines

Jeffrey Stedfast — 2013-04-05T14:11:18

Hi Paul,

On 4/5/2013 9:16 AM, Paul J Stevens wrote:

When a header is added, GMime currently destroys the cached header 
stream because it doesn't have any logic to merge unmodified headers 
with added/modified headers.

It looks like the Subject header above is broken in that there is no 
white space separating the beginning of the encoded-word token from the 
preceding text.


Looking at the header_fold() logic in gmime-utils.c, it doesn't look 
like it has all of the same complex logic of handling malformed rfc2047 
encoded-word tokens that the tokenize_rfc2047_text() code has.

As a workaround, you could register your own subject writer function 
that does something like:

static ssize_t
write_subject (GMimeStream *stream, const char *name, const char *value)
{
     char *decoded, *encoded, *unfolded, *folded;
     ssize_t n;

     decoded = g_mime_utils_header_decode_text (value);
     encoded = g_mime_utils_header_encode_text (decoded);
     g_free (decoded);
     unfolded = g_strdup_printf ("%s: %s\n", name, encoded);
     g_free (encoded);
     folded = g_mime_utils_unstructured_header_fold (unfolded);
     g_free (unfolded);

     n = g_mime_stream_write_string (stream, folded);
     g_free (folded);

     return n;
}

If you could submit a bug report about this and a sample message (with a 
Subject like the one you pasted above), I'll try to look into fixing 
this properly when I get a chance - probably by fixing the header_fold() 
logic to use the tokenizer or something.

Or maybe I'll fix it by enabling this identical workaround if 
g_mime_init() was initialized with the ENABLE_RFC2047_WORKAROUNDS flag 
(do you use that flag?).


Can't say that it does :-\

Jeff

[gmime-devel] Long boundary in MIME header

Lubos Hynek — 2013-03-11T13:28:49

Hi,

I have installed postfix, dbmail 2.2.11 and gmime 2.2.23.
It looks like the boundary in MIME header can be a maximum 62 characters
long. Then something (dbmail or gmime or anything else) cut the boundary
to
2 lines, like this:

MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary*0="=_65694ae2296c192d43aec4e2=631b8a19-d56b-5110-9c72-07052c91973";
boundary*1="d_="

This is by RFC2231, but MS OUTLOOK is not have implemented this RFC and
all attachments is in mail body.
According to the RFC, boundary can be up to 70 chars long, which is
perfect for me, because i have only 65 chars long boundary in original
email.
When postfix delivering mail through dbmail-lmtpd to dbmail, the boundary
is in original state (65 chars long).

Is any chance to change max. length to 70 chars or disable cutting?
If not is any settings in outlook, which help me?

Thanks for help.
Lubos

Re: [gmime-devel] Unecscaped Unicode

Robert Schroll — 2013-02-22T15:53:28

Thanks for thinking about this, Jeff.  I want to emphasize that the 
current implementation works, at least as long as there are no DEL 
characters in the email.  (And if you send an email with DEL characters, 
you deserve to have it mangled!)  So anything you do for GMime should 
make sense in the larger sense, and not just be a fix for this 
(non-)problem.

On 02/22/2013 12:08 PM, Jeffrey Stedfast wrote:

Looking for a string is slightly more complicated than for a single 
character, but there's no reason that wouldn't work.  But if I'm going 
to look for a string, I might as well just put in a unicode control 
character before the FilterHTML and look for the character entity on the 
other side.

This would make this last approach a bit more symmetric, since we'd be 
looking for the same thing on either side.  If it ends up in FilterHtML, 
we might utilize it.  But only bother if you see this as a generally 
useful feature -- there's plenty of ways to do what we need to right now.

Thanks,
Robert

Re: [gmime-devel] Unecscaped Unicode

Jeffrey Stedfast — 2013-02-22T15:08:39

Ah, Okay.


Wow, I'm so happy that someone is actually implementing their own filters!


Hmm, I think I understand what you're doing better now but I'm not sure 
I want to have a flag to define the quote char. That doesn't seem like 
the right solution.

Could you, instead of using a magic char, using a magic string? If you 
generate a randomish string then you could be fairly confidant that it 
wouldn't be in the message text. Then you could just look for that same 
string in the blockquote filter.

I could also add a flag like GMIME_FILTER_HTML_ALLOW_UTF8 which would 
pass utf8 through without encoding as entities:

             if (u >= 0x20 && u < 0x80) {
                 *outptr++ = (char) (u & 0xff);
             } else if (html->flags & GMIME_FILTER_HTML_ALLOW_UTF8) {
                 outptr += g_unichar_to_utf8 (u, outptr);
             } else if (html->flags & GMIME_FILTER_HTML_ESCAPE_8BIT) {
                 *outptr++ = '?';
             } else {
                 outptr += sprintf (outptr, "&#%u;", u);
             }

Jeff

Re: [gmime-devel] Unecscaped Unicode

Robert Schroll — 2013-02-21T13:19:29

Perhaps, but we're trying to make sure this works well with 
format=flowed emails, so this isn't as simple as changing the <font 
color=""> tags to <blockquote>s.

Here's a simple example.  I'll use ~ to represent space so you can 
better see what's going on.

 >~This~is~a~line~of~flowed~text~
 >~that~has~been~wrapped.
 >~But~this~is~a~new~line.
~>~This~looks~like~a~quote,~but~
it~isn't.

should become:

<blockquote>This is a line of flowed text that has been wrapped.
But this is a new line.</blockquote>
&lt; This looks like a quote, but isn't.

You can take a look at our implementation-in-progress, which uses a 
filter on either side of the FilterHTML.  Prior to the FilterHTML, 
FilterFlowed 
(https://github.com/rschroll/geary/blob/plaintext/src/engine/rfc822/rfc822-gmime-filter-flowed.vala) 
undoes line wrapping and space stuffing, and converts quote symbols to 
0x7f.  After FilterHTML, FilterBlockquote 
(https://github.com/rschroll/geary/blob/plaintext/src/engine/rfc822/rfc822-gmime-filter-blockquotes.vala) 
removes the 0x7f flags and inserts <blockquote>s appropriately.  This 
works fine, unless the email has lines starting with 0x7f.  (Which it 
won't, so why am I worrying?)

As you see, the latter is relatively simple, but the former is 
non-trivial.  Either we'd need all of that in FilterHTML, or FilterHTML 
would need a flag to indicate quote levels separate from >.  Actually, 
the second solution might be feasible.  FilterHTML could gain an 
optional "quote_marker" flag, defaulting to ">", so it would work 
automatically with unprocessed text.  But people like me could set it to 
be something else and do our own preprocessing.

Does that make sense?

Thanks,
Robert

Re: [gmime-devel] Need code snippet for MIME decoding

Jeffrey Stedfast — 2013-02-21T12:00:48

Hi Prassad,

It's possible that g_mime_part_get_filename() is returning null. It's 
not guaranteed to find a filename parameter and so in those cases will 
return null.

Hope that helps,

Jeff

On 2/21/2013 6:45 AM, prasad antony wrote:

Re: [gmime-devel] Need code snippet for MIME decoding

prasad antony — 2013-02-21T11:45:41

Hi Jeff,

i think the issue is resolved. It could be because of some memory issues
here. Sorry for the trouble. Let me see more on this.

thanks
Prasad.


On Thu, Feb 21, 2013 at 11:35 AM, prasad antony
<prasadantonymad-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

Re: [gmime-devel] Need code snippet for MIME decoding

prasad antony — 2013-02-21T06:05:47

Hi Jeff,

Here is my code snippet. I was not using gmime APIs to create my file path
location.

const char* outputfile = "/home/prasad/filepathdir/"
char *temp = NULL;
const char* name;
char *tempfile = NULL;

name = g_mime_part_get_filename ((GMimePart *) part);
            temp = (char *)malloc(strlen(name) + 1);  --------> In this
line itself I am getting segmentation fault because of
strlen(name)--------->
            strcpy(temp,name);
            printf("%s\n",temp);
            tempfile = (char*)malloc(strlen(outputfile) + strlen(name) + 1);
            strcpy(tempfile,outputfile);
            strcat(tempfile,temp);

            if ((fd = open (tempfile, O_CREAT | O_WRONLY, 0666)) != -1) {
                content = g_mime_part_get_content_object ((GMimePart *)
part);
                stream = g_mime_stream_fs_new (fd);

I can do this even without temp variable. strcat(tempfile,name). While
accessing the variable name to standard C functions it is giving
segmentation fault.

In parallel let me try with your suggestion to do it with gmime APIs....But
I would like to understand why the segmentation fault with my code....

Thanks so much Jeff for your quick reply....

Thanks
Prasad.


On Thu, Feb 21, 2013 at 6:26 AM, Jeffrey Stedfast <fejj-rDKQcyrBJuzYtjvyW6yDsg< at >public.gmane.org> wrote:

Re: [gmime-devel] Unecscaped Unicode

Jeffrey Stedfast — 2013-02-21T03:24:12

Hi Robert,

It doesn't look like there's any way to not escape unicode nor does it 
look like it'll accept invalid utf-8.

Maybe it would make sense to have gmime's html filter optionally add 
blockquotes instead of doing things the way your doing it?

Can you give me an example of the input and what you want as the output? 
Do you want to nest the <blockquote>'s according to the line's citation 
depth?

Not sure how big the can of worms is that my big mouth is offering to 
implement, but if it's not too difficult maybe I can add that feature ;-)

Jeff

On 2/20/2013 9:11 PM, Robert Schroll wrote: