gmane.mail.imap.dbmail

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Alan Hicks — 2012-05-24T17:10:33

On 24/05/2012 16:28, Daniel Schütze wrote:
I've just submitted an upgrade request, 
http://www.freebsd.org/cgi/query-pr.cgi?pr=168308

Alan Hicks

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Alan Hicks — 2012-05-24T17:07:48

Resending as it does not appear to have arrived!

On 24/05/2012 10:32, Alan Hicks wrote:
Submitted as http://www.freebsd.org/cgi/query-pr.cgi?pr=168308
_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Daniel Schütze — 2012-05-24T15:28:24

Yes I ran it without error (took 24 hrs on the test box mind you!).


I hadn't appreciated that at the time, though it is clearly laid out in the
docs which I saw when I re-read them.  I will want to migrate the messages
in due course to free up space of course.



I am using the freebsd ports and the latest version available is 2.4.24

Given the whole run is a test anyway I've started from scratch after making
some adjustments esp. as I want to see the system in action above rebuilding
the headers but before the message migration, then I'll try the migration
and see if that gives rise to the same issue.

Daniel Schütze

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Paul J Stevens — 2012-05-24T07:52:40

Did you run the full schema migration successfully first?


Running dbmail-util -M is not required and does not require downtime.
The data-store is backward compatible - only body searches will not work
on messageblks.

That said: I haven't seen this problem yet.

Since the core dump seems to originate from gmime, I would advise you to
upgrade to the latest gmime-2.4. I see rfc2047 related bugfixes at least
in 2.4.30, and probably later ones as well.

Re: switch to GMime-2.6 for development

Reindl Harald — 2012-05-23T17:50:03


Am 23.05.2012 19:46, schrieb Alan Hodgson:

your disks maybe

mine in a HP SAN-Storage are not nor is it the overhead
on upgrades, overhead on backup-space and so on

the "disks are cheap" is a urban legend from homeusers
calculatiing with their cheap SATA disks :-)

having 400 MB more or less on at least three build machines
and testing servers makes really a difference on dist-upgrades
twice a year, on normal updates and consumed time at all

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: switch to GMime-2.6 for development

Alan Hodgson — 2012-05-23T17:46:15

Well, no offense, but disk is cheap. Give mock 20GB and you can build anything.

Re: switch to GMime-2.6 for development

Reindl Harald — 2012-05-23T17:38:34


Am 23.05.2012 19:24, schrieb Alan Hodgson:

in theorey

with dbmail2.2 i rebuilt gmime22 on our build-environment
a rebuild of the current gmime versions pulls a ton of
BuildRequirements including crap like mono with a summary
around 300 MB

that said: our main server needs 2.0 GB for the rootfs
the specialized ones 1.3 GB

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: switch to GMime-2.6 for development

Alan Hodgson — 2012-05-23T17:24:39

Not much else really uses gmime. You can just rebuild FC17's and stick it in 
your local package repo.

Re: switch to GMime-2.6 for development

Reindl Harald — 2012-05-23T17:19:08


Am 23.05.2012 18:27, schrieb Jorge Bastos:

no, thanks :-)

no .deb after 10 years Redhat
no back to sysv after all the pain with systemd at begin

and no replacement of > 20 virtual servers with a internal
build/update/test-infrastructure and not after 7 successfull
dist-upgrades in this environment from F9 to F16 since 2008

not for one package and running the mailsevrer another
software as the rest leads in multiple work :-)

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: switch to GMime-2.6 for development

Jorge Bastos — 2012-05-23T16:27:50

Go Debian :)

Works fine for me, and 2.6 is only for the master not the stable 3.x

switch to GMime-2.6 for development

Reindl Harald — 2012-05-23T16:22:03

Hi


i am fine with this

but


http://koji.fedoraproject.org/koji/packageinfo?packageID=325
Fedora 17 (released in two weeks is the first one with 2.6)

i do not like to upgrade to F17 until it is hardly needed because
currently are too much php-pecl extensions still not supporting
PHP 5.4 (the most important suhosin)

please do not require gmime 2.6 too soon because Fedora
is bleeding edge and other distributions are behind

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: Sieve script vacation not working

Paul J Stevens — 2012-05-23T09:46:33

You need to specify what addresses you want to respond to. Also you
subject match always matches and is therefor redundant:

require "vacation";
if true {
  vacation :days 7 :addresses [leandro< at >texnet.it]
    :subject "Automatic response" "body text";
}

Sieve script vacation not working

Leandro — 2012-05-22T15:56:04

I am using dbmail 3.0 and while sieve scripts work fine, I can't use the
vacation script. I tried a very simple one like:

require ["vacation"];
   if header :matches "subject" "*" {
       vacation :subject "Automatic response"
                "I'm away -- send mail to foo in my absence";
   }

But it doesn't work. I get the following error in the debug log:

May 22 17:24:37 dbmail32 dbmail-lmtpd[4512]: [0x9bc3fb8] Debug:[sort]
sort_debugtrace(+625): sieve: [sv_interface,script.c,libsieve_eval:
[VACATION aborted: no match found in script.]

Any idea? Can you provide me your sieve script for example?

Leandro

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Daniel Urstöger — 2012-05-22T13:37:06

Hi Daniel,

just came to my mind: did you run dbmail util on your orginal storage/version?
Maybe there is a problem in the dataset that might get fixed this way?

Otherwise I think you have to wait till Paul gets on this topic.

Greetings,
Daniel

Am 22.05.2012 um 15:02 schrieb Daniel Schütze:

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Daniel Schütze — 2012-05-22T13:02:36

Actually it's now gotten to the point where dbmail-util -My doesnt run for
more than a couple of seconds before I get the errors I'm encountering.  I'm
not getting so many core dumps as just simple aborts to the command line:

May 22 13:56:22 dbm3 dbmail-util[14952]: [0x80220b040] Error:[db]
db_exec(+319): SQLException: Cannot add or update a child row: a foreign
key constraint fails (`dbmail`.`dbmail_partlists`, CONSTRAINT
`dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`) REFERENCES
`dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE)
May 22 13:56:22 dbm3 dbmail-util[14952]: [0x80220b040] Error:[db]
db_exec(+320): failed query [INSERT INTO dbmail_partlists (physmessage_id,
is_header, part_key, part_depth, part_order, part_id) VALUES
(657895,1,1,0,0,27)]
migrating physmessage_id: 657895
failed

From the messages (phymessage_id) affected it is clear that the frequency is
getting worse and high enough that the conversion exercise now seems futile.

Clearly I'll need to know/figure out what is causing this to try and
proceed.

Daniel Schütze

Re: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

Daniel Schütze — 2012-05-22T11:17:51

Just as an update to my message, it occurred to me the foreign key error
could well be a result of the core dump and nothing to do with what was
causing the error.

 

Im a little out of my depth here, but a look at the core dumps seems to
point towards a problem around gmime (using gmime-24-2.4.24), is that
possible?

 

#0  0x0000000801e15786 in memcpy () from /lib/libc.so.7

[New Thread 8024b61c0 (LWP 100286)]

[New Thread 8024041c0 (LWP 100208)]

(gdb) where

#0  0x0000000801e15786 in memcpy () from /lib/libc.so.7

#1  0x00000008007dfe51 in rfc2047_encode ()

   from /usr/local/lib/libgmime-2.4.so.6

#2  0x00007fffffffb740 in ?? ()

#3  0x00007fffffffb670 in ?? ()

#4  0x0000000801e1f41c in __uppercase_hex () from /lib/libc.so.7

#5  0x200a3e2200000001 in ?? ()

 

(retrieved from gdb /usr/local/sbin/dbmail-util *.core; where)

 

And a second dump which occurred as I was writing this e-mail

 

(gdb) where

#0  0x0000000801e12786 in memcpy () from /lib/libc.so.7

#1  0x00000008007dc911 in stream_read () from
/usr/local/lib/libgmime-2.4.so.6

#2  0x00000008007d8890 in g_mime_stream_write_to_stream ()

   from /usr/local/lib/libgmime-2.4.so.6

#3  0x00000008007d6e81 in mime_part_write_to_stream ()

   from /usr/local/lib/libgmime-2.4.so.6

#4  0x00000008007cafe2 in message_write_to_stream ()

   from /usr/local/lib/libgmime-2.4.so.6

#5  0x00000008007d08c6 in g_mime_object_to_string ()

   from /usr/local/lib/libgmime-2.4.so.6

#6  0x0000000800666c45 in dbmail_message_init_with_string ()

   from /usr/local/lib/dbmail/libdbmail.so.0

#7  0x00000008006676d5 in _retrieve ()

   from /usr/local/lib/dbmail/libdbmail.so.0

#8  0x0000000800667797 in dbmail_message_retrieve ()

   from /usr/local/lib/dbmail/libdbmail.so.0

#9  0x0000000000403dc8 in do_showhelp ()

#10 0x0000000000404d6e in main ()

 

 

 

Daniel Schütze

 

------------------------

CWA International

Balmoral House

9 John Street

London
WC1N 2ES

 

(t) + 44 (0)20 7242 8444

(e) dms< at >cwa.uk.com

(w) http://www.cwa.uk.com/

 

From: Daniel Schütze [mailto:daniel< at >cwa.uk.com] 
Sent: 22 May 2012 10:51
To: 'dbmail< at >dbmail.org'
Subject: Migration of Data from 2.2.17 to 3.0.2 (dbmail-util coredumping)

 

Im attempting a migration of a Dbmail 2.2.17 installation to 3.0.2 with a
backup of our production database on a spare machine to test the procedure.


Our dbmail installation contains some 900k messages and is some 360gig
(data; index 1.9gig).  The primary reason for wishing to migrate is for the
single instance storage of mime-parts to hopefully reduce the storage space
significantly (when I tried the same migration last year with the RC there
was a reduction by some 60%).

 

I am at the stage of running dbmail-util My to convert the message
attachments to the new storage system and am hitting a brick wall in the
sense that dbmail-util is repeatedly coredumping.  If I run the dbmail-util
command immediately again I receive foreign key errors:

 

May 22 10:34:35 dbm3 dbmail-util[6041]: [0x80240b040] Error:[db]
db_exec(+319): SQLException: Cannot add or update a child row: a foreign key
constraint fails (`dbmail`.`dbmail_partlists`, CONSTRAINT
`dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`) REFERENCES
`dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE)

May 22 10:34:35 dbm3 dbmail-util[6041]: [0x80240b040] Error:[db]
db_exec(+320): failed query [INSERT INTO dbmail_partlists (physmessage_id,
is_header, part_key, part_depth, part_order, part_id) VALUES
(612844,1,1,0,0,27)]

migrating physmessage_id: 612844 failed

 

The show innodb status;  gives

 

120522 10:34:35 Transaction:

TRANSACTION 0 10128353, ACTIVE 0 sec, OS thread id 34384941632 inserting,
thread declared inside InnoDB 500

mysql tables in use 1, locked 1

3 lock struct(s), heap size 1216, 1 row lock(s)

MySQL thread id 105, query id 3033949 localhost dbmail update

INSERT INTO dbmail_partlists (physmessage_id, is_header, part_key,
part_depth, part_order, part_id) VALUES (612844,1,1,0,0,27)

Foreign key constraint fails for table `dbmail`.`dbmail_partlists`:

,

  CONSTRAINT `dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`)
REFERENCES `dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE

Trying to add in child table, in index `message_parts` tuple:

DATA TUPLE: 8 fields;

0: len 8; hex 00000000000959ec; asc       Y ;; 1: len 2; hex 8001; asc   ;;
2: len 2; hex 8000; asc   ;; 3: len 2; hex 8000; asc   ;; 4: len 6; hex
0000009a8be1; asc       ;; 5: len 7; hex 800000002d0110; asc     -  ;; 6:
len 1; hex 81; asc  ;; 7: len 8; hex 000000000000001b; asc         ;;

 

But in parent table `dbmail`.`dbmail_physmessage`, in index `PRIMARY`,

the closest match we can find is record:

PHYSICAL RECORD: n_fields 6; compact format; info bits 0

0: len 8; hex 00000000000959ee; asc       Y ;; 1: len 6; hex 00000009d994;
asc       ;; 2: len 7; hex 800012000c0335; asc       5;; 3: len 8; hex
0000000000001124; asc        $;; 4: len 8; hex 000000000000116f; asc
o;; 5: len 8; hex 8000124a4658ef9f; asc    JFX  ;;

 

 

which I can avoid by deleting the offending message i.e. delete from
dbmail_messageblks where physmessage_id = 609063;  Deleting them is fine
for my testing purposes but obviously is going to be an issue if/when I
carry out the migration for real.

 

Can anyone suggest to me what might be going wrong, or if there is anything
useful I can examine?  So far this has happened 7 times up to physmessage_id
612844 (out of the 900k or so) so it is infrequent but being a coredump it
does mean it halts the migration which is inconvenient.  I am running MySQL
5.1.55, on freebsd 8.2 for general information.

 

Also, and separately, the migration is pretty slow overall which is an issue
for the downtime it will cause.  Looking at the test box (4 core Xeon, 4gig
ram, standard hdd) during the conversion the processor load is low (20%
overall i.e. 1 processor at under 100%), the disk utility is low (normally
under 50%) and innodb pool is set to 500meg which is obviously full, so can
someone suggest what is rate limiting the conversion?

 

Thank you


Daniel Schütze

------------------------

CWA International

Balmoral House

9 John Street

London
WC1N 2ES

 

(t) + 44 (0)20 7242 8444

(e) dms< at >cwa.uk.com

(w) http://www.cwa.uk.com/

 

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Migration of Data from 2.2.17 to 3.0.2 (dbmail-utilcoredumping)

Daniel Schütze — 2012-05-22T09:51:15

Im attempting a migration of a Dbmail 2.2.17 installation to 3.0.2 with a
backup of our production database on a spare machine to test the procedure.


Our dbmail installation contains some 900k messages and is some 360gig
(data; index 1.9gig).  The primary reason for wishing to migrate is for the
single instance storage of mime-parts to hopefully reduce the storage space
significantly (when I tried the same migration last year with the RC there
was a reduction by some 60%).

 

I am at the stage of running dbmail-util My to convert the message
attachments to the new storage system and am hitting a brick wall in the
sense that dbmail-util is repeatedly coredumping.  If I run the dbmail-util
command immediately again I receive foreign key errors:

 

May 22 10:34:35 dbm3 dbmail-util[6041]: [0x80240b040] Error:[db]
db_exec(+319): SQLException: Cannot add or update a child row: a foreign key
constraint fails (`dbmail`.`dbmail_partlists`, CONSTRAINT
`dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`) REFERENCES
`dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE)

May 22 10:34:35 dbm3 dbmail-util[6041]: [0x80240b040] Error:[db]
db_exec(+320): failed query [INSERT INTO dbmail_partlists (physmessage_id,
is_header, part_key, part_depth, part_order, part_id) VALUES
(612844,1,1,0,0,27)]

migrating physmessage_id: 612844 failed

 

The show innodb status;  gives

 

120522 10:34:35 Transaction:

TRANSACTION 0 10128353, ACTIVE 0 sec, OS thread id 34384941632 inserting,
thread declared inside InnoDB 500

mysql tables in use 1, locked 1

3 lock struct(s), heap size 1216, 1 row lock(s)

MySQL thread id 105, query id 3033949 localhost dbmail update

INSERT INTO dbmail_partlists (physmessage_id, is_header, part_key,
part_depth, part_order, part_id) VALUES (612844,1,1,0,0,27)

Foreign key constraint fails for table `dbmail`.`dbmail_partlists`:

,

  CONSTRAINT `dbmail_partlists_ibfk_1` FOREIGN KEY (`physmessage_id`)
REFERENCES `dbmail_physmessage` (`id`) ON DELETE CASCADE ON UPDATE CASCADE

Trying to add in child table, in index `message_parts` tuple:

DATA TUPLE: 8 fields;

0: len 8; hex 00000000000959ec; asc       Y ;; 1: len 2; hex 8001; asc   ;;
2: len 2; hex 8000; asc   ;; 3: len 2; hex 8000; asc   ;; 4: len 6; hex
0000009a8be1; asc       ;; 5: len 7; hex 800000002d0110; asc     -  ;; 6:
len 1; hex 81; asc  ;; 7: len 8; hex 000000000000001b; asc         ;;

 

But in parent table `dbmail`.`dbmail_physmessage`, in index `PRIMARY`,

the closest match we can find is record:

PHYSICAL RECORD: n_fields 6; compact format; info bits 0

0: len 8; hex 00000000000959ee; asc       Y ;; 1: len 6; hex 00000009d994;
asc       ;; 2: len 7; hex 800012000c0335; asc       5;; 3: len 8; hex
0000000000001124; asc        $;; 4: len 8; hex 000000000000116f; asc
o;; 5: len 8; hex 8000124a4658ef9f; asc    JFX  ;;

 

 

which I can avoid by deleting the offending message i.e. delete from
dbmail_messageblks where physmessage_id = 609063;  Deleting them is fine
for my testing purposes but obviously is going to be an issue if/when I
carry out the migration for real.

 

Can anyone suggest to me what might be going wrong, or if there is anything
useful I can examine?  So far this has happened 7 times up to physmessage_id
612844 (out of the 900k or so) so it is infrequent but being a coredump it
does mean it halts the migration which is inconvenient.  I am running MySQL
5.1.55, on freebsd 8.2 for general information.

 

Also, and separately, the migration is pretty slow overall which is an issue
for the downtime it will cause.  Looking at the test box (4 core Xeon, 4gig
ram, standard hdd) during the conversion the processor load is low (20%
overall i.e. 1 processor at under 100%), the disk utility is low (normally
under 50%) and innodb pool is set to 500meg which is obviously full, so can
someone suggest what is rate limiting the conversion?

 

Thank you


Daniel Schütze

------------------------

CWA International

Balmoral House

9 John Street

London
WC1N 2ES

 

(t) + 44 (0)20 7242 8444

(e) dms< at >cwa.uk.com

(w) http://www.cwa.uk.com/

 

_______________________________________________
DBmail mailing list
DBmail< at >dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

innodb tweaks

Kamenik, Aleksander — 2012-05-18T13:22:23

Hi,

Has anybody done any performance and storage tests for innodb with regard to dbmail 3.0. Like using compression and partitioning on the email data tables?

https://dev.mysql.com/doc/refman/5.5/en/innodb-compression.html
https://dev.mysql.com/doc/refman/5.5/en/partitioning.html

Also, any recommendations on setting the various innodb buffer sizes, that deviate from the standard recommendations? Maybe someone can simply share their experiences with tweaking innodb on a dbmail install.

Regards,

Aleksander Kamenik

Re: Housekeeping dbmail_headervalue

ITronic Harald Leithner — 2012-05-15T15:14:39

Hi Leandro,

there is already a good solution for this:  
http://www.gossamer-threads.com/lists/dbmail/users/31530#31530

But I think Paul has other things to develop, may you could write the code  
that do this?

bye

HArald

Am 15.05.2012, 17:08 Uhr, schrieb leandro <leandro< at >texnet.it>:

Re: Housekeeping dbmail_headervalue

Reindl Harald — 2012-05-15T15:11:59

yes this is a missing piece in dbmail-util and should be
done manually this time because this table had 80% of all
recods here some weeks ago

Am 15.05.2012 17:08, schrieb leandro:

Housekeeping dbmail_headervalue

leandro — 2012-05-15T15:08:29

Hello,
am I wrong or there is no dbmail-util procedure to delete headervalues
no more connected to messages?

After deleting some users directly from the dbmail_database and running
the dbmail-util -ay housekeeping utility, I noticed the
dbmail_headervalue table size remains pretty the same.

If I run manually the query:

select count(*) from dbmail_headervalue where id not in ( select
headervalue_id from dbmail_header );

I discovered a large number of no more connected header values.

Almost the same if I check the headernames with the query:

select count(*) from dbmail_headername where id not in ( select
headername_id from dbmail_header );

Leandro