gmane.mail.qmail.simscan

pcregrep multiline

alexus — 2013-02-22T02:58:35

hi,

I need to do a multiline regex, how would I do that in simcontrol?

Re: [SPAM] Re: [simscan] [SPAM] How to inject email to simscan ?

FC Mario Patty — 2013-02-19T04:07:00

Guys, I made it. I got the inspiration from
    http://www.mail-archive.com/linuxvadapav-hHKSG33TihhbjbujkaE4pw< at >public.gmane.org/msg02449.html

Since I use simscan, the route is slightly different
1. I only have "exec qmail-disclaimer" within the qmail-tail
2. Pipe the file into simscan from within qmail-disclaimer :
cat $INSPECT_DIR/in.$$ | /var/qmail/bin/simscan

cat >$INSPECT_DIR/in.$$ || { echo Error: Cannot write to disk; exit
$EX_TEMPFAIL; }

/usr/local/bin/altermime --input=$INSPECT_DIR/in.$$ \
--xheader="X-Disclaimer: $QMAILHOST" \
--disclaimer=/var/qmail/control/disclaimer/$QMAILHOST.txt \
--disclaimer-html=/var/qmail/control/disclaimer/$QMAILHOST.html || { echo
Error: message rejected; exit $EX_UNAVAILABLE; }

cat $INSPECT_DIR/in.$$
rm in.$$

#!/bin/sh
exec /usr/bin/qmail-qfilter /var/qmail/bin/qmail-disclaimer





On Wed, Jun 6, 2012 at 6:31 PM, FC Mario Patty <fcmario76-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:



!DSPAM:5122df1834141288574255!

[SPAM] Getting random failures with simscan cleaning up files

Bob Maple — 2012-11-12T07:20:08

Running simscan 1.4.0 with clamav and dspam support.

For several weeks (at least, I'm not sure precisely when it started 
happening to be honest) I've been getting random duplicate emails.  To 
make a long story short, I've been adding a bunch of fprintf()s to the 
code to try and figure out what is going on, and have it narrowed down 
to the remove_files() function returning failure, and within it, 
specifically the unlink() on a file.

But I'm at a loss as to why this is happening.  Clamav gets called, 
dspam gets called, and then qmail-queue gets called all with success.. 
then simscan fails when cleaning up the temp files, and fails the delivery.

I can't seem to find any rhyme or reason when comparing the failures to 
the successful deliveries... it just seems random.  It's taken me weeks 
to narrow this down, as I started adding fprintf()s and then have to 
wait until the problem happens again, then add some more, wait, etc.  I 
can't duplicate it on demand.

Anyway, here's the last few log lines from a failure:

2012-11-11 23:09:13 simscan: qmail-queue exited 0
2012-11-11 23:09:13 ---- remove_files did lstat
2012-11-11 23:09:13 ---- /var/qmail/simscan/1352700552.454197.26473 is a 
directory, doing chdir
2012-11-11 23:09:13 simscan: error unlinking 
spamc.msg.1352700552.454197.26473 errno=2
2012-11-11 23:09:13 simscan: error removing working files
2012-11-11 23:09:13 ---- remove_files did lstat
2012-11-11 23:09:13 ---- /var/qmail/simscan/1352700552.454197.26473 is a 
directory, doing chdir
2012-11-11 23:09:13 simscan: exit error code: 71

errno 2 being 'no such file or directory' - which indeed if I go into 
/var/qmail/simscan after the fact, it's empty...  So either the unlink() 
really IS succeeding but returning failure for some reason (library 
bug!?) or the file is somehow getting cleaned up by someone/something 
else first... ?

So, just wondering if anyone else has noticed anything weird going on 
with their own systems, or have any random ideas what could possibly be 
happening.  I'm about ready to just start patching the code so 
remove_files() doesn't return failure, but I'd really like to figure out 
the cause of why this has suddenly started happening.

Re: [SPAM] No title in simscan (qmail smtpd)

FC Mario Patty — 2012-10-25T04:41:27

Sorry, a "small" correction for my previous information:

I said that we must use --enable-spam-auth-user=y, but as you read from my
setup.sh it stated --enable-spam-auth-user=n. What I was trying to do is to
let users who had authenticated them self to get RELAYED automatically. But
since I needed a real subject instead of a '-', I inserted some codes into
simscan.c.

That's all. Thank you.

Regards,

On Sat, Apr 14, 2012 at 6:36 PM, FC Mario Patty <fcmario76-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:



!DSPAM:5088c2fd34171831133953!

Re: [SPAM] Re: [simscan] [SPAM] How to inject email to simscan ?

FC Mario Patty — 2012-06-06T11:31:49

Dear Eric,

You're right about the command; I piped it instead of write it (>) to
simscan*. I had a bad experience with that kind of command years ago so I
never try it again (in a production mail server too; thanks God I still had
the pre-configured source code so I just had to re-make it). By the way,
thank you for the information. I will try your suggestion and come with the
result.

* I tried "simscan << cat email" too but they got dumped by spamassassin;
when I copied the quarantined message to my maildirs and downloaded it,
there's nothing inside.

Regards,

Mario

On Wed, Jun 6, 2012 at 5:49 PM, Eric Koldeweij <eric-AIJAZ5tcjQHk1uMJSBkQmQ< at >public.gmane.org> wrote:



!DSPAM:4fcf3fa834207421421242!

[SPAM] Re: [simscan] [SPAM] How to inject email to simscan ?

Eric Koldeweij — 2012-06-06T10:49:51

Mario,

simscan is a "replacement" for qmail-queue and therefore must follow the 
same I/O interface. From the qmail-queue manual page:

==== quote ====
qmail-queue  reads a mail message from descriptor 0.  It then reads 
envelope information from descriptor 1.  It places the message into the 
outgoing queue for future delivery by qmail-send.

The envelope information is an envelope sender address followed by a 
list of envelope recipient addresses.  The sender address is preceded 
by  the letter  F  and  terminated  by  a  0  byte.  Each recipient 
address is preceded by the letter T and terminated by a 0 byte.  The 
list of recipient addresses is terminated by an extra 0 byte.  If 
qmail-queue sees end-of-file before the extra 0 byte, it aborts without 
placing the  message  into the queue.

Every envelope recipient address should contain a username, an < at > sign, 
and a fully qualified domain name.
==== end quote ====

Since this interface uses FD 0 and 1 a simple cat does not work because 
it will present the e-mail message on simscan's FD 0 which will not 
work. By the way, I assume you did "cat the-file | 
/var/qmail/bin/simscan" instead of "cat the-file > 
/var/qmail/bin/simscan", the latter will destroy your simscan executable.

It seems that your "new" qmail-queue has become 
/var/qmail/bin/qmail-tail. What it should do is present the envelope 
information to simscan on file descriptor 1 and the mail file with 
disclaimer to file descriptor 0. See the simscan source code on how to 
do that - it's doing exactly the same to qmail-queue.

Another option is to alter simscan itself to add the disclaimer to the 
e-mail message. This is not very difficult to do.

Regards,
Eric.


On 06/06/12 11:33, FC Mario Patty wrote:


!DSPAM:4fcf35d134209081215598!

[SPAM] How to inject email to simscan ?

FC Mario Patty — 2012-06-06T09:33:53

Hi guys,

I'm trying to add disclaimer to email. All this time we use qmail-ldap and
simscan. So I downloaded altermime and change the queue in
qmail-smtpd.rules/cdb from /var/qmail/bin/simscan to
/var/qmail/bin/qmail-tail and fill qmail-tail with

exec /usr/local/bin/qmail-disclaimer.

qmail-disclaimer is a script that utilize altermime. After running the
queue into altermime, I get new file consists of the real email with
disclaimer below it (within the same email). So far so good, but I don't
know how to put it back to simscan. I tried

     cat the-file > /var/qmail/bin/simscan

but it won't work (I believe it's not how simscan working). Is there a way
to do that? Thanks.

Regards,
Mario


!DSPAM:4fcf240434204702284232!

Re: [SPAM] No title in simscan (qmail smtpd)

FC Mario Patty — 2012-04-14T11:36:45

Guys,

I think if we want to see the subject inside simscan's log, we *must* use
--enable-spam-auth-user=y (we have no option at all). I found it out by
reading through simscan.c file on simscan-1.4.0 installation folder. So I
modify this simscan.c and add a new function name create_subject (to
extract the subject from emails without getting them scanned by
spamassassin) and call it before log_message("RELAYCLIENT", Subject, 0) ;
this Subject parameter was '-' before. (Sorry, I don't know how to call
those ifdef or ifndef parts in simscan.c - I haven't finished chapter 1 of
mr Dennis's book.)

Now I can get the subject listed in smtpd's log file. I attach "my version"
of simscan.c and setup.sh (to configure simscan) so people can use as
needed. But I hope anyone with a more advanced experience on c-programming
may check this modification to catch any bugs within the code. Last word,
thanks for your support.

Best regards,
Mario









On Tue, Apr 10, 2012 at 10:03 AM, FC Mario Patty <fcmario76-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:



!DSPAM:4f89615334841333113101!

[SPAM] Re: spam reject on scores less than configured

Eric Sisolak — 2012-04-11T18:10:09


I ran into this issue as well, though I discovered that the "rejected" message
was in fact delivered.  I dug through the code and found that this code was the
problem (in check_spam() in simscan.c):

#ifdef SPAM_HITS
  if ( PerDomainHits==1 && ( SpamHits >= PDHits ) ) {
#ifdef ENABLE_DROPMSG
    log_message("SPAM DROPPED", Subject, 1);
#else
    log_message("SPAM REJECT", Subject,1);
#endif
    return(1);
  } else if ( PerDomainHits==0 && ( SpamHits >= SPAM_HITS ) ) {
#ifdef ENABLE_DROPMSG
    log_message("SPAM DROPPED", Subject, 1);
#else
    log_message("SPAM REJECT", Subject,1);
#endif
    return(1);
  }

  if (SpamHits >= SPAM_HITS) {
#ifdef ENABLE_DROPMSG
    log_message("SPAM DROPPED", Subject, 1);
#else
    log_message("SPAM REJECT", Subject,1);
#endif
  } else {
    log_message("CLEAN", Subject,1);
  }

If you are using PerDomain and Spamhits and a messages is score between the
compiled spam-hits value and that domains current threshold, then it will match
the last if (if (SpamHits >= SPAM_HITS)) and log as rejected.  However there is
not return(1); so it is still delivered.

Since I could not imagine a scenario where we would want to hit that condition
(PerDomainHits should always be 0 or 1), I made a patch to remove it.  It
functions the same, but will correctly log that it delivered the message.

Eric Sisolak

## Begin pd_reject.patch
This patch addresses a bug where mail is accepted but logged
as rejected when using per domain spam hits.  See:
http://comments.gmane.org/gmane.mail.qmail.simscan/3875

Created for Simscan 1.4.0

Eric Sisolak


--- simscan.c.orig      2012-04-11 11:30:06.000000000 -0400
+++ simscan.c   2012-04-11 12:16:55.000000000 -0400
< at >< at > -1376,14 +1376,6 < at >< at >
     log_message("SPAM REJECT", Subject,1);
 #endif
     return(1);
-  }
-
-  if (SpamHits >= SPAM_HITS) {
-#ifdef ENABLE_DROPMSG
-    log_message("SPAM DROPPED", Subject, 1);
-#else
-    log_message("SPAM REJECT", Subject,1);
-#endif
   } else {
     log_message("CLEAN", Subject,1);
   }






!DSPAM:4f85ca3d34842027615567!

Re: [SPAM] No title in simscan (qmail smtpd)

FC Mario Patty — 2012-04-10T03:03:19

Dear Tom,

--disable-spam-auth-user=y gives me this error

         configure: error: invalid feature name: spam-auth-user=y

but --disable-spam-auth-user (without =y) works well with configure. I
restart all the services but the result is just the same. Is there anything
I can play with? Thank you for your help.

Regards,
Mario


On Tue, Apr 10, 2012 at 9:07 AM, Tom Collins <tom-lnEA/wrDJtNWk0Htik3J/w< at >public.gmane.org> wrote:



!DSPAM:4f83a2fb34841312011002!

Re: [SPAM] No title in simscan (qmail smtpd)

Tom Collins — 2012-04-10T02:07:56

It looks like the help for configure is wrong.  The options should be --enable-spam-auth-user and --disable-spam-auth-user.

Give that a shot and see if you get the desired behavior.

-Tom


On Apr 9, 2012, at 2:09 AM, FC Mario Patty wrote:


!DSPAM:4f83960a34841409462748!

[SPAM] No title in simscan (qmail smtpd)

FC Mario Patty — 2012-04-09T09:09:56

Guys,

I would like to know how to include the title of the email inside simscan
(smtpd) log file (current). I have 2 simscans running in my mail server: 1
to handle emails come through port 25, and 1 to handle emails come through
port 587 (for roaming users). With the first installation, the title get
listed in smtpd log, but on the second installation, I get only '-' (a
dash). The different between both installation is:

--enable-spam-auth-user=y on the first installation of simscan and
--enable-spam-auth-user=n on the second one.

They both use the same simcontrol file (on different control directory):
        :clam=yes,spam=yes

Here is the log from the second installation:
< at >400000004f82a34028fc4d4c
simscan:[8241]:RELAYCLIENT:0.9265s:-:182.0.220.162:user< at >mydomain
:user< at >mydomain

Thank you very much.

Best regards,
Mario


!DSPAM:4f82a76a34847656313713!

Re: Bug in simcsan.c (empty From/To)

Tom Collins — 2011-12-02T18:18:47

If no one emailed you back, here's my quick thought.

Look at the definition of RcptTo -- it's probably an array of character buffers to hold multiple email addresses.

If the first character ([0]) of the first address (RcptTo[0]) is a null byte ('\0'), then the address is empty.

That code shouldn't generate bounces for null senders, since there will be at least one email address in RcptTo[].

-Tom


On Nov 7, 2011, at 11:19 AM, Lutz Petersen wrote:



!DSPAM:4ed9168f32711403019570!

Bug in simcsan.c (empty From/To)

Lutz Petersen — 2011-11-07T19:19:49

Please give me a hint - isn't this wrong in simscan.c code ?
This produces bounces/failures if there is no from but an recipient
(instead of bounce only if there are neither both):


if ( MailFrom[0] == 0 && RcptTo[0][0] == 0 ) {
    if ( DebugFlag > 0 ) {
      fprintf(stderr, "simscan: no envelope information, deferred exit\n");
    }
    exit_clean(EXIT_454);
  }

I mean this:

Re: ripmime

Trey Nolen — 2011-10-14T15:19:32


On 10/14/2011 09:39 AM, Joshua Megerman wrote:

The problem with that is that only some emails are triggering this
segfault and I'm not sure which emails are doing it.  I'll try to run
some testing to see if I can figure out which ones are causing it.

Trey Nolen


!DSPAM:4e98530832716501812433!

Re: ripmime

Joshua Megerman — 2011-10-14T14:39:01

The only thing I can suggest is that you try running qmail-smtpd via
strace, although this might break simscan anyway (I know it used to break
qscanq when I used that, I can't remember if I've done it since I switched
to simscan).  Just insert '/usr/bin/strace -o/tmp/smtp.log -ff' before the
call to qmail-smtpd in your run file (substitute paths as needed) and then
try to send email - you'll get a metric buttload of logging (one file per
process, named /tmp/smtp.log.<PID>) that should hopefully let you see
where it's blowing up.  I just recommend doing it on a non-production port
so you don't end up messing up real mail :)

If that doesn't work, I'm out of ideas.

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
simscan-qx7goLWZlwtVohPdPC+VWJowlv4uC7bZ< at >public.gmane.org


!DSPAM:4e98498732717361575183!

Re: ripmime

Trey Nolen — 2011-10-14T14:00:28


On 10/14/2011 08:43 AM, Joshua Megerman wrote:

I thought that might be the issue, too, so we temporarily removed all
softlimits on qmail.   We are still seeing the same issue.  Could there
be a limit on ripmime itself that I'm not aware of?  Or clamd?

Trey Nolen


!DSPAM:4e98407f32711630515430!

Re: ripmime

Joshua Megerman — 2011-10-14T13:43:31

Try upping your softlimit for qmail-smtpd.  I don't remember if I've seen
segfaults from it being to low, but I've definitely seen scan failures,
and as clamav and spamassassin have gotten more complicated, their memory
footprints have grown somewhat.  If that doesn't help, try reinstalling
ripmime and/or and libraries it depends on ('ldd /path/to/ripmime' and go
from there).

Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
  - Layman's translation of the Laws of Thermodynamics
simscan-qx7goLWZlwtVohPdPC+VWJowlv4uC7bZ< at >public.gmane.org


!DSPAM:4e983c8532711367318120!

ripmime

Trey Nolen — 2011-10-14T13:21:24

I'm not sure if this is the best place to ask this, so forgive me if it 
is a little OT.

We're using simscan to call clamd and spamd, and lately (in the last 
week or so), we've been seeing a lot of these errors in the logs:

Oct 14 04:02:15 mail kernel: [9781102.945747] ripmime[14308]: segfault at 18a085e8 ip 08054588 sp bff61530 error 4 in ripmime[8048000+1d000]
Oct 14 04:02:41 mail kernel: [9781128.982628] ripmime[14585]: segfault at 18aca5e8 ip 08054588 sp bf8855a0 error 4 in ripmime[8048000+1d000]
Oct 14 04:03:47 mail kernel: [9781195.177554] ripmime[15062]: segfault at 18ab65e8 ip 08054588 sp bff9f420 error 4 in ripmime[8048000+1d000]
Oct 14 04:04:54 mail kernel: [9781261.239391] ripmime[15475]: segfault at 18aa75e8 ip 08054588 sp bfba78e0 error 4 in ripmime[8048000+1d000]
Oct 14 04:06:00 mail kernel: [9781327.084212] ripmime[16041]: segfault at 1898b5e8 ip 08054588 sp bfb065a0 error 4 in ripmime[8048000+1d000]
Oct 14 04:07:06 mail kernel: [9781393.238548] ripmime[16732]: segfault at 18ab45e8 ip 08054588 sp bf8a2770 error 4 in ripmime[8048000+1d000]
Oct 14 04:07:32 mail kernel: [9781419.179641] ripmime[17072]: segfault at 18a815e8 ip 08054588 sp bffa3820 error 4 in ripmime[8048000+1d000]


The ripmime version is v1.4.0.9 and it is running on Ubuntu 10.04.3.


Does anyone have an idea of what may be causing this?


Trey Nolen




!DSPAM:4e98375732711038919540!

Re: Private simscan rewrite - anyone interested ?

Matt Simpson — 2011-10-06T23:56:49

On Oct 6, 2011, at 5:18 PM, Lutz Petersen wrote:



It seems like a logical extension of the capability that is already there.  Currently, I can use simcontrol to tell simscan to apply different thresholds to specific domains, or to specific recipients within a domain.  I'm just suggesting enhancing that to apply a specific threshold to a group of recipients whose address has a specific prefix.  That's not much different from using the qmail address extension to specify different thresholds, which it already does (according to the doc).
!DSPAM:4e8e406d32711256419368!

Re: Private simscan rewrite - anyone interested ?

Lutz Petersen — 2011-10-06T21:18:17


 I'm not sure, is simscan really the rigth place to do what you described ?

 Thought about some domain/recipient regex too, but I saw no really need (and
 feared the work coding that ;-)). Instead, I made some more corrections (e.g.
 dspam results were wrong and such simple things). I'll make a version the
 next days (weeks) with is complete and also documented and then post the url
 where to download.

 If anyone has other ideas about what feature/.. can be added please post..


!DSPAM:4e8e1b2b32711997266428!