http://blog.gmane.org/gmane.mail.bogofilter.general hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://permalink.gmane.org/gmane.mail.bogofilter.general/11647 <pre> I don't know about the particular packaging/distribution of bogofilter you're using, but I'm guessing it didn't install the user-contributed scripts (of which trainbogo.sh is one). If you download the current bogofilter source tarball, the contrib files will be found inside it: http://sourceforge.net/projects/bogofilter/files/bogofilter-current/ Note that you seem to be running a very old version of bogofilter, so the newer scripts may not work as expected with your configuration. (However, trainbogo.sh seems to have been last modified in 2004, so perhaps it will be OK.) Maybe you can find the old version 1.0.2 of bogofilter source somewhere. Or migrate your setup to the current version of bogofilter. _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Matt Garretson 2012-05-21T15:23:38 http://permalink.gmane.org/gmane.mail.bogofilter.general/11646 <pre>For a very long time I've been training bogofilter with the command bash /usr/share/bogofilter/contrib/contrib/trainbogo.sh -c -H /home/anne/Maildir/.INBOX.Bogotrain_ham/cur/ -S /home/anne/Maildir/.INBOX.Bogotrain_spam/cur However, since I installed Scientific Linux 6 I find that I don't appear to have that file. I'm running bogofilter-1.0.2-6.el6.x86_64. I've got quite a store of training emails now. Could you please help me get back on track? Thanks Anne _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Anne Wilson 2012-05-21T13:45:07 http://permalink.gmane.org/gmane.mail.bogofilter.general/11645 <pre>Continued training is the solution to these sorts of issues. No special fixes. Just train. It's like training a pet. If your dog doesn't roll over on command, keep training. If your bogofilter marks some spams as hams or vice versa, keep training. Tom On 10/24/2011 12:21 AM, Tweeks wrote: _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Thomas Anderson 2011-11-02T22:05:28 http://permalink.gmane.org/gmane.mail.bogofilter.general/11644 <pre>For those of you who use bogofilter-milter.pl, the Milter implementation of bogofilter filtering I wrote, there is a new version available at http://stuff.mit.edu/~jik/software/bogofilter-milter/bogofilter-milter.pl.txt &lt;http://stuff.mit.edu/%7Ejik/software/bogofilter-milter/bogofilter-milter.pl.txt&gt; which you should replace the version you're using with (David, please take this new version for the contrib directory in the bogofilter distribution). This fixes a bug in the handling of incoming messages larger than one million bytes, or whatever you change the $MAX_INCORE_MSG_LENGTH limit to in the script, if you change it. In particular, the bug caused messages at least this large to be "chunked" into blocks of that many bytes, with only the last chunk preserved to be fed into bogofilter and/or archived in $archive_mbox or $ham_archive_mbox. I am grateful for Stephen Davies for not only pointing out the bug to me, but also doing a great deal of troubleshooting on it and pointing me right at the problematic code. Even with his help, however, I had to stare at the code for several hours before I figured out the problem. I hate bugs like that. :-/ As always, please let me know if you have any questions, comments or suggestions about bogofilter-milter. Regards, Jonathan Kamens _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Jonathan Kamens 2011-11-01T02:41:30 http://permalink.gmane.org/gmane.mail.bogofilter.general/11643 <pre>Q1: Bogofilter keeps all its tokens in a database. The default database is named wordlist.db. Individual IP address are kept, but not address ranges, i.e. the wordlist will save 1.2.3.4 but will not save 1.2.3/24. Q2: Read the FAQ. Reclassifying messages is simple. bogotune is complex. As a starting point, you'll need a set of 5000 (or more) ham and a comparable number of spam. If the FAQ doesn't have enough info, post again. I need to head out. Hope this helps. David On Mon, 24 Oct 2011 04:21:20 +0000 (UTC) Tweeks wrote: _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-10-24T11:35:49 http://permalink.gmane.org/gmane.mail.bogofilter.general/11642 <pre>Couple questions... Q1) I just started using bogofilter on my Kmail install, and it's flagging all mail coming from my mail server as spam. My server is not blacklisted anywhere, but IS in an old bogon IP range (98.129) that was released for use by the IANA some time back in 2006 (098/8)... and my fear is that my server's x-bogon is still being flagged by bogofilter as a bogon. If this is not the case, someone please show me where any bogon lists are kept and how I can verify that my MTA's primary IP is not in such a list. Q2) If I accidentally marked some HAM as spam.. how do I fix this? I can't find any good tutorials or anything on using bogotune (if that's eaven the right tool). Thanks! Tweeks _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Tweeks 2011-10-24T04:21:20 http://permalink.gmane.org/gmane.mail.bogofilter.general/11641 <pre>On Sat, 3 Sep 2011 06:47:14 -0700 (PDT) Joseph Harth wrote: Hi Joseph, I think what you've done may be sub-optimal, possibly not even useful. Part of the bayesian nature of bogofilter is to know how often words appear in spam and ham. In particular, bogofilter likes to know that "xxx" appears in y% of spam messages and in z% of ham messages. With these numbers, the appearance of "xxx" can be judged as good or bad. Consider the following .MSG_COUNT 1000 100 xxx 500 90 These values indicate that 1000 spam have been processed and 500 of them had xxx, for a 50% score 100 ham with 90 having xxx, for a score of 90%. With numbers like the above, the appearance of xxx indicates the message is more likely good than bad. With the wordlist you've created, run "bogoutil -d wordlist.db" to display your wordlist as text and see if you like the results. Alternatively you can test simple messages using "echo" or file indirection, for example echo this is a test | bogofilter -H -v bogofilter -v &lt; test_message.txt Of course, you could just nuke the wordlist you've created and train bogofilter with a bunch of saved ham and spam messages. That would ensure that bogofilter's wordlist is structured the way bogofilter expects it to be. Regards, David _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-09-03T21:11:28 http://permalink.gmane.org/gmane.mail.bogofilter.general/11640 <pre>On Fri, 2 Sep 2011 09:54:56 -0700 (PDT) Joseph Harth wrote: "bogoutil -l wordlist.db token_file.txt" will load the entries in token_file.txt into wordlist.db The format of token_file.txt is: token1 spam_count ham_count date token2 spam_count ham_count date .MSG_COUNT spam_messages ham_messages date Where spam_count is the number of times the token has been seen in spam messages and ham_count is the count for ham messages. Where spam_messages and ham_messages are the number of spam and ham messages processed in building the wordlist. If popfile doesn't have that information, you'll have to make up this information. Note: a reasonable estimate for spam_messages might be double (2x) the largest spam_count and 2x ham_count for ham_messages. The date field is the date the tokens are entered into the wordlist. The format is YYYYMMDD. You'll have to do some experimenting to determine the best way to convert your spam and ham files to the format needed by bogoutil. The following lines are (approx) what you'll need: sort &lt; spam.txt | uniq -c | awk '{print $2 $1 0}' | bogoutil -l wordlist.db sort &lt; ham.txt | uniq -c | awk '{print $2 0 $1}' | bogoutil -l wordlist.db bogoutil -d wordlist.db I've written the above commands "off the cuff" so you may have to tweak them a bit before they actually work. I've also left out .MSG_COUNT Hope this helps, David _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-09-03T12:20:07 http://permalink.gmane.org/gmane.mail.bogofilter.general/11639 <pre>Am 31.08.2011 12:05, schrieb RW: Sorry for that, fixed in FreeBSD's port bogofilter-1.2.2_3, -tc-1.2.2_2 and -sqlite-1.2.2_2. Thanks for posting the fix. _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Matthias Andree 2011-08-31T10:32:20 http://permalink.gmane.org/gmane.mail.bogofilter.general/11638 <pre>On Tue, 23 Aug 2011 20:53:00 -0400 David Relson wrote: I notice that trainbogo.sh still doesn't work on FreeBSD. All that's needed is a ports patch file to change md5=$(printf "%s" "${rndseed}${i}" | md5sum | sed "s/ -//") to md5=$(printf "%s" "${rndseed}${i}" | md5) _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> RW 2011-08-31T10:05:20 http://permalink.gmane.org/gmane.mail.bogofilter.general/11637 <pre>On Tue, 23 Aug 2011 17:08:58 +0100 Anne Wilson wrote: As you say, training continues. Bogofilter is nine years old and I've been running it for a few months less than that -- and I'm still training daily with unsures and the occasional false negative. _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-08-24T00:53:00 http://permalink.gmane.org/gmane.mail.bogofilter.general/11636 <pre>Thanks David. That solves the problem :-) Training continues, as always. Anne _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Anne Wilson 2011-08-23T16:08:58 http://permalink.gmane.org/gmane.mail.bogofilter.general/11635 <pre>My mail server has been upgraded to CentOS 6. Bogofilter is installed and appears to be tagging correctly. However, I'm having problems with retraining - bash /usr/share/bogofilter/contrib/contrib/trainbogo.sh -c -H /home/anne/Maildir/.INBOX.Bogotrain_ham/cur/ -S /home/anne/Maildir/.INBOX.Bogotrain_spam/cur/ bash: /usr/share/bogofilter/contrib/contrib/trainbogo.sh: No such file or directory On checking I find that there is no /usr/share/bogofilter directory, and no amount of searching is finding the scripts. Unfortunately I don't appear to have archived the messages where you originally helped me set this up. How can I get this working again? Anne </pre> Anne Wilson 2011-08-23T09:31:03 http://permalink.gmane.org/gmane.mail.bogofilter.general/11634 <pre>easily. Your /home/admin/.bogofilter can be used using /home/admin/.bogofilter &lt; MSG display the on its own. through modified it your procmail really like it. Gyepi Sam Gyepi Sam to the new. I Spam, or Unsure, /home/admin/.bogofilter /usr/include/sysexits.h is the same get the 75 error eventually die. 13:06:27 -0400 errors ----- #admin" #admin"... Message could from queue /home/admin/.bogofilter/wordlist.db ################################################ command line and Following the K.I.S.S. rule, and using Kmail which allows for a filter, here is my overnight script: --------------------------------------- /usr/local/bogofilter-1.2.2/contrib/trainbogo.sh -c -H /home/safe/.kde/share/apps/kmail/mail/Ham/cur/ -S /home/safe/.kde/share/apps/kmail/mail/Spam/cur/ ----------------------------------------- The above is all one long sentence. </pre> John Culleton 2011-07-07T13:14:01 http://permalink.gmane.org/gmane.mail.bogofilter.general/11633 <pre> Hi Bogofilterers, On Wed, 6 Jul 2011, at 23:13, David Relson wrote: Thanks for giving me a chance to add my 2c :) Additionally, I recommend to clarify in which context the scripts are running. As we are talking procmail here one might add LOG=`id` to the procmailrc. This runs the command _id_ and appends the output to the procmail logfile. It shows the current user and the groups it belongs to. Say, the script is run as user postfix, then one might want to test bogofilter with something like su postfix --shell /bin/sh -c "/usr/local/bin/bogofilter -u (...)" HTH, </pre> Andreas Schamanek 2011-07-07T08:20:03 http://permalink.gmane.org/gmane.mail.bogofilter.general/11632 <pre>On Wed, 6 Jul 2011 06:43:13 -0700 (PDT) Doug wrote: Hello Doug, The short answer is "Yes". The long answer is "Being summer it's travel and vacation season and also cycling season. For example, in June, my wife, daughter, and I pedaled our bikes across Michigan - a distance of 283 miles in 6 days (http://www.lmb.org/palm). All these activities impact support, so support times tend to be longer." Regards, David _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-07-07T03:30:21 http://permalink.gmane.org/gmane.mail.bogofilter.general/11631 <pre>Hello Jonathan, Bogotune is complex and tricky. For its operation, it requires a set of messages (as you're aware) and the messages must follow some constraints that are implemented in the code, but not readily described in words. The information you provide isn't sufficient to diagnose the actual problem. Bogotune's "-M" switch allows conversion of messages to the "message count" format, i.e. tokens that are sorted and have duplicates removed. This has the effect of anonymizing a message and obscuring its meaning. Effectively this removes sensitive meaning from the messages. If you'd care to use the "-M" switch and send me (off-list) a copy of the resulting ham and spam files, I'll run bogotune and see if I can determine why it's unhappy with your corpora. Be warned that it's summer travel season so I may not be able to look at your files right away. Regards, David On Wed, 22 Jun 2011 21:23:01 -0400 Jonathan Kamens wrote: _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-07-07T03:23:51 http://permalink.gmane.org/gmane.mail.bogofilter.general/11630 <pre>Hello Doug, Bogofilter can be tested from the command line quite easily. Your postfix setup runs bogofilter using command: /usr/local/bin/bogofilter -u -e -p -l -d /home/admin/.bogofilter Assuming your message is in file MSG, the same command can be used using MSG as redirected input, i.e. /usr/local/bin/bogofilter -u -e -p -l -d /home/admin/.bogofilter &lt; MSG adding an "echo $?" command after running bogofilter will display the exitcode. The above will let you verify that bogofilter is working on its own. I, too, use procmail -- a script developed years ago through trial and error and the advice of others. I haven't modified it in a long time and haven't the expertise to help debug your procmail script. HTH, David On Sat, 18 Jun 2011 20:55:02 -0700 (PDT) Doug wrote: _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> David Relson 2011-07-07T03:13:23 http://permalink.gmane.org/gmane.mail.bogofilter.general/11629 <pre>Is anyone listening on this list? I see a couple of questions including mine have gone unanswered.   _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Doug 2011-07-06T13:43:13 http://permalink.gmane.org/gmane.mail.bogofilter.general/11628 <pre>I'm trying to tune bogofilter and can't get bogotune to work reliably. Below is a transcript. Why is bogotune having trouble locking onto good settings, when I've got 3121 ham messages and 5818 spam messages? Also, the settings I'm currently using, which were generated by an earlier, successful run of bogotune about four months ago, are working just fine, with my spam detection rate at above 99%. I've noticed a few more false positives than I prefer when I receive email from new entities, which is why I'm trying to retune. Before bogotune was having this particular problem, it was having another one... it kept reporting that it couldn't read my wordlist.db. That problem went away after I used bogoutil to remove tokens from the word list that haven't been seen in 180 days, a maintenance task I do periodically to keep the size of the word list reasonable. I've also recently dumped and reloaded the word list into a new file, which brought its size down from 9MB to 3MB, but that didn't help bogotune. Oh, and I should mention that I actively, regularly retrain bogofilter with ham and spam, including fixing any mischaracterizations, so my word list is extremely accurate. Thanks for any advice you can provide. jik $ bogotune -v -T 0 -n /tmp/notspam -s /tmp/bogospam Reading /home/jik/.bogofilter/wordlist.db Reading /tmp/notspam 3121 messages Reading /tmp/bogospam 5818 messages wordlist's ham to spam ratio is 0.9 to 1.0 Calculating initial x value... Initial x value is 0.481636 Recommended db cache size is 11 MB Too few high-scoring non-spams in this data set. At target 1, cutoff is 0.037362. False-positive target is 1 (cutoff 0.037362) Performing final scoring: Spam... Non-Spam... 0.000000 0.037362 0.227615 0.033571 0.427900 0.030283 0.437421 0.025977 0.513999 0.024112 0.546457 0.020030 0.573949 0.010595 0.574819 0.003866 0.578010 0.003672 0.613205 0.000913 ### The following recommendations are provisional. ### Run bogotune with more messages when possible. Recommendations: ---cut--- db_cachesize=11 robs=0.0178 min_dev=0.020 robx=0.481636 sp_esf=1.000000 ns_esf=1.000000 spam_cutoff=0.033571 # for 0.05% fp (1); expect 0.02% fn (1). #spam_cutoff=0.025977 # for 0.10% fp (3); expect 0.02% fn (1). #spam_cutoff=0.010595 # for 0.20% fp (6); expect 0.02% fn (1). ham_cutoff=0.011 ---cut--- The small number and/or relative uniformity of the test messages imply that the recommended values (above), though appropriate to the test set, may not remain valid for long. Bogotune should be run again with more messages when that becomes possible. Tuning completed. _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Jonathan Kamens 2011-06-23T01:23:01 http://permalink.gmane.org/gmane.mail.bogofilter.general/11627 <pre>I have been running Bogofilter for several years and really like it. It works very well. I brought up bogofilter on a new system - This is the new system - slate:/usr/local/bin # bogofilter -V bogofilter-sqlite version 1.2.2     Database: SQLite 3.7.5 Copyright (C) 2002-2010 David Relson, Matthias Andree Copyright (C) 2002-2004 Greg Louis Copyright (C) 2002-2003 Eric S. Raymond, Adrian Otto, Gyepi Sam And this is the old system, I am currently using - bogofilter-sqlite version 1.2.2     Database: SQLite 3.7.3 Copyright (C) 2002-2010 David Relson, Matthias Andree Copyright (C) 2002-2004 Greg Louis Copyright (C) 2002-2003 Eric S. Raymond, Adrian Otto, Gyepi Sam I copied the database (wordlist.db) from the old system to the new. I am using the same procmail script. # filter mail through bogofilter, tagging it as Ham, Spam, or Unsure, # and updating the wordlist # :0fw: bogofilter.lock # | /usr/local/bin/bogofilter -u -e -p -l -d /home/admin/.bogofilter # if bogofilter failed, return the mail to the queue; # the MTA will retry to deliver it later # 75 is the value for EX_TEMPFAIL in /usr/include/sysexits.h # :0e # { EXITCODE=75 HOST } # file the mail to spam-bogofilter if it's spam. # :0: # * ^X-Bogosity: Spam, tests=bogofilter # IN.spam-bogofilter # file the mail to unsure-bogofilter # if it's neither ham nor spam. # :0: # * ^X-Bogosity: Unsure, tests=bogofilter # IN.unsure-bogofilter All lines are commented out because of it failing. This is the same script that works fine on the original system. On the new system and with this script operational I get the 75 error on all emails and they go back to the queue and eventually die. They give this error message - The original message was received at Mon, 13 Jun 2011 13:06:27 -0400 from localhost [127.0.0.1] with id p5DH6PuD009336    ----- The following addresses had permanent fatal errors ----- "|IFS=' ' &amp;&amp; exec /usr/bin/procmail -f- || exit 75 #admin"     (reason: Deferred)     (expanded from: admin)    ----- Transcript of session follows ----- "|IFS=' ' &amp;&amp; exec /usr/bin/procmail -f- || exit 75 #admin"... Deferred: prog mailer (/bin/sh) exited with EX_TEMPFAIL Message could not be delivered for 5 days Message will be deleted from queue Bogoutil seems to work with the database fine - slate:/usr/local/bin # bogoutil -H /home/admin/.bogofilter/wordlist.db Histogram score   count  pct  histogram 0.00   320367 11.51 ####### 0.05     3138  0.11 # 0.10     2899  0.10 # 0.15     1198  0.04 # 0.20     5535  0.20 # 0.25     1740  0.06 # 0.30     2901  0.10 # 0.35     1528  0.05 # 0.40     2347  0.08 # 0.45     1531  0.05 # 0.50     2328  0.08 # 0.55     2112  0.08 # 0.60     2513  0.09 # 0.65     2485  0.09 # 0.70     2538  0.09 # 0.75     3088  0.11 # 0.80     3580  0.13 # 0.85     4422  0.16 # 0.90     5399  0.19 # 0.95  2412035 86.65 ################################################ tot   2783684 hapaxes:  ham  186718 ( 6.71%), spam 1829218 (65.71%)    pure:  ham  315835 (11.35%), spam 2406205 (86.44%) How can I check that bogofilter is working from the command line and determine exactly where my error is? Doug _______________________________________________ Bogofilter mailing list Bogofilter&lt; at &gt;bogofilter.org http://www.bogofilter.org/mailman/listinfo/bogofilter </pre> Doug 2011-06-19T03:55:02 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.mail.bogofilter.general