gmane.mail.exim.devel

Re: Embedding Python

Phil Pennock — 2013-05-20T21:38:32

That's something handled by the Python top-level wrapper, which embeds
the Python library.

See Python's Modules/main.c, RunMainFromImporter() (which does pretty
much what Todd is doing), called from Py_Main() below.

We're not invoking programs, we're not hitting __main__ invocation,
we're loading the code as a module to make functions available for later
calling.

-Phil

Re: Embedding Python

Viktor Dukhovni — 2013-05-20T21:22:35


IIRC Python automatically adds the directory containing a script
to the search path for modules imported by *that* script.  So it
should not be necessary to globally add the containing directory
to the search path, this can make other code more fragile by breaking
namespace boundaries.  So if I am not mistaken, the system paths
should not be modified.

Re: Embedding Python

Phil Pennock — 2013-05-20T21:10:41

I looked *very* briefly with:

  git diff e2658fff tlyons/master_volatile_python

It looks good to me.


This should be called out very clearly, perhaps even in the security
section too, as it affects what code will be loaded into a root-run
process and folks should know that they will be implicitly trusting
everything in a directory from which they load code, not just the file
that they load.

It *might* also be worth making something that can be disabled.  Folks
who want it, will love it.  Folks who don't want it ... will not.
Perhaps instead, just make a third Exim option?
python_sys_path_prepend?

  python_sys_path_prepend = /etc/exim/python
  python_startup = /etc/exim/python/startup.py

Make sense?  I suspect, if you do it that way, the second can even
become "python_startup = startup.py", using the newly adjusted sys.path ?

I'm currently in SF, will be back in Pittsburgh early Saturday morning
and *might* have time to catch up on some Exim stuff early next week.

-Phil

Re: Embedding Python

Todd Lyons — 2013-05-20T18:02:14

Figured out the problem with using the list method for path
manipulation.  My latest code is at:
http://git.exim.org/users/tlyons/exim.git

So far, the code can do the following:
1. If python script alone is specified, exim attempts to load it from
python system paths.
2. If full path to python script is specified, it splits the path and
adds it to the system paths, then attempts to load it.
3. Expansion can call a function (ignores args)
4. Expansion can handle a returned string response.
5. Expansion can detect a returned {int,long_int,float}, but not
really do anything with it.

The following test code:
def string_output():
        # Just a simple string being returned
        return "Just a simple string"

def whole_number_output():
        # Just a simple number being returned
        return 5

def decimal_number_output():
        # A decimal number being returned
        return 8.8382912

def negative_number_output():
        return -800


Produces this output:
[exim-build< at >tlyons ~/projects/exim/src (

Re: logging of fakereject

Ted Cooper — 2013-05-13T13:44:09

I don't know if it's a normal thing or not, but none of my servers
deliver fake rejected emails to the same place as normal ones; they
always run through their own router/transport which makes them quite
obvious.

You can also use logwrite or similar to write your own line to the log
file which will have the same message id.

Changing the logfile format is always an issue. It's like changing a
well known and stable API. It certainly shouldn't break something as
fundamental as "<=".

Re: logging of fakereject

Jeremy Harris — 2013-05-13T12:22:04

Well, we could do something like " <=F " rather than " <= ",
perhaps, if enought people care.  Or add a doc note.

Re: logging of fakereject

Graeme Fowler — 2013-05-13T08:21:09

Coincidentally I'm using fakereject to drip-feed sa-learn with messages
we have classified as spam in other ways.

The message is logged in its entirety, from "<=" to "Completed".

I've put in a separate logwrite line to log it to the reject log:

   accept condition   = ${if eq{$acl__c_test}{true}}
          logwrite    = :main,reject: Message rejected (but fed to
sa-learn): LBORO-DENY-LOCAL - Local Listing <other stuff snipped>
          control     = fakereject/Your message has been rejected due to
the sending host ($sender_host_address) having the following local
listings: <other stuff snipped>
          control     = queue_only

But aside from the logwrite line, the message logging is perfectly
normal.

I'm not 100% sure what you could do to change that, really...

Graeme

[Bug 425] Put captured subexpressions of regex and mime_regex ACL conditions in $1, $2 etc.

Jeremy Harris — 2013-05-12T17:40:41

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=425

Jeremy Harris <jgh146exb< at >wizmail.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #622 is|0                           |1
           obsolete|                            |




--- Comment #3 from Jeremy Harris <jgh146exb< at >wizmail.org>  2013-05-12 18:40:40 ---
Created an attachment (id=626)
 --> (http://bugs.exim.org/attachment.cgi?id=626)
Mk. 2: supports mime_regex also.

I've noted an issue:  this may be regarded as an incompatible change as it
modifies variables which were previously untouched.

logging of fakereject

Jeremy Harris — 2013-05-12T15:02:33

Am I right that a fakereject still gets a "<=" line logged?
Should we make it distinguishable in some way?

Re: Embedding Python

Todd Lyons — 2013-05-09T13:52:21

I've been experimenting with this.  So far I cannot get path
manipulation to work using the list methods.  I can only get it to
work using string methods.

Doing this works:

char *path, *newpath;
path = Py_GetPath();
newpath = (char *)string_sprintf("%s:%s", syspath,"/etc/exim");
PySys_SetPath(newpath);
/* Now check it */
path = Py_GetPath();

And using the list method (again, very simplified, no error checking
yet) does not:

char *path;
PyObject *syspath, *pName;
syspath = PySys_GetObject("path");
pName = PyString_FromString("/etc/exim");
PyList_Insert(syspath, 0, pName);
PySys_SetObject("path", syspath);
/* Now check it */
path = Py_GetPath();

When I print out the path string, the "/etc/exim" is not in it.


It will depend upon if Py_Initialize() does the site loading (which in
my googling seems to imply that it does).  To answer a second part of
your question, my plan was to take the python_startup setting (for
example "/etc/exim/exim_python.py"), split the path and filename, then
load the module "exi

[Bug 1353] New: Automated checks for unsorted optionlists

Nigel Metheringham — 2013-05-08T12:58:35

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=1353
           Summary: Automated checks for unsorted option lists
           Product: Exim
           Version: N/A
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: bug
          Priority: medium
         Component: Release process
        AssignedTo: nigel< at >exim.org
        ReportedBy: nigel< at >exim.org
                CC: exim-dev< at >exim.org


Exim option lists (in the exim C code) must be in a lexicographically sorted
order.
If they are not sorted bad things can happen - see bug #1352

We need some automated checking to check all of the sets of option lists, and
alert should an option be inserted in a position that makes the option list
unsorted.

[Bug 1352] Exim configuration error: main option"dns_retry" unknown

Todd Lyons — 2013-05-08T12:49:49

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=1352

Todd Lyons <tlyons< at >ivenue.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tlyons< at >ivenue.com
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED




--- Comment #2 from Todd Lyons <tlyons< at >ivenue.com>  2013-05-08 13:49:48 ---
Fix commited and pushed.

[Bug 1352] Exim configuration error: main option"dns_retry" unknown

Todd Lyons — 2013-05-08T12:48:43

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=1352




--- Comment #1 from Todd Lyons <tlyons< at >ivenue.com>  2013-05-08 13:48:40 ---
On Wed, May 8, 2013 at 3:01 AM, Dmitry Isaikin <isaikin-dmitry< at >yandex.ru>
wrote:

Re: [Bug 1352] New: Exim configuration error: main option "dns_retry" unknown

Todd Lyons — 2013-05-08T12:48:16

[Bug 1352] New: Exim configuration error: main option "dns_retry" unknown

Dmitry Isaikin — 2013-05-08T10:01:52

------- You are receiving this mail because: -------
You are on the CC list for the bug.

http://bugs.exim.org/show_bug.cgi?id=1352
           Summary: Exim configuration error: main option "dns_retry"
                    unknown
           Product: Exim
           Version: N/A
          Platform: Other
        OS/Version: Linux
            Status: NEW
          Severity: bug
          Priority: medium
         Component: General execution
        AssignedTo: nigel< at >exim.org
        ReportedBy: isaikin-dmitry< at >yandex.ru
                CC: exim-dev< at >exim.org


We use option "dns_retry".
After rebuilding Exim from master branch, Exim makes core:

/etc/init.d/exim restart
Shutting down exim:                                        [  OK  ]
Starting exim: 2013-05-08 13:27:32 Exim configuration error in line 81 of
/usr/exim/conf/configure:
  main option "dns_retry" unknown

Fix:

diff --git a/src/src/readconf.c b/src/src/readconf.c
index d684b90..84577e5 100644
--- a/src/src/readconf.c
+++ b/src/src/readconf.c
< at >< at > -2

Security considerations: use_shell

Phil Pennock — 2013-05-06T01:35:28

I've pushed commit 5336c0d adding a new section to the Security
Considerations chapter of The Exim Specification.

There's probably more we can add here.

Context:
 https://www.redteam-pentesting.de/de/advisories/rt-sa-2013-001/-exim-with-dovecot-typical-misconfiguration-leads-to-remote-command-execution

People keep using use_shell and our warnings are perhaps not strong
enough as they stand.  Adding them into the security considerations
chapter might provide for better review, with security teams talking to
postmasters to find workable balances in the use of features called out
as dangerous.

-Phil

Re: Opportunistic DANE TLS mode for SMTP

Viktor Dukhovni — 2013-05-03T13:48:01


One more thing I forgot to respond to from March, a DANE implementation
cannot be limited to just the "IN TLSA x 0 0" use case.  In practice
almost all DANE TLSA RRs will be of the "IN TLSA 3 1 1" variety, but
one needs to also be prepared for "IN TLSA 2 1 0" and "IN TLSA 2 1 1".

Even with "0 0 0" and "2 0 0", the certificate in the TLSA RR may
be an intermediate certificate and not a root!  Though in practice
almost nobody will publish such RRs since full certificates are
too bulky to store in DNS.

Yes, this means that implementating a DANE verifier is more complex
than just adding another root via a CAfile and asking the SSL
library to do PKI as usual.

One needs to support digest matches, and these may match at any
depth in the certificate chain.  One even needs to support "2 1 0"
and "0 1 0", by checking whether the top of the server chain is
signed by a given public key (without the corresponding certificate
in hand).

Opportunistic DANE TLS mode for SMTP

Viktor Dukhovni — 2013-05-02T17:37:57

I see that DANE was actively discussed on this list in March, sorry
I was not here to participate.

I am writing a draft RFC for an "experimental" protocol that
implements "opportunistic DANE TLS".  This is designed to match
SMTP protocol reality whenever it is in conflict with strict
conformance with RFC 6698 or other potentially relevant IETF
documents.  I'd like to see major MTAs implement the opportunistic
DANE TLS protocol.  I'm also attempting to reach out to Sendmail
and Microsoft.

I am writing a separate draft on operational guidance for DANE TLSA
implementations that is not SMTP specific and covers some cornder
cases that need a bit more explanation than provided in RFC 6698.
With luck the second draft could be a basis for an update to 6698.

A couple of follow-ups to the March discussion:

    - It would be a mistake to use security status to trump MX
      preference.  Some sites have MX hosts that refuse all mail
      unless the secondary believe that the primary is down.

      Connecting to

Re: Embedding Python

Johannes Berg — 2013-05-02T11:51:11


I had this same problem once, and did this workaround:

#define PYTHON_DIR "..."

static const char pypath[] =
"import sys\n"
"sys.path.append(\"" PYTHON_DIR "\")\n";

...

Py_Initialize();

m = PyImport_AddModule("__main__");
globals = PyModule_GetDict(m);

res = PyRun_String(pypath, Py_file_input, globals, globals);
if (!res) {
PyErr_Print();
return -1;
} else
Py_DECREF(res);

johannes

Re: Embedding Python

Phil Pennock — 2013-05-02T20:36:59

Messing with environment seems bad, and affects the ability of sysadmin
to do so if needed.

Todd should just be able to use PyList_Insert() to modify sys.path,
after importing sys.  Note that this has to happen after the interpreter
is loaded, since Py_GetPath() is used to provide the default value.

The main things to figure out and document are:

 1. Does site initialisation get loaded?  If not, does another fixed
    module name get loaded instead, letting administrators symlink it to
    pull in any site customisations of sys.path done therein?  This is
    `site.py` inside the main python system modules directory, and it
    affects whether or not distutils-provided packages can be used, etc
    etc.

 2. Is threading initialised?  If not, threading calls will all succeed
    but the other threads will never actually be scheduled.  It's
    GIL-based, and OS-level threads are unaffected, since effectively
    it's just a single real thread, but given that Exim is traditionally
    single-threaded, eve

Re: Embedding Python

2013-05-02T11:01:41

http://docs.python.org/2/install/index.html
says:
==========
There are two environment variables that can modify sys.path. PYTHONHOME 
sets an alternate value for the prefix of the Python installation. For 
example, if PYTHONHOME is set to /www/python, the search path will be 
set to ['', '/www/python/lib/pythonX.Y/', 
'/www/python/lib/pythonX.Y/plat-linux2', ...].

The PYTHONPATH variable can be set to a list of paths that will be added 
to the beginning of sys.path. For example, if PYTHONPATH is set to 
/www/python:/opt/py, the search path will begin with ['/www/python', 
'/opt/py']. (Note that directories must exist in order to be added to 
sys.path; the site module removes paths that don’t exist.)
==========

So presumably if the exim process modifies its own environment in that
way before starting Python, we get what we need?