gmane.comp.python.lxml.devel

Re: Undefined preference when buildinglxmlxsltLibxsltVersion

Stefan Behnel — 2008-08-21T17:11:06

Hi, Nguyễn Khánh Duy wrote: You need to have the "xslt-config" script in your PATH. It comes with libxslt, but in your case, lxml's setup.py script can't find it. Stefan _______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Undefined preference when building lxmlxsltLibxsltVersion

Nguyễn Khánh Duy — 2008-08-21T15:43:34

I am trying to build lxml for python 2.3 on windows, using mingw32. I've successfully downloaded and installed libxslt and libxml2, put the include header files and the lib files where they need to be, but when I build lxml the following error appears: Am I missing any .lib files? libxslt, libxml2, libexslt, iconv and zlib are all in the python 2.3 libs folder. D:\LXML\lxml-2.1.1>python setup.py build -cmingw32 Building lxml version 2.1.1. NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' needs to be available. ERROR: 'xslt-config' is not recognized as an internal or external command, operable program or batch file. ** make sure the development packages of libxml2 and libxslt are installed ** Using build configuration of libxslt running build running build_py running build_ext building 'lxml.etree' extension writing build\temp.win32-2.3\Release\src\lxml\etree.def C:\MinGW\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.3\Release\src\lxm l\lxml.etree.o build\temp.win32-2.3\Release\src\lxml\etree.def -LD:\Python23\lib s -LD:\Python23\PCBuild -llibxslt -llibexslt -llibxml2 -liconv -lzlib -lWS2_32 - lpython23 -o build\lib.win32-2.3\lxml\etree.pyd build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x67e20): undefined reference to `xsltProcessOneNode' build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x83946): undefined reference to `xsltLibxsltVersion' build\temp.win32-2.3\Release\src\lxml\lxml.etree.o:lxml.etree.c:(.text+0x839e2): undefined reference to `xsltDocDefaultLoader' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1

Re: objectify.ObjectPath("wrongroot.a.b")(root, None) should not raise an exception

2008-08-21T06:24:06

_______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Re: objectify.ObjectPath("wrongroot.a.b")(root, None) should not raise an exception

Stefan Behnel — 2008-08-20T18:55:10

Hi, Holger Joukl wrote: Please do. Stefan

Fwd: [xml] Security fix for libxml2

Stefan Behnel — 2008-08-20T18:50:26

FYI -------- Original-Message -------- Subject: [xml] Security fix for libxml2 Date: Wed, 20 Aug 2008 19:00:51 +0200 From: Daniel Veillard redhat.com> To: xml< at >gnome.org Bad news, when checking against recursive entities expansion problem back when it was made official (c.f. the billion laught attack circa 2004) I had checked for the normal recursion, but when happening in an attribute value the resource consumption is way faster and the recursion detection in place is not sufficient to catch the problem. Basically when this happen within an attribute just checking for a recursion depth is not sufficient, and the only good method I could find was to count the number of entities replacement taking place while parsing a given document, and drop parsing after half a million substitution. I think it's a fair default process and what the patches below implements for various libxml2 versions, but i can understand that in some case that may be problematic. So i intend in the next release (2.7.0 hopefully available soon) to add a parser flag removing the hardcoded limits (there is also a maximum document depth in place). Distributions have been made aware of the problem for a couple of weeks and updates should be available soon from normal update channels I'm updating SVN with the fix too, Daniel Index: include/libxml/parser.h =================================================================== --- include/libxml/parser.h(revision 3771) +++ include/libxml/parser.h(working copy) < at >< at > -297,6 +297,7 < at >< at > struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** Index: include/libxml/entities.h =================================================================== --- include/libxml/entities.h(revision 3771) +++ include/libxml/entities.h(working copy) < at >< at > -57,6 +57,7 < at >< at > struct _xmlEntity { const xmlChar *URI;/* the full URI as computed */ int owner;/* does the entity own the childrens */ int checked;/* was the entity content checked */ + unsigned long nbentities;/* the number of entities references */ }; /* Index: entities.c =================================================================== --- entities.c(revision 3771) +++ entities.c(working copy) < at >< at > -31,35 +31,35 < at >< at > static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0, 1 + NULL, NULL, NULL, NULL, 0, 1, 0 }; /** Index: parserInternals.c =================================================================== --- parserInternals.c(revision 3771) +++ parserInternals.c(working copy) < at >< at > -1670,6 +1670,7 < at >< at > xmlInitParserCtxt(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); return(0); } Index: parser.c =================================================================== --- parser.c(revision 3771) +++ parser.c(working copy) < at >< at > -2379,7 +2379,7 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } < at >< at > -2417,6 +2417,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { < at >< at > -2462,6 +2467,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { if (ent->content == NULL) { if (xmlLoadEntityContent(ctxt, ent) < 0) { < at >< at > -2501,6 +2511,7 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: if (rep != NULL) xmlFree(rep); if (buffer != NULL) < at >< at > -3542,6 +3553,9 < at >< at > xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); +ctxt->nbentities++; +if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { < at >< at > -4844,6 +4858,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { < at >< at > -5068,6 +5083,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else < at >< at > -6477,6 +6493,11 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; +ctxt->nbentities++; +if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; +} was_checked = ent->checked; if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { < at >< at > -6537,6 +6558,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled < at >< at > -6559,6 +6581,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; < at >< at > -6571,6 +6594,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; < at >< at > -6629,6 +6653,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) } ent->checked = 1; } + ctxt->nbentities += ent->nbentities; if (ent->children == NULL) { /* < at >< at > -11800,7 +11825,7 < at >< at > xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -12010,7 +12035,8 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -12154,6 +12180,7 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; < at >< at > -12254,7 +12281,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -12379,6 +12406,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; < at >< at > -13695,6 +13723,7 < at >< at > xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) { --- include/libxml/parser.h(revision 3771) +++ include/libxml/parser.h(working copy) < at >< at > -297,6 +297,7 < at >< at > struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** --- include/libxml/entities.h.orig2005-01-04 15:49:49.000000000 +0100 +++ include/libxml/entities.h2008-08-11 17:56:53.000000000 +0200 < at >< at > -56,6 +56,7 < at >< at > struct _xmlEntity { struct _xmlEntity *nexte;/* unused */ const xmlChar *URI;/* the full URI as computed */ int owner;/* does the entity own the childrens */ + unsigned long nbentities;/* the number of entities references */ }; /* --- entities.c.orig2006-03-09 17:39:46.000000000 +0100 +++ entities.c2008-08-11 18:01:06.000000000 +0200 < at >< at > -31,35 +31,35 < at >< at > static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; /** --- parser.c.orig2006-04-23 11:39:15.000000000 +0200 +++ parser.c2008-08-11 18:36:56.000000000 +0200 < at >< at > -2174,7 +2176,7 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } < at >< at > -2212,6 +2214,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { < at >< at > -2258,6 +2265,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { xmlChar *rep; < at >< at > -2294,6 +2306,9 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: + if (buffer != NULL) + xmlFree(buffer); return(NULL); } < at >< at > -3100,6 +3115,9 < at >< at > xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); +ctxt->nbentities++; +if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { < at >< at > -4342,6 +4360,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { < at >< at > -4551,6 +4570,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else < at >< at > -5927,6 +5947,11 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; +ctxt->nbentities++; +if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; +} if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { xmlNodePtr list = NULL; < at >< at > -5985,6 +6010,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled < at >< at > -6007,6 +6033,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; < at >< at > -6019,6 +6046,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; < at >< at > -6075,6 +6103,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) } } } + ctxt->nbentities += ent->nbentities; if ((ctxt->sax != NULL) && (ctxt->sax->reference != NULL) && (ctxt->replaceEntities == 0) && (!ctxt->disableSAX)) { /* < at >< at > -11035,7 +11064,7 < at >< at > xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11220,7 +11249,8 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11363,6 +11393,7 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; < at >< at > -11463,7 +11494,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11587,6 +11618,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; < at >< at > -12883,6 +12915,7 < at >< at > xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) { --- include/libxml/parser.h(revision 3771) +++ include/libxml/parser.h(working copy) < at >< at > -297,6 +297,7 < at >< at > struct _xmlParserCtxt { */ xmlError lastError; xmlParserMode parseMode; /* the parser mode */ + unsigned long nbentities; /* number of entities references */ }; /** --- include/libxml/entities.h.orig2005-01-04 15:49:49.000000000 +0100 +++ include/libxml/entities.h2008-08-11 17:56:53.000000000 +0200 < at >< at > -56,6 +56,7 < at >< at > struct _xmlEntity { struct _xmlEntity *nexte;/* unused */ const xmlChar *URI;/* the full URI as computed */ int owner;/* does the entity own the childrens */ + unsigned long nbentities;/* the number of entities references */ }; /* --- entities.c.orig2006-03-09 17:39:46.000000000 +0100 +++ entities.c2008-08-11 18:01:06.000000000 +0200 < at >< at > -31,35 +31,35 < at >< at > static xmlEntity xmlEntityLt = { NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "<", BAD_CAST "<", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityGt = { NULL, XML_ENTITY_DECL, BAD_CAST "gt", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST ">", BAD_CAST ">", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityAmp = { NULL, XML_ENTITY_DECL, BAD_CAST "amp", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "&", BAD_CAST "&", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityQuot = { NULL, XML_ENTITY_DECL, BAD_CAST "quot", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "\"", BAD_CAST "\"", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; static xmlEntity xmlEntityApos = { NULL, XML_ENTITY_DECL, BAD_CAST "apos", NULL, NULL, NULL, NULL, NULL, NULL, BAD_CAST "'", BAD_CAST "'", 1, XML_INTERNAL_PREDEFINED_ENTITY, - NULL, NULL, NULL, NULL, 0 + NULL, NULL, NULL, NULL, 0, 0 }; /** --- parser.c.orig2006-04-23 11:39:15.000000000 +0200 +++ parser.c2008-08-11 18:36:56.000000000 +0200 < at >< at > -2174,7 +2176,7 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt return(NULL); last = str + len; - if (ctxt->depth > 40) { + if ((ctxt->depth > 40) || (ctxt->nbentities >= 500000)) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return(NULL); } < at >< at > -2212,6 +2214,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt "String decoding Entity Reference: %.30s\n", str); ent = xmlParseStringEntityRef(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (ent->content != NULL) { < at >< at > -2258,6 +2265,11 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt xmlGenericError(xmlGenericErrorContext, "String decoding PE Reference: %.30s\n", str); ent = xmlParseStringPEReference(ctxt, &str); + if (ctxt->lastError.code == XML_ERR_ENTITY_LOOP) + goto int_error; + ctxt->nbentities++; + if (ent != NULL) + ctxt->nbentities += ent->nbentities; if (ent != NULL) { xmlChar *rep; < at >< at > -2294,6 +2306,9 < at >< at > xmlStringLenDecodeEntities(xmlParserCtxt mem_error: xmlErrMemory(ctxt, NULL); +int_error: + if (buffer != NULL) + xmlFree(buffer); return(NULL); } < at >< at > -3100,6 +3115,9 < at >< at > xmlParseAttValueComplex(xmlParserCtxtPtr } } else { ent = xmlParseEntityRef(ctxt); +ctxt->nbentities++; +if (ent != NULL) + ctxt->nbentities += ent->nbentities; if ((ent != NULL) && (ent->etype == XML_INTERNAL_PREDEFINED_ENTITY)) { if (len > buf_size - 10) { < at >< at > -4342,6 +4360,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt int isParameter = 0; xmlChar *orig = NULL; int skipped; + unsigned long oldnbent = ctxt->nbentities; /* GROW; done in the caller */ if (CMP8(CUR_PTR, '<', '!', 'E', 'N', 'T', 'I', 'T', 'Y')) { < at >< at > -4551,6 +4570,7 < at >< at > xmlParseEntityDecl(xmlParserCtxtPtr ctxt } } if (cur != NULL) { + cur->nbentities = ctxt->nbentities - oldnbent; if (cur->orig != NULL) xmlFree(orig); else < at >< at > -5927,6 +5947,11 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) if (ent == NULL) return; if (!ctxt->wellFormed) return; +ctxt->nbentities++; +if (ctxt->nbentities >= 500000) { + xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); + return; +} if ((ent->name != NULL) && (ent->etype != XML_INTERNAL_PREDEFINED_ENTITY)) { xmlNodePtr list = NULL; < at >< at > -5985,6 +6010,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlFreeNodeList(list); } } else { + unsigned long oldnbent = ctxt->nbentities; /* * 4.3.2: An internal general parsed entity is well-formed * if its replacement text matches the production labeled < at >< at > -6007,6 +6033,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) ret = xmlParseBalancedChunkMemoryInternal(ctxt, value, user_data, &list); ctxt->depth--; + } else if (ent->etype == XML_EXTERNAL_GENERAL_PARSED_ENTITY) { ctxt->depth++; < at >< at > -6019,6 +6046,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) xmlErrMsgStr(ctxt, XML_ERR_INTERNAL_ERROR, "invalid entity type found\n", NULL); } + ent->nbentities = ctxt->nbentities - oldnbent; if (ret == XML_ERR_ENTITY_LOOP) { xmlFatalErr(ctxt, XML_ERR_ENTITY_LOOP, NULL); return; < at >< at > -6075,6 +6103,7 < at >< at > xmlParseReference(xmlParserCtxtPtr ctxt) } } } + ctxt->nbentities += ent->nbentities; if ((ctxt->sax != NULL) && (ctxt->sax->reference != NULL) && (ctxt->replaceEntities == 0) && (!ctxt->disableSAX)) { /* < at >< at > -11035,7 +11064,7 < at >< at > xmlParseCtxtExternalEntity(xmlParserCtxt if (ctx == NULL) return(-1); - if (ctx->depth > 40) { + if ((ctx->depth > 40) || (ctx->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11220,7 +11249,8 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr xmlChar start[4]; xmlCharEncoding enc; - if (depth > 40) { + if ((depth > 40) || + ((oldctxt != NULL) && (oldctxt->nbentities >= 500000))) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11363,6 +11393,7 < at >< at > xmlParseExternalEntityPrivate(xmlDocPtr oldctxt->node_seq.maximum = ctxt->node_seq.maximum; oldctxt->node_seq.length = ctxt->node_seq.length; oldctxt->node_seq.buffer = ctxt->node_seq.buffer; + oldctxt->nbentities += ctxt->nbentities; ctxt->node_seq.maximum = 0; ctxt->node_seq.length = 0; ctxt->node_seq.buffer = NULL; < at >< at > -11463,7 +11494,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP int size; xmlParserErrors ret = XML_ERR_OK; - if (oldctxt->depth > 40) { + if ((oldctxt->depth > 40) || (oldctxt->nbentities >= 500000)) { return(XML_ERR_ENTITY_LOOP); } < at >< at > -11587,6 +11618,7 < at >< at > xmlParseBalancedChunkMemoryInternal(xmlP ctxt->myDoc->last = last; } + oldctxt->nbentities += ctxt->nbentities; ctxt->sax = oldsax; ctxt->dict = NULL; ctxt->attsDefault = NULL; < at >< at > -12883,6 +12915,7 < at >< at > xmlCtxtReset(xmlParserCtxtPtr ctxt) ctxt->depth = 0; ctxt->charset = XML_CHAR_ENCODING_UTF8; ctxt->catalogs = NULL; + ctxt->nbentities = 0; xmlInitNodeInfoSeq(&ctxt->node_seq); if (ctxt->attsDefault != NULL) { _______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Re: namespace strangeness in lxml 1.1

Stefan Behnel — 2008-08-20T07:07:55

Hi, coming back to this after a while... Eric Jahn wrote: I think you misunderstood my example (and apparently didn't try it on your side). Isn't this what you wanted: >>> import lxml.etree as et >>> root = et.XML('') >>> root[0].set("type", et.QName("{http://my/ns}tname")) >>> et.tostring(root) '' This has been working for quite a while now. Stefan

objectify.ObjectPath("wrongroot.a.b")(root, None) should not raise an exception

Holger Joukl — 2008-08-20T15:09:09

_______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-19T19:45:36

Hi, Sidnei da Silva wrote: here's a patch that tries making filenames absolute when users pass a file-like object. This only works for file objects (and maybe compatible objects), but "file://..." URLs should be provided as absolute URLs anyway. Is there anything that's missing? You mentioned problems with urlopen(), but didn't elaborate on them any further. Stefan _______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Re: Failure with custom resolvers and relative xsl:import

John Lovell — 2008-08-19T16:16:32

I wound up using tlslite the only issue was Windows for which there wasn't an installer for Python 2.5 and it seemed I needed a version of DevStudio (that I don't have) to build one. I tried manually copying the files and still no joy. Anyway, back to lxml stuff. John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell< at >nwesd.org www.nwesd.org Together We Can ... -----Original Message----- From: Stefan Behnel [mailto:stefan_ml< at >behnel.de] Sent: Tuesday, August 19, 2008 9:02 AM To: Mike Meyer Cc: John Lovell; ML-Lxml-dev Subject: Re: [lxml-dev] Failure with custom resolvers and relative xsl:import Mike Meyer wrote: Ah, sure, that's a portability issue then. ... for which there is tlslite, at least I know that one. From a quick look at their web site, it's supposed to work with httplib also. I never used it either, but a couple of blog entries seem to be happy with it. Stefan

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-19T16:02:06

Ah, sure, that's a portability issue then. ... for which there is tlslite, at least I know that one. From a quick look at their web site, it's supposed to work with httplib also. I never used it either, but a couple of blog entries seem to be happy with it. Stefan

Re: Failure with custom resolvers and relative xsl:import

Eric Tiffany — 2008-08-19T15:49:03

I have used M2Crypto for clientTLS connections from MacOS, Ubuntu, and Vista, all using the same codebase (including LXML) running under Plone. Works brillantly. ET On 8/19/08 11:07 AM, "John Lovell" esd189.org> wrote:

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-19T15:39:19

I never needed to try (I'd start with urllib2 anyway...), but this sounds like a question for comp.lang.python rather than this list. Stefan

Re: Failure with custom resolvers and relative xsl:import

John Lovell — 2008-08-19T15:07:06

Stefan: So what is your "preferred Python HTTPS client library?" I recently tried very hard and ultimately failed at handling https cross platform with Python. Thanks for all you do, John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell< at >nwesd.org www.nwesd.org Together We Can ... -----Original Message----- From: Stefan Behnel [mailto:stefan_ml< at >behnel.de] Sent: Tuesday, August 19, 2008 3:38 AM To: John Lovell Cc: ML-Lxml-dev Subject: Re: [lxml-dev] Failure with custom resolvers and relative xsl:import John Lovell wrote: Hmmm, isn't it enough to use a custom resolver to intercept only "https://..." requests (return None for everything else) and to redirect them to your preferred Python HTTPS client library? I'm not sure implementing this directly in lxml makes sense, as it wouldn't allow using certificates and these things. But it shouldn't be too hard to do on your own. Stefan

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-19T10:38:25

Hmmm, isn't it enough to use a custom resolver to intercept only "https://..." requests (return None for everything else) and to redirect them to your preferred Python HTTPS client library? I'm not sure implementing this directly in lxml makes sense, as it wouldn't allow using certificates and these things. But it shouldn't be too hard to do on your own. Stefan

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-19T10:26:38

If you return a custom object, you can give it a "filename" attribute or a "geturl()" method (as provided by urlopen() 'files'). lxml will recognise at. A StringIO object can't pass on its filename or source URL, so you have to provide a meaningful base_url from your resolver in this case. Stefan

Re: Failure with custom resolvers and relative xsl:import

John Lovell — 2008-08-18T20:50:38

Stefan: If you find a currently maintained pure Python way to add SSL, please let us know. John W. Lovell Web Applications Engineer Northwest Educational Service District 1601 R Avenue Anacortes, WA 98221 (360) 299-4086 jlovell< at >nwesd.org www.nwesd.org Together We Can ... -----Original Message----- From: lxml-dev-bounces< at >codespeak.net [mailto:lxml-dev-bounces< at >codespeak.net] On Behalf Of Stefan Behnel Sent: Monday, August 18, 2008 1:17 PM To: Sidnei da Silva Cc: ML-Lxml-dev Subject: Re: [lxml-dev] Failure with custom resolvers and relative xsl:import Hi, Sidnei da Silva wrote: wrote: the file. though. I'm working on fixing this. However, do you really need something from urlopen that lxml can't handle by itself? Any "HTTP GET" or "FTP get" request should work when passed as an encoded URL. HTTP POSTs and request options won't work, but they are pretty rarely used when requesting pages. HTTPS also won't work, which I would consider more important. Stefan _______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev

Re: Failure with custom resolvers and relative xsl:import

Sidnei da Silva — 2008-08-18T20:45:29

Yes, HTTPS and I'm also using code that handles caching. So, I overly simplified here. I actually use something that resembles urlopen but that might return an open file handle (or something that resembles a file, like a StringIO) if the requested url is cached locally.

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-18T20:16:53

Hi, Sidnei da Silva wrote: I'm working on fixing this. However, do you really need something from urlopen that lxml can't handle by itself? Any "HTTP GET" or "FTP get" request should work when passed as an encoded URL. HTTP POSTs and request options won't work, but they are pretty rarely used when requesting pages. HTTPS also won't work, which I would consider more important. Stefan

Re: Failure with custom resolvers and relative xsl:import

Sidnei da Silva — 2008-08-18T19:21:30

Okay, I think that solves one of the problems, when the filename is local. I suspect it won't solve the other problem I had, which was when the file comes from urlopen(). Since the files are relatively small though I can keep using resolve_string() for now.

Re: Failure with custom resolvers and relative xsl:import

Stefan Behnel — 2008-08-18T07:04:58

Hi, Sidnei da Silva wrote: The problem here is that lxml currently only takes the .name attribute of a file object to determine the file name. The right thing to do would be to use os.path.abspath(f.name) instead to retrieve the absolute path name of the file. Using resolve_filename(filename) instead of resolve_file(open(filename)) provides a more efficient work around, though. Stefan

Re: instance from schema

2008-08-15T10:56:51

_______________________________________________ lxml-dev mailing list lxml-dev< at >codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev