gmane.comp.python.lxml.devel

Re: Possible bug in lxml.objectify

Stefan Behnel — 2013-06-17T19:12:01

Amaury Forgeot d'Arc, 17.06.2013 18:20:

Yes, that's a bug. Thanks!

https://github.com/lxml/lxml/commit/1c6084a1754e83c45286fe2b112d2ef327cadc67

Stefan

_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Possible bug in lxml.objectify

Amaury Forgeot d'Arc — 2013-06-17T16:20:05

Hi,

By doing the cffi port, I noticed this suspect line in lxml.objectify.pyx
(line 1689):

    elif c_ns.prefix is not NULL or c_ns.prefix[0] != c'\0':

It seems to me that the connector should be "and" instead, otherwise a
prefix of "" would yield a typename starting with a colon character...
I haven't tried though. But it looks like a bad negation of a similar
statement above.

Re: New version for PyPy, using cffi

Stefan Behnel — 2013-06-17T07:15:59

Amaury Forgeot d'Arc, 16.06.2013 22:34:

Yes, those are very delicate. The proxy module (mis)uses several CPython
internals and the whole purpose of the macros is speed. IIRC, they gave a
performance boost of about 30% over the function based implementation and
most of the places where they are used are very performance critical, often
processing an entire tree in one tight loop.



Ok. Plus, at least some parts of the Cython version could start using pure
Python syntax, which would make the reuse easier.

http://docs.cython.org/src/tutorial/pure.html

apihelpers.pxi might get in the way, though, because it contains a couple
of helper functions that need to get included in order to inline them. So
it won't easily work to normally import it in PyPy and include it in Cython
code.



I see. This should rarely be a problem in practice, but it might bite some
users quite unexpectedly. But then again, any PyPy user will have to become
aware of this difference at some point.



I guess so.



Sure. I meant that the

Re: New version for PyPy, using cffi

Amaury Forgeot d'Arc — 2013-06-16T20:34:45

2013/6/15 Stefan Behnel <stefan_ml< at >behnel.de>


Indeed. I meant that I did not have to change the application design: same
module names, same functions, very similar code.
Except maybe for the proxy module, and the FOR_EACH_FROM macros.



The diff between the two branches:
https://github.com/amauryfa/lxml/compare/master...cffi
shows that I added many files, but modification to existing files are small.

This means that the cffi version can coexist with the cython one.



Well, this issue is already mentioned in the lxml docs:
http://lxml.de/objectify.html#defining-additional-data-classes
"""Be aware, though, that this does not immediately apply to elements to
which there already is a Python reference. Their Python class will only be
changed after all references are gone and the Python object is garbage
collected.
"""
With a deterministic garbage collector, this is often not an issue. But
with a delayed gc, many tests fail.
For example:

    from lxml import etree
    from lxml import objectify
    parser =

Re: New version for PyPy, using cffi

Amaury Forgeot d'Arc — 2013-06-16T19:37:10

2013/6/15 F Wolff <friedel< at >translate.org.za>



Oh, you need a newer version of cffi.
I did not realized that "new_handle" was not present in 2.0.2 :-(

Try with a (recent) nightly build of pypy, it has it for sure.

Re: New version for PyPy, using cffi

F Wolff — 2013-06-15T13:43:02

On Fri, Jun 14, 2013 at 10:20 PM, Amaury Forgeot d'Arc
<amauryfa< at >gmail.com> wrote:


Hi Amaury. Well done on the work.

Here is the result of my first try:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml/../lxml-cffi/etree.py", line 2859, in <module>
    from .xmlschema import XMLSchema, XMLSchemaParseError
  File "lxml/../lxml-cffi/xmlschema.py", line 34, in <module>
    namespaces={u'xs': u'http://www.w3.org/2001/XMLSchema'})
  File "lxml/../lxml-cffi/xpath.py", line 379, in __init__
    self.set_context(xpathCtxt)
  File "lxml/../lxml-cffi/xpath.py", line 155, in set_context
    self._context.set_context(xpathCtxt)
  File "lxml/../lxml-cffi/xpath.py", line 90, in set_context
    self._set_xpath_context(xpathCtxt)
  File "lxml/../lxml-cffi/extensions.py", line 111, in _set_xpath_context
    handle = xpath.ffi.new_handle(self)
AttributeError: 'FFI' object has no attribute 'new_handle'

This is with:
Mageia 2
pypy-2.0.2 built from source
lib64xml2-devel-2.7.8
lib64xslt-d

Re: New version for PyPy, using cffi

Stefan Behnel — 2013-06-15T09:24:32

Hi Amaury,

first of all, thank you for the effort that you put into this. I'm not
entirely happy about having a separate fork of the code base, but I'm sure
(some?) PyPy users will appreciate it.

Amaury Forgeot d'Arc, 14.06.2013 23:20:

Most of it, it seems. From what I gathered, there are at least some major
adaptations that seem to have required some thought. The transformation is
certainly nothing that could be automated.



Not sure what you mean. Which __init__.py where?



Could you explain what the problem is here? lxml.objectify is certainly
quite tightly coupled with lxml.etree, but it mainly uses standard features
of the API, mainly custom element classes.



Sure.



Is your goal to reintegrate this with mainline lxml? This will certainly
require more work. And I wouldn't be happy about the additional maintenance
overhead just to keep the additional cffi sources up-to-date.



Sounds like you'd better write a completely separate one. Most of what lxml
is doing at install time is configuring the

New version for PyPy, using cffi

Amaury Forgeot d'Arc — 2013-06-14T21:20:36

Hello,

I almost finished a port of lxml that uses cffi bindings instead of Cython.
The goal is to run it with PyPy, which support for the CPython API is known
to be very slow and somewhat incomplete.
cffi on the other hand is much more friendly to PyPy's JIT.

The code is there: https://github.com/amauryfa/lxml in the "cffi" branch.
Note that it works directly from the source tree, no need to compile.
You have to use a recent PyPy, ensure that the "lxml" directory can be seen
from your PYTHONPATH, and that you have libxml development packages.

The conversion -- from .pyx files to pure Python -- turned out to be quite
straightforward. And the result is interesting IMO:

- Impact on the existing code is minimal (in __init__.py), of course there
is a lot of duplication.

- Only 5 tests fail in the whole suite (if you except objectify, which is
hard to support correctly on a non-refcounting gc)

- Performance is on par with the version installed on my laptop, except
from some catastrophic benchmarks, probably

Re: etree.XML() nsmap gives partial namespace

Stefan Behnel — 2013-06-03T16:06:32

Sangeeth Saravanaraj, 03.06.2013 16:34:

Does this help?

http://lxml.de/FAQ.html#how-can-i-find-out-which-namespace-prefixes-are-used-in-a-document

Stefan

_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

etree.XML() nsmap gives partial namespace

Sangeeth Saravanaraj — 2013-06-03T14:34:36

I am using nsmap on etree.XML() to get the list of all the namespaces used
in a given xml document. The following code i.e. <code-1> works fine which
gives the expected output i.e. <output-1>. The nsmap contains all the
namespace that are used in the document.

One point to note here is - all the xml namespaces used in this document
has been highlighted in the root element itself.

<code-1>
from lxml import etree

xml_str_1 = '''<?xml version='1.0' encoding='UTF-8'?>
<ns0:root xmlns:ns0="http://thisisroot.com/rootelem/" xmlns:ns1="
http://thisischild.com/childelem1/" xmlns:ns2="
http://thisischild.com/childelem2/" xmlns:ns3="
http://thisischild.com/childelem3/" xmlns:ns4="
http://thisischild.com/childelem4/">
  <ns1:child1>child1</ns1:child1>
  <ns2:child2>child2</ns2:child2>
  <ns3:child3>child3</ns3:child3>
  <ns4:child4>child4</ns4:child4>
</ns0:root>'''

xml_1 = etree.XML(xml_str_1)
ns_1 = xml_1.nsmap

print "xml document = ", xml_str_1
print "namespace    = ", ns_1
</code-1>

<output-1>
xml document =

attributes() in DTD - possible bug()

Nikolay Amiantov — 2013-06-03T11:34:12

Hello,
I'm trying to iterate over all attributes in DTD. Here is a log from REPL:
http://pastebin.com/qfSqYjCB
attributes() is empty when using DTD from internalDTD, but they are
definitely used (default values for attributes are properly set). Also,
when using etree.DTD attributes are properly listed.
Is this a bug or there is something that I have missed?

P.S.: externalDTD is empty, this seems right to me.

Nikolay Amiantov.
_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

attributes() in DTD - possible bug?

Nikolay Amiantov — 2013-06-03T11:27:57

Hello,
I'm trying to iterate over all attributes in DTD. Here is a log from REPL:
http://pastebin.com/qfSqYjCB
attributes() is empty when using DTD from internalDTD, but they are
definitely used (default values for attributes are properly set). Also,
when using etree.DTD attributes are properly listed.
Is this a bug or there is something that I have missed?

P.S.: externalDTD is empty, and this seems right to me.
_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Re: Changing encoding while parsing an HTML string

Stefan Behnel — 2013-06-02T07:04:28

Michael O'Leary, 24.05.2013 21:22:

It's an XML parser, so, yes, it does.



The latter is the normal way of doing it in HTML.



Yes, it does.



It should do it automatically if it can, defaulting to Latin-1 if there is
no content-type meta tag that specifies the encoding. If you know the
encoding in advance for some reason, you can configure an HTMLParser with
it and pass it into the parse() function (for example). If you have
different encodings in your files, use differently configured parsers.

You cannot change the encoding once the parser has started parsing. If you
feel like changing it along the way, I suggest you just start over.

You might also want to consider using iterparse() instead of passing a
target object into the parser. It has less overhead.

Stefan

_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Re: lxml object architecture quandary

John W. Shipman — 2013-05-21T19:42:20

Thanks for the reply, Holger!  Actually it did occur to me over
the weekend that inheritance was the way to go.

I rewrote my interface module using inheritance from etree.Element
only to find that this does not work (and if I had read the lxml.de
documentation more careful, I would have known that).

Fortunately, it occurred to me that inheriting from the standard
Python module xml.etree.ElementTree might work, and indeed it
did.  The code is all working now.

Just for your amusement, I've attached the working interface
class.

Best regards,
John Shipman (john< at >nmt.edu), Applications Specialist
New Mexico Tech Computer Center, Speare 146, Socorro, NM 87801
(575) 835-5735, http://www.nmt.edu/~john
   ``Let's go outside and commiserate with nature.''  --Dave Farber

On Tue, 21 May 2013, Holger Joukl wrote:

'''newadds.py: Complete rewrite of readadds.py

  class AddsFile:  Represents one semester's CMS course requests.
    AddsFile(acadYear, semCode, host=None):
      [ (acadYear is an academic year as a f

Re: wrong ELF class

Wolfgang Buesser — 2013-05-27T16:41:55






Hi Simon, Hi Holger,

with your help I found out that the rpm was actually i586. I don't know how that happened.
As far as I remember it was part of official SuSE release. 
Anyway I replaced with an rpm from rpm-find and now my application is running.

Thanks

Wolfgang




_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Re: wrong ELF class

Holger Joukl — 2013-05-27T09:29:29

Hi,


This might also be helpful for checking to which package the file actually
belongs
or which files your downloaded package contains:

http://llamalabs.com/2007/04/13/which-package-does-a-file-belong-to-rpm-style/

Holger

Landesbank Baden-Wuerttemberg
Anstalt des oeffentlichen Rechts
Hauptsitze: Stuttgart, Karlsruhe, Mannheim, Mainz
HRA 12704
Amtsgericht Stuttgart
_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Re: wrong ELF class

Simon Sapin — 2013-05-27T09:00:17

Le 27/05/2013 01:55, Wolfgang Buesser a écrit :

It depends, where did you get this RPM file?



Try running this command:
file /usr/lib/python2.7/site-packages/lxml/etree.so

Cheers,

wrong ELF class

Wolfgang Buesser — 2013-05-26T17:55:20

Hello,

I am running SUSE 12.3 on an AMD-64 System.

I have installed python-lxml-2.3.4-6.1.1.x86_64.rpm

When lxml gets called from gnuradio-companion I get the following
error-message:

ImportError: /usr/lib/python2.7/site-packages/lxml/etree.so: wrong ELF
class: ELFCLASS32

Could it be possible, that the architecture in the RPM does not match
64-Bit (despite it's name)?

How can I check?

How can I get a working python-lxml for my system?

Thanks

Wolfgang



_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Re: Error seen when running unit tests

Stefan Behnel — 2013-05-25T08:28:07

Am 22.05.2013 19:54, schrieb Lindykid:

My educated guess is that it's an issue with libxml2. The 2.7.x versions
below 2.7.8 are known to have varying minor issues. If you can't
(easily) upgrade your system library, consider a static lxml build,
preferably using the latest libxml2 2.9.x release.

Alternatively, test your own code with what you have and if that works,
it's unlikely to magically start failing without modifications. Bugs in
libxml2 often just impact specific features (such as push parsing
above), not all of lxml.

Stefan

_________________________________________________________________
Mailing list for the lxml Python XML toolkit - http://lxml.de/
lxml< at >lxml.de
https://mailman-mail5.webfaction.com/listinfo/lxml

Changing encoding while parsing an HTML string

Michael O'Leary — 2013-05-24T19:22:56

If I understand how the XML parser works, if it parses an XML string that
begins with a header line like

<?xml version="1.0" encoding="utf-8"?>

it will parse the rest of the XML string using the encoding of the encoding
it has found. So first of all, does the XML parser do this?

If so, I'd like to know if something like this can be done, either
automatically or manually, with the HTML parser.

I have been parsing HTML strings that have been obtained from a random
sampling of web sites, and I am wondering if there is a way to do something
similar with these HTML strings. Typically, these HTML strings do not start
with a header like, but most of them do have a meta tag of the form

<meta charset="utf-8">

or

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" >

I wouldn't expect that the HTML parser reads these tags and automatically
sets the encoding to be used while parsing the rest of the HTML string (but
I might as well ask: does it?). I could include code in my target's start
tag handl

Re: lxml.objectify drops leading whitespace

Holger Joukl — 2013-05-24T16:33:31

Hi,


It is intentional. For data binding the text content of elements with
children is (usually) irrelevant.

IIRC this only drops whitespace for elements with children, not for
elements
with text-only content.

You should be able to change objectify's behaviour by changing it's
default parser using makeparser() and set_default_parser:

def set_default_parser(new_parser = None):
    u"""set_default_parser(new_parser = None)

    Replace the default parser used by objectify's Element() and
    fromstring() functions.

    The new parser must be an etree.XMLParser.

    Call without arguments to reset to the original parser.
    """

def makeparser(**kw):
    u"""makeparser(remove_blank_text=True, **kw)

    Create a new XML parser for objectify trees.

    You can pass all keyword arguments that are supported by
    ``etree.XMLParser()``.  Note that this parser defaults to removing
    blank text.  You can disable this by passing the
    ``remove_blank_text`` boolean keyword option yourself.
    """