genshi/genshi-test: genshi/input.py annotate

annotate genshi/input.py @ 930:0ec0a695ec96

Merge r1138 from py3k: add python 3 support to _speedups C extension

author	hodgestar
date	Fri, 18 Mar 2011 09:04:14 +0000
parents	fbe34d12acde
children	9fafb35032a1

rev	line source
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	1 # -- coding: utf-8 --
821114ec4f69 Initial import. cmlenz parents: diff changeset	2 #
854 0d9e87c6cf6e More work on reducing the size of the diff produced by 2to3. cmlenz parents: 853 diff changeset	3 # Copyright (C) 2006-2009 Edgewall Software
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	4 # All rights reserved.
821114ec4f69 Initial import. cmlenz parents: diff changeset	5 #
821114ec4f69 Initial import. cmlenz parents: diff changeset	6 # This software is licensed as described in the file COPYING, which
821114ec4f69 Initial import. cmlenz parents: diff changeset	7 # you should have received as part of this distribution. The terms
230 24757b771651 Renamed Markup to Genshi in repository. cmlenz parents: 213 diff changeset	8 # are also available at http://genshi.edgewall.org/wiki/License.
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	9 #
821114ec4f69 Initial import. cmlenz parents: diff changeset	10 # This software consists of voluntary contributions made by many
821114ec4f69 Initial import. cmlenz parents: diff changeset	11 # individuals. For the exact contribution history, see the revision
230 24757b771651 Renamed Markup to Genshi in repository. cmlenz parents: 213 diff changeset	12 # history and logs, available at http://genshi.edgewall.org/log/.
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	13
425 5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	14 """Support for constructing markup streams from files, strings, or other
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	15 sources.
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	16 """
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	17
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	18 from itertools import chain
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	19 import htmlentitydefs as entities
fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	20 import HTMLParser as html
fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	21 from StringIO import StringIO
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	22 from xml.parsers import expat
821114ec4f69 Initial import. cmlenz parents: diff changeset	23
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	24 from genshi.core import Attrs, QName, Stream, stripentities
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	25 from genshi.core import START, END, XML_DECL, DOCTYPE, TEXT, START_NS, \
fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	26 END_NS, START_CDATA, END_CDATA, PI, COMMENT
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	27
290 a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	28 __all__ = ['ET', 'ParseError', 'XMLParser', 'XML', 'HTMLParser', 'HTML']
425 5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	29 __docformat__ = 'restructuredtext en'
290 a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	30
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	31
290 a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	32 def ET(element):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	33 """Convert a given ElementTree element to a markup stream.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	34
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	35 :param element: an ElementTree element
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	36 :return: a markup stream
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	37 """
290 a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	38 tag_name = QName(element.tag.lstrip('{'))
458 160f787cc818 The `ET()` function now correctly handles attributes with a namespace. cmlenz parents: 434 diff changeset	39 attrs = Attrs([(QName(attr.lstrip('{')), value)
160f787cc818 The `ET()` function now correctly handles attributes with a namespace. cmlenz parents: 434 diff changeset	40 for attr, value in element.items()])
290 a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	41
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	42 yield START, (tag_name, attrs), (None, -1, -1)
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	43 if element.text:
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	44 yield TEXT, element.text, (None, -1, -1)
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	45 for child in element.getchildren():
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	46 for item in ET(child):
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	47 yield item
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	48 yield END, tag_name, (None, -1, -1)
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	49 if element.tail:
a6738047c85e Move the ElementTree ''element-to-stream'' adaptation function `ET()` into the `genshi.input` module. cmlenz parents: 230 diff changeset	50 yield TEXT, element.tail, (None, -1, -1)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	51
821114ec4f69 Initial import. cmlenz parents: diff changeset	52
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	53 class ParseError(Exception):
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	54 """Exception raised when fatal syntax errors are found in the input being
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	55 parsed.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	56 """
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	57
422 95089b6e37ca More work to include absolute file paths in exceptions. cmlenz parents: 419 diff changeset	58 def __init__(self, message, filename=None, lineno=-1, offset=-1):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	59 """Exception initializer.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	60
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	61 :param message: the error message from the parser
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	62 :param filename: the path to the file that was parsed
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	63 :param lineno: the number of the line on which the error was encountered
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	64 :param offset: the column number where the error was encountered
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	65 """
422 95089b6e37ca More work to include absolute file paths in exceptions. cmlenz parents: 419 diff changeset	66 self.msg = message
95089b6e37ca More work to include absolute file paths in exceptions. cmlenz parents: 419 diff changeset	67 if filename:
434 e065d7906b68 * Better method to propogate the full path to the template file on parse errors. Supersedes r513. cmlenz parents: 433 diff changeset	68 message += ', in ' + filename
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	69 Exception.__init__(self, message)
422 95089b6e37ca More work to include absolute file paths in exceptions. cmlenz parents: 419 diff changeset	70 self.filename = filename or '<string>'
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	71 self.lineno = lineno
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	72 self.offset = offset
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	73
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	74
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	75 class XMLParser(object):
821114ec4f69 Initial import. cmlenz parents: diff changeset	76 """Generator-based XML parser based on roughly equivalent code in
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	77 Kid/ElementTree.
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	78
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	79 The parsing is initiated by iterating over the parser object:
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	80
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	81 >>> parser = XMLParser(StringIO('<root id="2"><child>Foo</child></root>'))
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	82 >>> for kind, data, pos in parser:
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	83 ... print('%s %s' % (kind, data))
857 24733a5854d9 Avoid unicode literals in `repr`s of `QName` and `Namespace` when not necessary. cmlenz parents: 856 diff changeset	84 START (QName('root'), Attrs([(QName('id'), u'2')]))
24733a5854d9 Avoid unicode literals in `repr`s of `QName` and `Namespace` when not necessary. cmlenz parents: 856 diff changeset	85 START (QName('child'), Attrs())
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	86 TEXT Foo
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	87 END child
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	88 END root
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	89 """
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	90
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	91 _entitydefs = ['<!ENTITY %s "&#%d;">' % (name, value) for name, value in
856 1e2be9fb3348 Add a couple of fallback imports for Python 3.0. cmlenz parents: 854 diff changeset	92 entities.name2codepoint.items()]
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	93 _external_dtd = '\n'.join(_entitydefs)
38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	94
316 4ab9edf5e83b Configurable encoding of template files, closing #65. cmlenz parents: 312 diff changeset	95 def __init__(self, source, filename=None, encoding=None):
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	96 """Initialize the parser for the given XML input.
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	97
425 5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	98 :param source: the XML text as a file-like object
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	99 :param filename: the name of the file, if appropriate
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	100 :param encoding: the encoding of the file; if not specified, the
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	101 encoding is assumed to be ASCII, UTF-8, or UTF-16, or
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	102 whatever the encoding specified in the XML declaration
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	103 (if any)
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	104 """
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	105 self.source = source
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	106 self.filename = filename
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	107
821114ec4f69 Initial import. cmlenz parents: diff changeset	108 # Setup the Expat parser
316 4ab9edf5e83b Configurable encoding of template files, closing #65. cmlenz parents: 312 diff changeset	109 parser = expat.ParserCreate(encoding, '}')
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	110 parser.buffer_text = True
821114ec4f69 Initial import. cmlenz parents: diff changeset	111 parser.returns_unicode = True
160 faea6db52ef1 Attribute order in parsed XML is now preserved. cmlenz parents: 146 diff changeset	112 parser.ordered_attributes = True
faea6db52ef1 Attribute order in parsed XML is now preserved. cmlenz parents: 146 diff changeset	113
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	114 parser.StartElementHandler = self._handle_start
821114ec4f69 Initial import. cmlenz parents: diff changeset	115 parser.EndElementHandler = self._handle_end
821114ec4f69 Initial import. cmlenz parents: diff changeset	116 parser.CharacterDataHandler = self._handle_data
821114ec4f69 Initial import. cmlenz parents: diff changeset	117 parser.StartDoctypeDeclHandler = self._handle_doctype
821114ec4f69 Initial import. cmlenz parents: diff changeset	118 parser.StartNamespaceDeclHandler = self._handle_start_ns
821114ec4f69 Initial import. cmlenz parents: diff changeset	119 parser.EndNamespaceDeclHandler = self._handle_end_ns
143 ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	120 parser.StartCdataSectionHandler = self._handle_start_cdata
ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	121 parser.EndCdataSectionHandler = self._handle_end_cdata
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	122 parser.ProcessingInstructionHandler = self._handle_pi
460 6b5544bb5a99 Apply patch by Alec Thomas for processing XML declarations (#111). Thanks! cmlenz parents: 458 diff changeset	123 parser.XmlDeclHandler = self._handle_xml_decl
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	124 parser.CommentHandler = self._handle_comment
209 5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	125
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	126 # Tell Expat that we'll handle non-XML entities ourselves
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	127 # (in _handle_other)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	128 parser.DefaultHandler = self._handle_other
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	129 parser.SetParamEntityParsing(expat.XML_PARAM_ENTITY_PARSING_ALWAYS)
209 5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	130 parser.UseForeignDTD()
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	131 parser.ExternalEntityRefHandler = self._build_foreign
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	132
821114ec4f69 Initial import. cmlenz parents: diff changeset	133 self.expat = parser
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	134 self._queue = []
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	135
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	136 def parse(self):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	137 """Generator that parses the XML source, yielding markup events.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	138
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	139 :return: a markup event stream
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	140 :raises ParseError: if the XML text is not well formed
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	141 """
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	142 def _generate():
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	143 try:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	144 bufsize = 4 * 1024 # 4K
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	145 done = False
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	146 while 1:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	147 while not done and len(self._queue) == 0:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	148 data = self.source.read(bufsize)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	149 if data == '': # end of data
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	150 if hasattr(self, 'expat'):
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	151 self.expat.Parse('', True)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	152 del self.expat # get rid of circular references
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	153 done = True
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	154 else:
207 0619a27f5e67 The `XMLParser` now correctly handles unicode input. Closes #43. cmlenz parents: 182 diff changeset	155 if isinstance(data, unicode):
0619a27f5e67 The `XMLParser` now correctly handles unicode input. Closes #43. cmlenz parents: 182 diff changeset	156 data = data.encode('utf-8')
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	157 self.expat.Parse(data, False)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	158 for event in self._queue:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	159 yield event
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	160 self._queue = []
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	161 if done:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	162 break
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	163 except expat.ExpatError, e:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	164 msg = str(e)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	165 raise ParseError(msg, self.filename, e.lineno, e.offset)
146 db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	166 return Stream(_generate()).filter(_coalesce)
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	167
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	168 def __iter__(self):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	169 return iter(self.parse())
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	170
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	171 def _build_foreign(self, context, base, sysid, pubid):
38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	172 parser = self.expat.ExternalEntityParserCreate(context)
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	173 parser.ParseFile(StringIO(self._external_dtd))
293 38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	174 return 1
38adb4aa7df5 Fix a bug in the XML parser, where attributes containing HTML entity references would get pulled out of the attribute value, and instead added as a text node just before the associated start tag. Thanks to Hamish Lawson for [http://groups.google.com/group/genshi/browse_thread/thread/c64eb48676b0ff96/0e6ce786e8820f3d pointing out the problem]. cmlenz parents: 290 diff changeset	175
143 ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	176 def _enqueue(self, kind, data=None, pos=None):
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	177 if pos is None:
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	178 pos = self._getpos()
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	179 if kind is TEXT:
134 df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	180 # Expat reports the end of the text event as current position. We
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	181 # try to fix that up here as much as possible. Unfortunately, the
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	182 # offset is only valid for single-line text. For multi-line text,
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	183 # it is apparently not possible to determine at what offset it
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	184 # started
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	185 if '\n' in data:
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	186 lines = data.splitlines()
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	187 lineno = pos[1] - len(lines) + 1
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	188 offset = -1
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	189 else:
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	190 lineno = pos[1]
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	191 offset = pos[2] - len(data)
df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	192 pos = (pos[0], lineno, offset)
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	193 self._queue.append((kind, data, pos))
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	194
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	195 def _getpos_unknown(self):
134 df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	196 return (self.filename, -1, -1)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	197
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	198 def _getpos(self):
134 df44110ca91d * Improve the accuracy of line numbers for text nodes, so that reported errors about syntax or evaluation errors in expressions point to the right line (not quite perfect yet, though). cmlenz parents: 69 diff changeset	199 return (self.filename, self.expat.CurrentLineNumber,
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	200 self.expat.CurrentColumnNumber)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	201
821114ec4f69 Initial import. cmlenz parents: diff changeset	202 def _handle_start(self, tag, attrib):
403 32b283e1d310 Remove some magic/overhead from `Attrs` creation and manipulation by not automatically wrapping attribute names in `QName`. cmlenz parents: 378 diff changeset	203 attrs = Attrs([(QName(name), value) for name, value in
32b283e1d310 Remove some magic/overhead from `Attrs` creation and manipulation by not automatically wrapping attribute names in `QName`. cmlenz parents: 378 diff changeset	204 zip([iter(attrib)] 2)])
32b283e1d310 Remove some magic/overhead from `Attrs` creation and manipulation by not automatically wrapping attribute names in `QName`. cmlenz parents: 378 diff changeset	205 self._enqueue(START, (QName(tag), attrs))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	206
821114ec4f69 Initial import. cmlenz parents: diff changeset	207 def _handle_end(self, tag):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	208 self._enqueue(END, QName(tag))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	209
821114ec4f69 Initial import. cmlenz parents: diff changeset	210 def _handle_data(self, text):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	211 self._enqueue(TEXT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	212
460 6b5544bb5a99 Apply patch by Alec Thomas for processing XML declarations (#111). Thanks! cmlenz parents: 458 diff changeset	213 def _handle_xml_decl(self, version, encoding, standalone):
6b5544bb5a99 Apply patch by Alec Thomas for processing XML declarations (#111). Thanks! cmlenz parents: 458 diff changeset	214 self._enqueue(XML_DECL, (version, encoding, standalone))
6b5544bb5a99 Apply patch by Alec Thomas for processing XML declarations (#111). Thanks! cmlenz parents: 458 diff changeset	215
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	216 def _handle_doctype(self, name, sysid, pubid, has_internal_subset):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	217 self._enqueue(DOCTYPE, (name, pubid, sysid))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	218
821114ec4f69 Initial import. cmlenz parents: diff changeset	219 def _handle_start_ns(self, prefix, uri):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	220 self._enqueue(START_NS, (prefix or '', uri))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	221
821114ec4f69 Initial import. cmlenz parents: diff changeset	222 def _handle_end_ns(self, prefix):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	223 self._enqueue(END_NS, prefix or '')
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	224
143 ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	225 def _handle_start_cdata(self):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	226 self._enqueue(START_CDATA)
143 ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	227
ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	228 def _handle_end_cdata(self):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	229 self._enqueue(END_CDATA)
143 ef761afcedff CDATA sections in XML input now appear as CDATA sections in the output. This should address the problem with escaping the contents of `<style>` and `<script>` elements, which would only get interpreted correctly if the output was served as `application/xhtml+xml`. Closes #24. cmlenz parents: 140 diff changeset	230
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	231 def _handle_pi(self, target, data):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	232 self._enqueue(PI, (target, data))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	233
821114ec4f69 Initial import. cmlenz parents: diff changeset	234 def _handle_comment(self, text):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	235 self._enqueue(COMMENT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	236
821114ec4f69 Initial import. cmlenz parents: diff changeset	237 def _handle_other(self, text):
821114ec4f69 Initial import. cmlenz parents: diff changeset	238 if text.startswith('&'):
821114ec4f69 Initial import. cmlenz parents: diff changeset	239 # deal with undefined entities
821114ec4f69 Initial import. cmlenz parents: diff changeset	240 try:
856 1e2be9fb3348 Add a couple of fallback imports for Python 3.0. cmlenz parents: 854 diff changeset	241 text = unichr(entities.name2codepoint[text[1:-1]])
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	242 self._enqueue(TEXT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	243 except KeyError:
209 5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	244 filename, lineno, offset = self._getpos()
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	245 error = expat.error('undefined entity "%s": line %d, column %d'
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	246 % (text, lineno, offset))
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	247 error.code = expat.errors.XML_ERROR_UNDEFINED_ENTITY
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	248 error.lineno = lineno
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	249 error.offset = offset
5b422db07359 * Fix bug in handling of undefined entities. Thanks to Arnar for reporting the issue on IRC. cmlenz parents: 207 diff changeset	250 raise error
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	251
821114ec4f69 Initial import. cmlenz parents: diff changeset	252
821114ec4f69 Initial import. cmlenz parents: diff changeset	253 def XML(text):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	254 """Parse the given XML source and return a markup stream.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	255
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	256 Unlike with `XMLParser`, the returned stream is reusable, meaning it can be
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	257 iterated over multiple times:
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	258
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	259 >>> xml = XML('<doc><elem>Foo</elem><elem>Bar</elem></doc>')
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	260 >>> print(xml)
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	261 <doc><elem>Foo</elem><elem>Bar</elem></doc>
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	262 >>> print(xml.select('elem'))
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	263 <elem>Foo</elem><elem>Bar</elem>
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	264 >>> print(xml.select('elem/text()'))
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	265 FooBar
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	266
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	267 :param text: the XML source
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	268 :return: the parsed XML event stream
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	269 :raises ParseError: if the XML text is not well-formed
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	270 """
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	271 return Stream(list(XMLParser(StringIO(text))))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	272
821114ec4f69 Initial import. cmlenz parents: diff changeset	273
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	274 class HTMLParser(html.HTMLParser, object):
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	275 """Parser for HTML input based on the Python `HTMLParser` module.
821114ec4f69 Initial import. cmlenz parents: diff changeset	276
821114ec4f69 Initial import. cmlenz parents: diff changeset	277 This class provides the same interface for generating stream events as
821114ec4f69 Initial import. cmlenz parents: diff changeset	278 `XMLParser`, and attempts to automatically balance tags.
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	279
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	280 The parsing is initiated by iterating over the parser object:
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	281
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	282 >>> parser = HTMLParser(StringIO('<UL compact><LI>Foo</UL>'))
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	283 >>> for kind, data, pos in parser:
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	284 ... print('%s %s' % (kind, data))
857 24733a5854d9 Avoid unicode literals in `repr`s of `QName` and `Namespace` when not necessary. cmlenz parents: 856 diff changeset	285 START (QName('ul'), Attrs([(QName('compact'), u'compact')]))
24733a5854d9 Avoid unicode literals in `repr`s of `QName` and `Namespace` when not necessary. cmlenz parents: 856 diff changeset	286 START (QName('li'), Attrs())
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	287 TEXT Foo
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	288 END li
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	289 END ul
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	290 """
821114ec4f69 Initial import. cmlenz parents: diff changeset	291
821114ec4f69 Initial import. cmlenz parents: diff changeset	292 _EMPTY_ELEMS = frozenset(['area', 'base', 'basefont', 'br', 'col', 'frame',
821114ec4f69 Initial import. cmlenz parents: diff changeset	293 'hr', 'img', 'input', 'isindex', 'link', 'meta',
821114ec4f69 Initial import. cmlenz parents: diff changeset	294 'param'])
821114ec4f69 Initial import. cmlenz parents: diff changeset	295
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	296 def __init__(self, source, filename=None, encoding='utf-8'):
01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	297 """Initialize the parser for the given HTML input.
01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	298
425 5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	299 :param source: the HTML text as a file-like object
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	300 :param filename: the name of the file, if known
5b248708bbed Try to use proper reStructuredText for docstrings throughout. cmlenz parents: 423 diff changeset	301 :param filename: encoding of the file; ignored if the input is unicode
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	302 """
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	303 html.HTMLParser.__init__(self)
821114ec4f69 Initial import. cmlenz parents: diff changeset	304 self.source = source
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	305 self.filename = filename
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	306 self.encoding = encoding
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	307 self._queue = []
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	308 self._open_tags = []
821114ec4f69 Initial import. cmlenz parents: diff changeset	309
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	310 def parse(self):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	311 """Generator that parses the HTML source, yielding markup events.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	312
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	313 :return: a markup event stream
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	314 :raises ParseError: if the HTML text is not well formed
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	315 """
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	316 def _generate():
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	317 try:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	318 bufsize = 4 * 1024 # 4K
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	319 done = False
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	320 while 1:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	321 while not done and len(self._queue) == 0:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	322 data = self.source.read(bufsize)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	323 if data == '': # end of data
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	324 self.close()
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	325 done = True
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	326 else:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	327 self.feed(data)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	328 for kind, data, pos in self._queue:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	329 yield kind, data, pos
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	330 self._queue = []
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	331 if done:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	332 open_tags = self._open_tags
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	333 open_tags.reverse()
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	334 for tag in open_tags:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	335 yield END, QName(tag), pos
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	336 break
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	337 except html.HTMLParseError, e:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	338 msg = '%s: line %d, column %d' % (e.msg, e.lineno, e.offset)
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	339 raise ParseError(msg, self.filename, e.lineno, e.offset)
146 db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	340 return Stream(_generate()).filter(_coalesce)
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	341
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	342 def __iter__(self):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	343 return iter(self.parse())
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	344
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	345 def _enqueue(self, kind, data, pos=None):
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	346 if pos is None:
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	347 pos = self._getpos()
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	348 self._queue.append((kind, data, pos))
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	349
21 eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	350 def _getpos(self):
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	351 lineno, column = self.getpos()
eca77129518a * Include paths are now interpreted relative to the path of the including template. Closes #3. cmlenz parents: 1 diff changeset	352 return (self.filename, lineno, column)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	353
821114ec4f69 Initial import. cmlenz parents: diff changeset	354 def handle_starttag(self, tag, attrib):
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	355 fixed_attrib = []
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	356 for name, value in attrib: # Fixup minimized attributes
039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	357 if value is None:
312 7e743338a799 Follow-up to [385]: also decode attribute values in the `HTMLParser`. cmlenz parents: 311 diff changeset	358 value = unicode(name)
7e743338a799 Follow-up to [385]: also decode attribute values in the `HTMLParser`. cmlenz parents: 311 diff changeset	359 elif not isinstance(value, unicode):
7e743338a799 Follow-up to [385]: also decode attribute values in the `HTMLParser`. cmlenz parents: 311 diff changeset	360 value = value.decode(self.encoding, 'replace')
403 32b283e1d310 Remove some magic/overhead from `Attrs` creation and manipulation by not automatically wrapping attribute names in `QName`. cmlenz parents: 378 diff changeset	361 fixed_attrib.append((QName(name), stripentities(value)))
26 039fc5b87405 * Split out the XPath tests into a separate `unittest`-based file. cmlenz parents: 21 diff changeset	362
182 41db0260ebb1 Renamed `Attributes` to `Attrs` to reduce the verbosity. cmlenz parents: 160 diff changeset	363 self._enqueue(START, (QName(tag), Attrs(fixed_attrib)))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	364 if tag in self._EMPTY_ELEMS:
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	365 self._enqueue(END, QName(tag))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	366 else:
821114ec4f69 Initial import. cmlenz parents: diff changeset	367 self._open_tags.append(tag)
821114ec4f69 Initial import. cmlenz parents: diff changeset	368
821114ec4f69 Initial import. cmlenz parents: diff changeset	369 def handle_endtag(self, tag):
821114ec4f69 Initial import. cmlenz parents: diff changeset	370 if tag not in self._EMPTY_ELEMS:
821114ec4f69 Initial import. cmlenz parents: diff changeset	371 while self._open_tags:
821114ec4f69 Initial import. cmlenz parents: diff changeset	372 open_tag = self._open_tags.pop()
378 fff4a81ffc56 Improve handling of incorrectly nested tags in the HTML parser. cmlenz parents: 376 diff changeset	373 self._enqueue(END, QName(open_tag))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	374 if open_tag.lower() == tag.lower():
821114ec4f69 Initial import. cmlenz parents: diff changeset	375 break
821114ec4f69 Initial import. cmlenz parents: diff changeset	376
821114ec4f69 Initial import. cmlenz parents: diff changeset	377 def handle_data(self, text):
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	378 if not isinstance(text, unicode):
01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	379 text = text.decode(self.encoding, 'replace')
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	380 self._enqueue(TEXT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	381
821114ec4f69 Initial import. cmlenz parents: diff changeset	382 def handle_charref(self, name):
423 7589a0e51001 Applied patch for #106 (handling of hex charrefs in HTML parser). cmlenz parents: 422 diff changeset	383 if name.lower().startswith('x'):
7589a0e51001 Applied patch for #106 (handling of hex charrefs in HTML parser). cmlenz parents: 422 diff changeset	384 text = unichr(int(name[1:], 16))
7589a0e51001 Applied patch for #106 (handling of hex charrefs in HTML parser). cmlenz parents: 422 diff changeset	385 else:
7589a0e51001 Applied patch for #106 (handling of hex charrefs in HTML parser). cmlenz parents: 422 diff changeset	386 text = unichr(int(name))
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	387 self._enqueue(TEXT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	388
821114ec4f69 Initial import. cmlenz parents: diff changeset	389 def handle_entityref(self, name):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	390 try:
856 1e2be9fb3348 Add a couple of fallback imports for Python 3.0. cmlenz parents: 854 diff changeset	391 text = unichr(entities.name2codepoint[name])
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	392 except KeyError:
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	393 text = '&%s;' % name
28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	394 self._enqueue(TEXT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	395
821114ec4f69 Initial import. cmlenz parents: diff changeset	396 def handle_pi(self, data):
376 74b6bf92f0cd Fix parsing of processing instructions in HTML input. cmlenz parents: 326 diff changeset	397 target, data = data.split(None, 1)
74b6bf92f0cd Fix parsing of processing instructions in HTML input. cmlenz parents: 326 diff changeset	398 if data.endswith('?'):
74b6bf92f0cd Fix parsing of processing instructions in HTML input. cmlenz parents: 326 diff changeset	399 data = data[:-1]
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	400 self._enqueue(PI, (target.strip(), data.strip()))
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	401
821114ec4f69 Initial import. cmlenz parents: diff changeset	402 def handle_comment(self, text):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	403 self._enqueue(COMMENT, text)
1 821114ec4f69 Initial import. cmlenz parents: diff changeset	404
821114ec4f69 Initial import. cmlenz parents: diff changeset	405
311 01e2c48f6dfb * The `HTMLParser` class and the `HTML` function now accept an `encoding` parameter to properly deal with bytestring input (defaults to UTF-8). cmlenz parents: 293 diff changeset	406 def HTML(text, encoding='utf-8'):
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	407 """Parse the given HTML source and return a markup stream.
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	408
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	409 Unlike with `HTMLParser`, the returned stream is reusable, meaning it can be
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	410 iterated over multiple times:
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	411
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	412 >>> html = HTML('<body><h1>Foo</h1></body>')
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	413 >>> print(html)
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	414 <body><h1>Foo</h1></body>
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	415 >>> print(html.select('h1'))
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	416 <h1>Foo</h1>
853 4376010bb97e Convert a bunch of print statements to py3k compatible syntax. cmlenz parents: 852 diff changeset	417 >>> print(html.select('h1/text()'))
433 6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	418 Foo
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	419
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	420 :param text: the HTML source
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	421 :return: the parsed XML event stream
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	422 :raises ParseError: if the HTML text is not well-formed, and error recovery
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	423 fails
6d01e91f2a49 More API docs. cmlenz parents: 425 diff changeset	424 """
859 fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	425 return Stream(list(HTMLParser(StringIO(text), encoding=encoding)))
fbe34d12acde More bits of 2to3 related cleanup. cmlenz parents: 857 diff changeset	426
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	427
146 db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	428 def _coalesce(stream):
144 28b56f09a7e1 * Coalesce adjacent text events that the parsers would produce when text crossed the buffer boundaries. Fixes #26. cmlenz parents: 143 diff changeset	429 """Coalesces adjacent TEXT events into a single event."""
146 db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	430 textbuf = []
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	431 textpos = None
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	432 for kind, data, pos in chain(stream, [(None, None, None)]):
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	433 if kind is TEXT:
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	434 textbuf.append(data)
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	435 if textpos is None:
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	436 textpos = pos
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	437 else:
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	438 if textbuf:
852 04945cd67dad Remove usage of unicode literals in a couple of places where they were not strictly necessary. cmlenz parents: 750 diff changeset	439 yield TEXT, ''.join(textbuf), textpos
146 db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	440 del textbuf[:]
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	441 textpos = None
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	442 if kind:
db0dacc1239a Simplifed `CoalesceFilter` (now a function) cmlenz parents: 145 diff changeset	443 yield kind, data, pos

Mercurial > genshi > genshi-test

annotate genshi/input.py @ 930:0ec0a695ec96