cmlenz@500: .. -*- mode: rst; encoding: utf-8 -*-
cmlenz@500: 
cmlenz@500: ==============
cmlenz@500: Stream Filters
cmlenz@500: ==============
cmlenz@500: 
cmlenz@500: `Markup Streams`_ showed how to write filters and how they are applied to
cmlenz@500: markup streams. This page describes the features of the various filters that
cmlenz@500: come with Genshi itself.
cmlenz@500: 
cmlenz@500: .. _`Markup Streams`: streams.html
cmlenz@500: 
cmlenz@500: .. contents:: Contents
cmlenz@500:    :depth: 1
cmlenz@500: .. sectnum::
cmlenz@500: 
cmlenz@500: 
cmlenz@500: HTML Form Filler
cmlenz@500: ================
cmlenz@500: 
cmlenz@820: The filter ``genshi.filters.html.HTMLFormFiller`` can automatically populate an
cmlenz@820: HTML form from values provided as a simple dictionary. When using this filter,
cmlenz@820: you can basically omit any ``value``, ``selected``, or ``checked`` attributes
cmlenz@820: from form controls in your templates, and let the filter do all that work for
cmlenz@820: you.
cmlenz@500: 
cmlenz@500: ``HTMLFormFiller`` takes a dictionary of data to populate the form with, where
cmlenz@500: the keys should match the names of form elements, and the values determine the
cmlenz@820: values of those controls. For example:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@500: 
cmlenz@500:   >>> from genshi.filters import HTMLFormFiller
cmlenz@500:   >>> from genshi.template import MarkupTemplate
cmlenz@820:   
cmlenz@500:   >>> template = MarkupTemplate("""<form>
cmlenz@500:   ...   <p>
cmlenz@500:   ...     <label>User name:
cmlenz@500:   ...       <input type="text" name="username" />
cmlenz@500:   ...     </label><br />
cmlenz@500:   ...     <label>Password:
cmlenz@500:   ...       <input type="password" name="password" />
cmlenz@500:   ...     </label><br />
cmlenz@500:   ...     <label>
cmlenz@500:   ...       <input type="checkbox" name="remember" /> Remember me
cmlenz@500:   ...     </label>
cmlenz@500:   ...   </p>
cmlenz@500:   ... </form>""")
cmlenz@500:   >>> filler = HTMLFormFiller(data=dict(username='john', remember=True))
cmlenz@500:   >>> print template.generate() | filler
cmlenz@500:   <form>
cmlenz@500:     <p>
cmlenz@500:       <label>User name:
cmlenz@500:         <input type="text" name="username" value="john"/>
cmlenz@500:       </label><br/>
cmlenz@500:       <label>Password:
cmlenz@500:         <input type="password" name="password"/>
cmlenz@500:       </label><br/>
cmlenz@500:       <label>
cmlenz@500:         <input type="checkbox" name="remember" checked="checked"/> Remember me
cmlenz@500:       </label>
cmlenz@500:     </p>
cmlenz@500:   </form>
cmlenz@500: 
cmlenz@500: .. note:: This processing is done without in any way reparsing the template
cmlenz@500:           output. As any stream filter it operates after the template output is
cmlenz@500:           generated but *before* that output is actually serialized.
cmlenz@500: 
cmlenz@500: The filter will of course also handle radio buttons as well as ``<select>`` and
cmlenz@500: ``<textarea>`` elements. For radio buttons to be marked as checked, the value in
cmlenz@500: the data dictionary needs to match the ``value`` attribute of the ``<input>``
cmlenz@500: element, or evaluate to a truth value if the element has no such attribute. For
cmlenz@500: options in a ``<select>`` box to be marked as selected, the value in the data
cmlenz@500: dictionary needs to match the ``value`` attribute of the ``<option>`` element,
cmlenz@500: or the text content of the option if it has no ``value`` attribute. Password and
cmlenz@500: file input fields are not populated, as most browsers would ignore that anyway
cmlenz@500: for security reasons.
cmlenz@500: 
cmlenz@500: You'll want to make sure that the values in the data dictionary have already
cmlenz@500: been converted to strings. While the filter may be able to deal with non-string
cmlenz@500: data in some cases (such as check boxes), in most cases it will either not
cmlenz@500: attempt any conversion or not produce the desired results.
cmlenz@500: 
cmlenz@500: You can restrict the form filler to operate only on a specific ``<form>`` by
cmlenz@500: passing either the ``id`` or the ``name`` keyword argument to the initializer.
cmlenz@500: If either of those is specified, the filter will only apply to form tags with
cmlenz@500: an attribute matching the specified value.
cmlenz@500: 
cmlenz@500: 
cmlenz@500: HTML Sanitizer
cmlenz@500: ==============
cmlenz@500: 
cmlenz@820: The filter ``genshi.filters.html.HTMLSanitizer`` filter can be used to clean up
cmlenz@500: user-submitted HTML markup, removing potentially dangerous constructs that could
cmlenz@820: be used for various kinds of abuse, such as cross-site scripting (XSS) attacks:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@500: 
cmlenz@500:   >>> from genshi.filters import HTMLSanitizer
cmlenz@500:   >>> from genshi.input import HTML
cmlenz@820:   
cmlenz@500:   >>> html = HTML("""<div>
cmlenz@500:   ...   <p>Innocent looking text.</p>
cmlenz@500:   ...   <script>alert("Danger: " + document.cookie)</script>
cmlenz@500:   ... </div>""")
cmlenz@500:   >>> sanitize = HTMLSanitizer()
cmlenz@500:   >>> print html | sanitize
cmlenz@500:   <div>
cmlenz@500:     <p>Innocent looking text.</p>
cmlenz@500:   </div>
cmlenz@500: 
cmlenz@500: In this example, the ``<script>`` tag was removed from the output.
cmlenz@500: 
cmlenz@500: You can determine which tags and attributes should be allowed by initializing
cmlenz@500: the filter with corresponding sets. See the API documentation for more
cmlenz@500: information.
cmlenz@500: 
cmlenz@500: Inline ``style`` attributes are forbidden by default. If you allow them, the
cmlenz@500: filter will still perform sanitization on the contents any encountered inline
cmlenz@500: styles: the proprietary ``expression()`` function (supported only by Internet
cmlenz@500: Explorer) is removed, and any property using an ``url()`` which a potentially
cmlenz@820: dangerous URL scheme (such as ``javascript:``) are also stripped out:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@500: 
cmlenz@500:   >>> from genshi.filters import HTMLSanitizer
cmlenz@500:   >>> from genshi.input import HTML
cmlenz@820:   
cmlenz@500:   >>> html = HTML("""<div>
cmlenz@500:   ...   <br style="background: url(javascript:alert(document.cookie); color: #000" />
cmlenz@500:   ... </div>""")
cmlenz@500:   >>> sanitize = HTMLSanitizer(safe_attrs=HTMLSanitizer.SAFE_ATTRS | set(['style']))
cmlenz@500:   >>> print html | sanitize
cmlenz@500:   <div>
cmlenz@500:     <br style="color: #000"/>
cmlenz@500:   </div>
cmlenz@500: 
cmlenz@500: .. warning:: You should probably not rely on the ``style`` filtering, as
cmlenz@500:              sanitizing mixed HTML, CSS, and Javascript is very complicated and
cmlenz@500:              suspect to various browser bugs. If you can somehow get away with
cmlenz@500:              not allowing inline styles in user-submitted content, that would
cmlenz@500:              definitely be the safer route to follow.
cmlenz@820: 
cmlenz@820: 
cmlenz@820: Transformer
cmlenz@820: ===========
cmlenz@820: 
cmlenz@820: The filter ``genshi.filters.transform.Transformer`` provides a convenient way to
cmlenz@820: transform or otherwise work with markup event streams. It allows you to specify
cmlenz@820: which parts of the stream you're interested in with XPath expressions, and then
cmlenz@820: attach a variety of transformations to the parts that match:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@820: 
cmlenz@820:   >>> from genshi.builder import tag
cmlenz@820:   >>> from genshi.core import TEXT
cmlenz@820:   >>> from genshi.filters import Transformer
cmlenz@820:   >>> from genshi.input import HTML
cmlenz@820:   
cmlenz@820:   >>> html = HTML('''<html>
cmlenz@820:   ...   <head><title>Some Title</title></head>
cmlenz@820:   ...   <body>
cmlenz@820:   ...     Some <em>body</em> text.
cmlenz@820:   ...   </body>
cmlenz@820:   ... </html>''')
cmlenz@820:   
cmlenz@820:   >>> print html | Transformer('body/em').map(unicode.upper, TEXT) \
cmlenz@820:   ...                                    .unwrap().wrap(tag.u).end() \
cmlenz@820:   ...                                    .select('body/u') \
cmlenz@820:   ...                                    .prepend('underlined ')
cmlenz@820:   <html>
cmlenz@820:     <head><title>Some Title</title></head>
cmlenz@820:     <body>
cmlenz@820:       Some <u>underlined BODY</u> text.
cmlenz@820:     </body>
cmlenz@820:   </html>
cmlenz@820: 
cmlenz@820: This example sets up a transformation that:
cmlenz@820: 
cmlenz@820:  1. matches any `<em>` element anywhere in the body,
cmlenz@820:  2. uppercases any text nodes in the element,
cmlenz@820:  3. strips off the `<em>` start and close tags,
cmlenz@820:  4. wraps the content in a `<u>` tag, and
cmlenz@820:  5. inserts the text `underlined` inside the `<u>` tag.
cmlenz@820: 
cmlenz@820: A number of commonly useful transformations are available for this filter.
cmlenz@820: Please consult the API documentation a complete list.
cmlenz@820: 
cmlenz@820: In addition, you can also perform custom transformations. For example, the
cmlenz@820: following defines a transformation that changes the name of a tag:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@820: 
cmlenz@820:   >>> from genshi import QName
cmlenz@820:   >>> from genshi.filters.transform import ENTER, EXIT
cmlenz@820:   
cmlenz@820:   >>> class RenameTransformation(object):
cmlenz@820:   ...    def __init__(self, name):
cmlenz@820:   ...        self.name = QName(name)
cmlenz@820:   ...    def __call__(self, stream):
cmlenz@820:   ...        for mark, (kind, data, pos) in stream:
cmlenz@820:   ...            if mark is ENTER:
cmlenz@820:   ...                data = self.name, data[1]
cmlenz@820:   ...            elif mark is EXIT:
cmlenz@820:   ...                data = self.name
cmlenz@820:   ...            yield mark, (kind, data, pos)
cmlenz@820: 
cmlenz@820: A transformation can be any callable object that accepts an augmented event
cmlenz@820: stream. In this case we define a class, so that we can initialize it with the
cmlenz@820: tag name.
cmlenz@820: 
cmlenz@820: Custom transformations can be applied using the `apply()` method of a
cmlenz@820: transformer instance:
cmlenz@820: 
cmlenz@820: .. code-block:: pycon
cmlenz@820: 
cmlenz@820:   >>> xform = Transformer('body//em').map(unicode.upper, TEXT) \
cmlenz@820:   >>> xform = xform.apply(RenameTransformation('u'))
cmlenz@820:   >>> print html | xform
cmlenz@820:   <html>
cmlenz@820:     <head><title>Some Title</title></head>
cmlenz@820:     <body>
cmlenz@820:       Some <u>BODY</u> text.
cmlenz@820:     </body>
cmlenz@820:   </html>
cmlenz@820: 
cmlenz@820: .. note:: The transformation filter was added in Genshi 0.5.
cmlenz@820: 
cmlenz@820: 
cmlenz@820: Translator
cmlenz@820: ==========
cmlenz@820: 
cmlenz@820: The ``genshi.filters.i18n.Translator`` filter implements basic support for
cmlenz@820: internationalizing and localizing templates. When used as a filter, it
cmlenz@820: translates a configurable set of text nodes and attribute values using a
cmlenz@820: ``gettext``-style translation function.
cmlenz@820: 
cmlenz@820: The ``Translator`` class also defines the ``extract`` class method, which can
cmlenz@820: be used to extract localizable messages from a template.
cmlenz@820: 
cmlenz@820: Please refer to the API documentation for more information on this filter.
cmlenz@820: 
cmlenz@820: .. note:: The translation filter was added in Genshi 0.4.