Mercurial > genshi > mirror

annotate genshi/template/interpolation.py @ 724:919809e55d16 experimental-match-fastpaths

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
update to trunk to track r847, fixing python 2.4 compatibility issues in speedup (and fixing copyrights, apparently :))
author aflett
date Mon, 21 Apr 2008 19:36:53 +0000
parents d143dd73789b
children 8825ac5014b1
rev   line source
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
1 # -*- coding: utf-8 -*-
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
2 #
724
919809e55d16 update to trunk to track r847, fixing python 2.4 compatibility issues in speedup (and fixing copyrights, apparently :))
aflett
parents: 718
diff changeset
3 # Copyright (C) 2007-2008 Edgewall Software
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
4 # All rights reserved.
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
5 #
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
6 # This software is licensed as described in the file COPYING, which
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
7 # you should have received as part of this distribution. The terms
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
8 # are also available at http://genshi.edgewall.org/wiki/License.
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
9 #
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
10 # This software consists of voluntary contributions made by many
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
11 # individuals. For the exact contribution history, see the revision
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
12 # history and logs, available at http://genshi.edgewall.org/log/.
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
13
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
14 """String interpolation routines, i.e. the splitting up a given text into some
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
15 parts that are literal strings, and others that are Python expressions.
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
16 """
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
17
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
18 from itertools import chain
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
19 import os
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
20 from tokenize import tokenprog
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
21
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
22 from genshi.core import TEXT
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
23 from genshi.template.base import TemplateSyntaxError, EXPR
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
24 from genshi.template.eval import Expression
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
25
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
26 __all__ = ['interpolate']
425
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
27 __docformat__ = 'restructuredtext en'
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
28
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
29 NAMESTART = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_'
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
30 NAMECHARS = NAMESTART + '.0123456789'
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
31 PREFIX = '$'
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
32
718
d143dd73789b update to trunk through r833
aflett
parents: 606
diff changeset
33 def interpolate(text, filepath=None, lineno=-1, offset=0, lookup='strict'):
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
34 """Parse the given string and extract expressions.
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
35
425
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
36 This function is a generator that yields `TEXT` events for literal strings,
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
37 and `EXPR` events for expressions, depending on the results of parsing the
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
38 string.
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
39
442
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
40 >>> for kind, data, pos in interpolate("hey ${foo}bar"):
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
41 ... print kind, `data`
442
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
42 TEXT u'hey '
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
43 EXPR Expression('foo')
442
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
44 TEXT u'bar'
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
45
425
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
46 :param text: the text to parse
718
d143dd73789b update to trunk through r833
aflett
parents: 606
diff changeset
47 :param filepath: absolute path to the file in which the text was found
d143dd73789b update to trunk through r833
aflett
parents: 606
diff changeset
48 (optional)
425
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
49 :param lineno: the line number at which the text was found (optional)
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
50 :param offset: the column number at which the text starts in the source
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
51 (optional)
442
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
52 :param lookup: the variable lookup mechanism; either "lenient" (the
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
53 default), "strict", or a custom lookup class
425
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
54 :return: a list of `TEXT` and `EXPR` events
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
55 :raise TemplateSyntaxError: when a syntax error in an expression is
073640758a42 Try to use proper reStructuredText for docstrings throughout.
cmlenz
parents: 422
diff changeset
56 encountered
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
57 """
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
58 pos = [filepath, lineno, offset]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
59
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
60 textbuf = []
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
61 textpos = None
422
5d08a744636e More work to include absolute file paths in exceptions.
cmlenz
parents: 407
diff changeset
62 for is_expr, chunk in chain(lex(text, pos, filepath), [(True, '')]):
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
63 if is_expr:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
64 if textbuf:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
65 yield TEXT, u''.join(textbuf), textpos
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
66 del textbuf[:]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
67 textpos = None
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
68 if chunk:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
69 try:
442
97544725bb7f Back out [510] and instead implement configurable error handling modes. The default is the old 0.3.x behaviour, but more strict error handling is available as an option.
cmlenz
parents: 425
diff changeset
70 expr = Expression(chunk.strip(), pos[0], pos[1],
601
59fbd7586454 Simplify implementation of `py:with` directive by compiling to a `Suite`, instead of manually breaking up the statement and compiling each part to an `Expression`. Also, the first line of code in a `Suite` is now stored as the "function name" of the bytecode, so that it shows up in tracebacks.
cmlenz
parents: 544
diff changeset
71 lookup=lookup)
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
72 yield EXPR, expr, tuple(pos)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
73 except SyntaxError, err:
422
5d08a744636e More work to include absolute file paths in exceptions.
cmlenz
parents: 407
diff changeset
74 raise TemplateSyntaxError(err, filepath, pos[1],
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
75 pos[2] + (err.offset or 0))
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
76 else:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
77 textbuf.append(chunk)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
78 if textpos is None:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
79 textpos = tuple(pos)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
80
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
81 if '\n' in chunk:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
82 lines = chunk.splitlines()
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
83 pos[1] += len(lines) - 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
84 pos[2] += len(lines[-1])
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
85 else:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
86 pos[2] += len(chunk)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
87
422
5d08a744636e More work to include absolute file paths in exceptions.
cmlenz
parents: 407
diff changeset
88 def lex(text, textpos, filepath):
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
89 offset = pos = 0
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
90 end = len(text)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
91 escaped = False
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
92
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
93 while 1:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
94 if escaped:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
95 offset = text.find(PREFIX, offset + 2)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
96 escaped = False
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
97 else:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
98 offset = text.find(PREFIX, pos)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
99 if offset < 0 or offset == end - 1:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
100 break
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
101 next = text[offset + 1]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
102
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
103 if next == '{':
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
104 if offset > pos:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
105 yield False, text[pos:offset]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
106 pos = offset + 2
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
107 level = 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
108 while level:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
109 match = tokenprog.match(text, pos)
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
110 if match is None:
422
5d08a744636e More work to include absolute file paths in exceptions.
cmlenz
parents: 407
diff changeset
111 raise TemplateSyntaxError('invalid syntax', filepath,
5d08a744636e More work to include absolute file paths in exceptions.
cmlenz
parents: 407
diff changeset
112 *textpos[1:])
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
113 pos = match.end()
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
114 tstart, tend = match.regs[3]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
115 token = text[tstart:tend]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
116 if token == '{':
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
117 level += 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
118 elif token == '}':
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
119 level -= 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
120 yield True, text[offset + 2:pos - 1]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
121
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
122 elif next in NAMESTART:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
123 if offset > pos:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
124 yield False, text[pos:offset]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
125 pos = offset
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
126 pos += 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
127 while pos < end:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
128 char = text[pos]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
129 if char not in NAMECHARS:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
130 break
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
131 pos += 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
132 yield True, text[offset + 1:pos].strip()
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
133
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
134 elif not escaped and next == PREFIX:
526
8779b2a775d9 Fix interpolation of short-form expressions that include literal text before the expression. Thanks to Alec for reporting the issue.
cmlenz
parents: 442
diff changeset
135 if offset > pos:
8779b2a775d9 Fix interpolation of short-form expressions that include literal text before the expression. Thanks to Alec for reporting the issue.
cmlenz
parents: 442
diff changeset
136 yield False, text[pos:offset]
407
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
137 escaped = True
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
138 pos = offset + 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
139
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
140 else:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
141 yield False, text[pos:offset + 1]
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
142 pos = offset + 1
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
143
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
144 if pos < end:
f37d8e6acdf2 Move string interpolation code into separate module (`genshi.template.interpolation`).
cmlenz
parents:
diff changeset
145 yield False, text[pos:]
Copyright (C) 2012-2017 Edgewall Software