Mercurial > babel > mirror

annotate babel/messages/extract.py @ 569:1b801a0cb2cb trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
author fschwarz
date Mon, 26 Sep 2011 20:01:01 +0000
parents 86524be05b60
children 99983baf1067
rev   line source
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
1 # -*- coding: utf-8 -*-
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
2 #
530
ca203b2af83c Update the copyright line.
jruigrok
parents: 525
diff changeset
3 # Copyright (C) 2007-2011 Edgewall Software
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
4 # All rights reserved.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
5 #
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
6 # This software is licensed as described in the file COPYING, which
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
7 # you should have received as part of this distribution. The terms
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
8 # are also available at http://babel.edgewall.org/wiki/License.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
9 #
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
10 # This software consists of voluntary contributions made by many
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
11 # individuals. For the exact contribution history, see the revision
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
12 # history and logs, available at http://babel.edgewall.org/log/.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
13
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
14 """Basic infrastructure for extracting localizable messages from source files.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
15
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
16 This module defines an extensible system for collecting localizable message
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
17 strings from a variety of sources. A native extractor for Python source files
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
18 is builtin, extractors for other sources can be added using very simple plugins.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
19
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
20 The main entry points into the extraction functionality are the functions
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
21 `extract_from_dir` and `extract_from_file`.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
22 """
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
23
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
24 import os
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
25 import sys
162
32be08ab2440 alphabetize imports
pjenvey
parents: 154
diff changeset
26 from tokenize import generate_tokens, COMMENT, NAME, OP, STRING
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
27
525
2baa2cedd6f9 Cleanup round #1: get rid of the frozenset/set utility code and imports.
jruigrok
parents: 426
diff changeset
28 from babel.util import parse_encoding, pathmatch, relpath
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
29 from textwrap import dedent
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
30
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
31 __all__ = ['extract', 'extract_from_dir', 'extract_from_file']
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
32 __docformat__ = 'restructuredtext en'
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
33
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
34 GROUP_NAME = 'babel.extractors'
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
35
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
36 DEFAULT_KEYWORDS = {
10
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
37 '_': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
38 'gettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
39 'ngettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
40 'ugettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
41 'ungettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
42 'dgettext': (2,),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
43 'dngettext': (2, 3),
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
44 'N_': None,
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
45 'pgettext': ((1, 'c'), 2)
10
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
46 }
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
47
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
48 DEFAULT_MAPPING = [('**.py', 'python')]
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
49
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
50 empty_msgid_warning = (
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
51 '%s: warning: Empty msgid. It is reserved by GNU gettext: gettext("") '
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
52 'returns the header entry with meta information, not the empty string.')
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
53
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
54
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
55 def _strip_comment_tags(comments, tags):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
56 """Helper function for `extract` that strips comment tags from strings
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
57 in a list of comment lines. This functions operates in-place.
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
58 """
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
59 def _strip(line):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
60 for tag in tags:
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
61 if line.startswith(tag):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
62 return line[len(tag):].strip()
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
63 return line
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
64 comments[:] = map(_strip, comments)
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
65
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
66
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
67 def extract_from_dir(dirname=os.getcwd(), method_map=DEFAULT_MAPPING,
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
68 options_map=None, keywords=DEFAULT_KEYWORDS,
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
69 comment_tags=(), callback=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
70 """Extract messages from any source files found in the given directory.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
71
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
72 This function generates tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
73
82
540bb484f6e0 Missed some param's documentation regarding translator comments.
palgarvio
parents: 81
diff changeset
74 ``(filename, lineno, message, comments)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
75
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
76 Which extraction method is used per file is determined by the `method_map`
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
77 parameter, which maps extended glob patterns to extraction method names.
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
78 For example, the following is the default mapping:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
79
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
80 >>> method_map = [
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
81 ... ('**.py', 'python')
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
82 ... ]
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
83
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
84 This basically says that files with the filename extension ".py" at any
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
85 level inside the directory should be processed by the "python" extraction
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
86 method. Files that don't match any of the mapping patterns are ignored. See
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
87 the documentation of the `pathmatch` function for details on the pattern
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
88 syntax.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
89
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
90 The following extended mapping would also use the "genshi" extraction
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
91 method on any file in "templates" subdirectory:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
92
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
93 >>> method_map = [
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
94 ... ('**/templates/**.*', 'genshi'),
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
95 ... ('**.py', 'python')
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
96 ... ]
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
97
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
98 The dictionary provided by the optional `options_map` parameter augments
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
99 these mappings. It uses extended glob patterns as keys, and the values are
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
100 dictionaries mapping options names to option values (both strings).
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
101
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
102 The glob patterns of the `options_map` do not necessarily need to be the
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
103 same as those used in the method mapping. For example, while all files in
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
104 the ``templates`` folders in an application may be Genshi applications, the
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
105 options for those files may differ based on extension:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
106
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
107 >>> options_map = {
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
108 ... '**/templates/**.txt': {
144
14fe2a8fb842 Some doc fixes.
cmlenz
parents: 138
diff changeset
109 ... 'template_class': 'genshi.template:TextTemplate',
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
110 ... 'encoding': 'latin-1'
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
111 ... },
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
112 ... '**/templates/**.html': {
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
113 ... 'include_attrs': ''
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
114 ... }
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
115 ... }
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
116
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
117 :param dirname: the path to the directory to extract messages from
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
118 :param method_map: a list of ``(pattern, method)`` tuples that maps of
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
119 extraction method names to extended glob patterns
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
120 :param options_map: a dictionary of additional options (optional)
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
121 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
122 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
123 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
124 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
125 :param comment_tags: a list of tags of translator comments to search for
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
126 and include in the results
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
127 :param callback: a function that is called for every file that message are
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
128 extracted from, just before the extraction itself is
75
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
129 performed; the function is passed the filename, the name
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
130 of the extraction method and and the options dictionary as
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
131 positional arguments, in that order
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
132 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
133 tags to be removed from the collected comments.
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
134 :return: an iterator over ``(filename, lineno, funcname, message, context)``
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
135 tuples
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
136 :rtype: ``iterator``
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
137 :see: `pathmatch`
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
138 """
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
139 if options_map is None:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
140 options_map = {}
56
f40fc143439c Add actual data structures for handling message catalogs, so that more code can be reused here between the frontends.
cmlenz
parents: 54
diff changeset
141
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
142 absname = os.path.abspath(dirname)
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
143 for root, dirnames, filenames in os.walk(absname):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
144 for subdir in dirnames:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
145 if subdir.startswith('.') or subdir.startswith('_'):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
146 dirnames.remove(subdir)
154
31478eb3fb9e The default ordering of messages in generated POT files, which is based on the order those messages are found when walking the source tree, is no longer subject to differences between platforms; directory and file names are now always sorted alphabetically.
cmlenz
parents: 147
diff changeset
147 dirnames.sort()
31478eb3fb9e The default ordering of messages in generated POT files, which is based on the order those messages are found when walking the source tree, is no longer subject to differences between platforms; directory and file names are now always sorted alphabetically.
cmlenz
parents: 147
diff changeset
148 filenames.sort()
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
149 for filename in filenames:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
150 filename = relpath(
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
151 os.path.join(root, filename).replace(os.sep, '/'),
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
152 dirname
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
153 )
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
154 for pattern, method in method_map:
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
155 if pathmatch(pattern, filename):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
156 filepath = os.path.join(absname, filename)
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
157 options = {}
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
158 for opattern, odict in options_map.items():
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
159 if pathmatch(opattern, filename):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
160 options = odict
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
161 if callback:
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
162 callback(filename, method, options)
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
163 for lineno, message, comments, context in \
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
164 extract_from_file(method, filepath,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
165 keywords=keywords,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
166 comment_tags=comment_tags,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
167 options=options,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
168 strip_comment_tags=
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
169 strip_comment_tags):
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
170 yield filename, lineno, message, comments, context
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
171 break
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
172
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
173
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
174 def extract_from_file(method, filename, keywords=DEFAULT_KEYWORDS,
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
175 comment_tags=(), options=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
176 """Extract messages from a specific file.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
177
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
178 This function returns a list of tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
179
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
180 ``(lineno, funcname, message)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
181
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
182 :param filename: the path to the file to extract messages from
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
183 :param method: a string specifying the extraction method (.e.g. "python")
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
184 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
185 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
186 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
187 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
188 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
189 in the results
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
190 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
191 tags to be removed from the collected comments.
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
192 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
193 :return: the list of extracted messages
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
194 :rtype: `list`
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
195 """
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
196 fileobj = open(filename, 'U')
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
197 try:
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
198 return list(extract(method, fileobj, keywords, comment_tags, options,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
199 strip_comment_tags))
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
200 finally:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
201 fileobj.close()
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
202
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
203
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
204 def extract(method, fileobj, keywords=DEFAULT_KEYWORDS, comment_tags=(),
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
205 options=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
206 """Extract messages from the given file-like object using the specified
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
207 extraction method.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
208
560
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
209 This function returns tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
210
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
211 ``(lineno, message, comments)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
212
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
213 The implementation dispatches the actual extraction to plugins, based on the
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
214 value of the ``method`` parameter.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
215
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
216 >>> source = '''# foo module
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
217 ... def run(argv):
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
218 ... print _('Hello, world!')
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
219 ... '''
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
220
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
221 >>> from StringIO import StringIO
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
222 >>> for message in extract('python', StringIO(source)):
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
223 ... print message
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
224 (3, u'Hello, world!', [], None)
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
225
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
226 :param method: a string specifying the extraction method (.e.g. "python");
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
227 if this is a simple name, the extraction function will be
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
228 looked up by entry point; if it is an explicit reference
329
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
229 to a function (of the form ``package.module:funcname`` or
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
230 ``package.module.funcname``), the corresponding function
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
231 will be imported and used
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
232 :param fileobj: the file-like object the messages should be extracted from
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
233 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
234 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
235 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
236 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
237 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
238 in the results
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
239 :param options: a dictionary of additional options (optional)
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
240 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
241 tags to be removed from the collected comments.
560
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
242 :return: an iterator over ``(lineno, message, comments)`` tuples
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
243 :rtype: `iterator`
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
244 :raise ValueError: if the extraction method is not registered
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
245 """
322
b03a6a87e4cb fix invalid message extraction methods causing:
pjenvey
parents: 258
diff changeset
246 func = None
329
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
247 if ':' in method or '.' in method:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
248 if ':' not in method:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
249 lastdot = method.rfind('.')
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
250 module, attrname = method[:lastdot], method[lastdot + 1:]
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
251 else:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
252 module, attrname = method.split(':', 1)
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
253 func = getattr(__import__(module, {}, {}, [attrname]), attrname)
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
254 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
255 try:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
256 from pkg_resources import working_set
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
257 except ImportError:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
258 # pkg_resources is not available, so we resort to looking up the
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
259 # builtin extractors directly
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
260 builtin = {'ignore': extract_nothing, 'python': extract_python}
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
261 func = builtin.get(method)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
262 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
263 for entry_point in working_set.iter_entry_points(GROUP_NAME,
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
264 method):
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
265 func = entry_point.load(require=True)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
266 break
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
267 if func is None:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
268 raise ValueError('Unknown extraction method %r' % method)
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
269
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
270 results = func(fileobj, keywords.keys(), comment_tags,
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
271 options=options or {})
366
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
272
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
273 for lineno, funcname, messages, comments in results:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
274 if funcname:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
275 spec = keywords[funcname] or (1,)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
276 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
277 spec = (1,)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
278 if not isinstance(messages, (list, tuple)):
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
279 messages = [messages]
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
280 if not messages:
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
281 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
282
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
283 # Validate the messages against the keyword's specification
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
284 context = None
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
285 msgs = []
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
286 invalid = False
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
287 # last_index is 1 based like the keyword spec
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
288 last_index = len(messages)
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
289 for index in spec:
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
290 if isinstance(index, tuple):
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
291 context = messages[index[0] - 1]
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
292 continue
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
293 if last_index < index:
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
294 # Not enough arguments
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
295 invalid = True
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
296 break
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
297 message = messages[index - 1]
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
298 if message is None:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
299 invalid = True
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
300 break
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
301 msgs.append(message)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
302 if invalid:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
303 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
304
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
305 # keyword spec indexes are 1 based, therefore '-1'
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
306 if isinstance(spec[0], tuple):
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
307 # context-aware *gettext method
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
308 first_msg_index = spec[1] - 1
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
309 else:
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
310 first_msg_index = spec[0] - 1
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
311 if not messages[first_msg_index]:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
312 # An empty string msgid isn't valid, emit a warning
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
313 where = '%s:%i' % (hasattr(fileobj, 'name') and \
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
314 fileobj.name or '(unknown)', lineno)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
315 print >> sys.stderr, empty_msgid_warning % where
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
316 continue
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
317
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
318 messages = tuple(msgs)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
319 if len(messages) == 1:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
320 messages = messages[0]
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
321
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
322 if strip_comment_tags:
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
323 _strip_comment_tags(comments, comment_tags)
569
1b801a0cb2cb Support for context-aware methods during message extraction (fixes #229, patch by David Rios)
fschwarz
parents: 560
diff changeset
324 yield lineno, messages, comments, context
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
325
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
326
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
327 def extract_nothing(fileobj, keywords, comment_tags, options):
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
328 """Pseudo extractor that does not actually extract anything, but simply
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
329 returns an empty list.
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
330 """
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
331 return []
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
332
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
333
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
334 def extract_python(fileobj, keywords, comment_tags, options):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
335 """Extract messages from Python source code.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
336
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
337 :param fileobj: the seekable, file-like object the messages should be
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
338 extracted from
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
339 :param keywords: a list of keywords (i.e. function names) that should be
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
340 recognized as translation functions
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
341 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
342 in the results
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
343 :param options: a dictionary of additional options (optional)
81
85af04c72ccd Fixed and added some documentation about the translator comments implemented in [81].
palgarvio
parents: 80
diff changeset
344 :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
345 :rtype: ``iterator``
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
346 """
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
347 funcname = lineno = message_lineno = None
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
348 call_stack = -1
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
349 buf = []
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
350 messages = []
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
351 translator_comments = []
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
352 in_def = in_translator_comments = False
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
353 comment_tag = None
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
354
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
355 encoding = parse_encoding(fileobj) or options.get('encoding', 'iso-8859-1')
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
356
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
357 tokens = generate_tokens(fileobj.readline)
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
358 for tok, value, (lineno, _), _, _ in tokens:
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
359 if call_stack == -1 and tok == NAME and value in ('def', 'class'):
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
360 in_def = True
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
361 elif tok == OP and value == '(':
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
362 if in_def:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
363 # Avoid false positives for declarations such as:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
364 # def gettext(arg='message'):
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
365 in_def = False
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
366 continue
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
367 if funcname:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
368 message_lineno = lineno
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
369 call_stack += 1
223
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
370 elif in_def and tok == OP and value == ':':
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
371 # End of a class definition without parens
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
372 in_def = False
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
373 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
374 elif call_stack == -1 and tok == COMMENT:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
375 # Strip the comment token from the line
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
376 value = value.decode(encoding)[1:].strip()
147
63a93d33511a simplify
pjenvey
parents: 144
diff changeset
377 if in_translator_comments and \
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
378 translator_comments[-1][0] == lineno - 1:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
379 # We're already inside a translator comment, continue appending
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
380 translator_comments.append((lineno, value))
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
381 continue
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
382 # If execution reaches this point, let's see if comment line
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
383 # starts with one of the comment tags
85
04a2f16bdd04 Fixed de-pluralization bug introduced in [85] regarding the extraction of translator comments.
palgarvio
parents: 84
diff changeset
384 for comment_tag in comment_tags:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
385 if value.startswith(comment_tag):
147
63a93d33511a simplify
pjenvey
parents: 144
diff changeset
386 in_translator_comments = True
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
387 translator_comments.append((lineno, value))
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
388 break
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
389 elif funcname and call_stack == 0:
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
390 if tok == OP and value == ')':
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
391 if buf:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
392 messages.append(''.join(buf))
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
393 del buf[:]
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
394 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
395 messages.append(None)
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
396
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
397 if len(messages) > 1:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
398 messages = tuple(messages)
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
399 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
400 messages = messages[0]
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
401 # Comments don't apply unless they immediately preceed the
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
402 # message
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
403 if translator_comments and \
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
404 translator_comments[-1][0] < message_lineno - 1:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
405 translator_comments = []
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
406
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
407 yield (message_lineno, funcname, messages,
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
408 [comment[1] for comment in translator_comments])
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
409
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
410 funcname = lineno = message_lineno = None
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
411 call_stack = -1
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
412 messages = []
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
413 translator_comments = []
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
414 in_translator_comments = False
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
415 elif tok == STRING:
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
416 # Unwrap quotes in a safe manner, maintaining the string's
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
417 # encoding
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
418 # https://sourceforge.net/tracker/?func=detail&atid=355470&
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
419 # aid=617979&group_id=5470
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
420 value = eval('# coding=%s\n%s' % (encoding, value),
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
421 {'__builtins__':{}}, {})
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
422 if isinstance(value, str):
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
423 value = value.decode(encoding)
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
424 buf.append(value)
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
425 elif tok == OP and value == ',':
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
426 if buf:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
427 messages.append(''.join(buf))
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
428 del buf[:]
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
429 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
430 messages.append(None)
366
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
431 if translator_comments:
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
432 # We have translator comments, and since we're on a
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
433 # comma(,) user is allowed to break into a new line
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
434 # Let's increase the last comment's lineno in order
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
435 # for the comment to still be a valid one
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
436 old_lineno, old_comment = translator_comments.pop()
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
437 translator_comments.append((old_lineno+1, old_comment))
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
438 elif call_stack > 0 and tok == OP and value == ')':
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
439 call_stack -= 1
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
440 elif funcname and call_stack == -1:
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
441 funcname = None
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
442 elif tok == NAME and value in keywords:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
443 funcname = value
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
444
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
445
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
446 def extract_javascript(fileobj, keywords, comment_tags, options):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
447 """Extract messages from JavaScript source code.
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
448
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
449 :param fileobj: the seekable, file-like object the messages should be
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
450 extracted from
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
451 :param keywords: a list of keywords (i.e. function names) that should be
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
452 recognized as translation functions
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
453 :param comment_tags: a list of translator tags to search for and include
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
454 in the results
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
455 :param options: a dictionary of additional options (optional)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
456 :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
457 :rtype: ``iterator``
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
458 """
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
459 from babel.messages.jslexer import tokenize, unquote_string
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
460 funcname = message_lineno = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
461 messages = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
462 last_argument = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
463 translator_comments = []
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
464 concatenate_next = False
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
465 encoding = options.get('encoding', 'utf-8')
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
466 last_token = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
467 call_stack = -1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
468
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
469 for token in tokenize(fileobj.read().decode(encoding)):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
470 if token.type == 'operator' and token.value == '(':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
471 if funcname:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
472 message_lineno = token.lineno
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
473 call_stack += 1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
474
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
475 elif call_stack == -1 and token.type == 'linecomment':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
476 value = token.value[2:].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
477 if translator_comments and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
478 translator_comments[-1][0] == token.lineno - 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
479 translator_comments.append((token.lineno, value))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
480 continue
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
481
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
482 for comment_tag in comment_tags:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
483 if value.startswith(comment_tag):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
484 translator_comments.append((token.lineno, value.strip()))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
485 break
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
486
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
487 elif token.type == 'multilinecomment':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
488 # only one multi-line comment may preceed a translation
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
489 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
490 value = token.value[2:-2].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
491 for comment_tag in comment_tags:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
492 if value.startswith(comment_tag):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
493 lines = value.splitlines()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
494 if lines:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
495 lines[0] = lines[0].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
496 lines[1:] = dedent('\n'.join(lines[1:])).splitlines()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
497 for offset, line in enumerate(lines):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
498 translator_comments.append((token.lineno + offset,
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
499 line))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
500 break
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
501
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
502 elif funcname and call_stack == 0:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
503 if token.type == 'operator' and token.value == ')':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
504 if last_argument is not None:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
505 messages.append(last_argument)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
506 if len(messages) > 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
507 messages = tuple(messages)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
508 elif messages:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
509 messages = messages[0]
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
510 else:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
511 messages = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
512
426
palgarvio
parents: 414
diff changeset
513 # Comments don't apply unless they immediately precede the
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
514 # message
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
515 if translator_comments and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
516 translator_comments[-1][0] < message_lineno - 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
517 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
518
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
519 if messages is not None:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
520 yield (message_lineno, funcname, messages,
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
521 [comment[1] for comment in translator_comments])
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
522
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
523 funcname = message_lineno = last_argument = None
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
524 concatenate_next = False
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
525 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
526 messages = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
527 call_stack = -1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
528
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
529 elif token.type == 'string':
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
530 new_value = unquote_string(token.value)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
531 if concatenate_next:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
532 last_argument = (last_argument or '') + new_value
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
533 concatenate_next = False
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
534 else:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
535 last_argument = new_value
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
536
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
537 elif token.type == 'operator':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
538 if token.value == ',':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
539 if last_argument is not None:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
540 messages.append(last_argument)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
541 last_argument = None
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
542 else:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
543 messages.append(None)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
544 concatenate_next = False
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
545 elif token.value == '+':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
546 concatenate_next = True
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
547
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
548 elif call_stack > 0 and token.type == 'operator' \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
549 and token.value == ')':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
550 call_stack -= 1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
551
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
552 elif funcname and call_stack == -1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
553 funcname = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
554
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
555 elif call_stack == -1 and token.type == 'name' and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
556 token.value in keywords and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
557 (last_token is None or last_token.type != 'name' or
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
558 last_token.value != 'function'):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
559 funcname = token.value
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
560
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
561 last_token = token
Copyright (C) 2012-2017 Edgewall Software