Mercurial > babel > mirror

annotate babel/messages/extract.py @ 560:86524be05b60 trunk

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix docstring for babel.messages.extract() so it mentions the correct return type
author fschwarz
date Sat, 03 Sep 2011 08:31:43 +0000
parents ca203b2af83c
children 1b801a0cb2cb
rev   line source
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
1 # -*- coding: utf-8 -*-
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
2 #
530
ca203b2af83c Update the copyright line.
jruigrok
parents: 525
diff changeset
3 # Copyright (C) 2007-2011 Edgewall Software
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
4 # All rights reserved.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
5 #
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
6 # This software is licensed as described in the file COPYING, which
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
7 # you should have received as part of this distribution. The terms
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
8 # are also available at http://babel.edgewall.org/wiki/License.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
9 #
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
10 # This software consists of voluntary contributions made by many
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
11 # individuals. For the exact contribution history, see the revision
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
12 # history and logs, available at http://babel.edgewall.org/log/.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
13
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
14 """Basic infrastructure for extracting localizable messages from source files.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
15
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
16 This module defines an extensible system for collecting localizable message
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
17 strings from a variety of sources. A native extractor for Python source files
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
18 is builtin, extractors for other sources can be added using very simple plugins.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
19
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
20 The main entry points into the extraction functionality are the functions
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
21 `extract_from_dir` and `extract_from_file`.
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
22 """
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
23
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
24 import os
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
25 import sys
162
32be08ab2440 alphabetize imports
pjenvey
parents: 154
diff changeset
26 from tokenize import generate_tokens, COMMENT, NAME, OP, STRING
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
27
525
2baa2cedd6f9 Cleanup round #1: get rid of the frozenset/set utility code and imports.
jruigrok
parents: 426
diff changeset
28 from babel.util import parse_encoding, pathmatch, relpath
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
29 from textwrap import dedent
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
30
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
31 __all__ = ['extract', 'extract_from_dir', 'extract_from_file']
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
32 __docformat__ = 'restructuredtext en'
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
33
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
34 GROUP_NAME = 'babel.extractors'
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
35
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
36 DEFAULT_KEYWORDS = {
10
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
37 '_': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
38 'gettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
39 'ngettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
40 'ugettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
41 'ungettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
42 'dgettext': (2,),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
43 'dngettext': (2, 3),
179
31beb381d62f added 'N_' (gettext noop) to the extractor's default keywords
pjenvey
parents: 164
diff changeset
44 'N_': None
10
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings.
palgarvio
parents: 1
diff changeset
45 }
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
46
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
47 DEFAULT_MAPPING = [('**.py', 'python')]
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
48
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
49 empty_msgid_warning = (
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
50 '%s: warning: Empty msgid. It is reserved by GNU gettext: gettext("") '
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
51 'returns the header entry with meta information, not the empty string.')
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
52
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
53
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
54 def _strip_comment_tags(comments, tags):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
55 """Helper function for `extract` that strips comment tags from strings
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
56 in a list of comment lines. This functions operates in-place.
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
57 """
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
58 def _strip(line):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
59 for tag in tags:
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
60 if line.startswith(tag):
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
61 return line[len(tag):].strip()
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
62 return line
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
63 comments[:] = map(_strip, comments)
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
64
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
65
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
66 def extract_from_dir(dirname=os.getcwd(), method_map=DEFAULT_MAPPING,
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
67 options_map=None, keywords=DEFAULT_KEYWORDS,
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
68 comment_tags=(), callback=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
69 """Extract messages from any source files found in the given directory.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
70
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
71 This function generates tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
72
82
540bb484f6e0 Missed some param's documentation regarding translator comments.
palgarvio
parents: 81
diff changeset
73 ``(filename, lineno, message, comments)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
74
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
75 Which extraction method is used per file is determined by the `method_map`
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
76 parameter, which maps extended glob patterns to extraction method names.
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
77 For example, the following is the default mapping:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
78
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
79 >>> method_map = [
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
80 ... ('**.py', 'python')
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
81 ... ]
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
82
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
83 This basically says that files with the filename extension ".py" at any
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
84 level inside the directory should be processed by the "python" extraction
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
85 method. Files that don't match any of the mapping patterns are ignored. See
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
86 the documentation of the `pathmatch` function for details on the pattern
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
87 syntax.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
88
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
89 The following extended mapping would also use the "genshi" extraction
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
90 method on any file in "templates" subdirectory:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
91
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
92 >>> method_map = [
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
93 ... ('**/templates/**.*', 'genshi'),
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
94 ... ('**.py', 'python')
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
95 ... ]
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
96
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
97 The dictionary provided by the optional `options_map` parameter augments
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
98 these mappings. It uses extended glob patterns as keys, and the values are
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
99 dictionaries mapping options names to option values (both strings).
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
100
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
101 The glob patterns of the `options_map` do not necessarily need to be the
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
102 same as those used in the method mapping. For example, while all files in
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
103 the ``templates`` folders in an application may be Genshi applications, the
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
104 options for those files may differ based on extension:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
105
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
106 >>> options_map = {
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
107 ... '**/templates/**.txt': {
144
14fe2a8fb842 Some doc fixes.
cmlenz
parents: 138
diff changeset
108 ... 'template_class': 'genshi.template:TextTemplate',
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
109 ... 'encoding': 'latin-1'
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
110 ... },
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
111 ... '**/templates/**.html': {
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
112 ... 'include_attrs': ''
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
113 ... }
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
114 ... }
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
115
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
116 :param dirname: the path to the directory to extract messages from
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
117 :param method_map: a list of ``(pattern, method)`` tuples that maps of
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
118 extraction method names to extended glob patterns
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
119 :param options_map: a dictionary of additional options (optional)
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
120 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
121 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
122 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
123 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
124 :param comment_tags: a list of tags of translator comments to search for
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
125 and include in the results
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
126 :param callback: a function that is called for every file that message are
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
127 extracted from, just before the extraction itself is
75
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
128 performed; the function is passed the filename, the name
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
129 of the extraction method and and the options dictionary as
0f74337264ce Fixed MIME type of new doc page.
cmlenz
parents: 62
diff changeset
130 positional arguments, in that order
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
131 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
132 tags to be removed from the collected comments.
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
133 :return: an iterator over ``(filename, lineno, funcname, message)`` tuples
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
134 :rtype: ``iterator``
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
135 :see: `pathmatch`
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
136 """
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
137 if options_map is None:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
138 options_map = {}
56
f40fc143439c Add actual data structures for handling message catalogs, so that more code can be reused here between the frontends.
cmlenz
parents: 54
diff changeset
139
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
140 absname = os.path.abspath(dirname)
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
141 for root, dirnames, filenames in os.walk(absname):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
142 for subdir in dirnames:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
143 if subdir.startswith('.') or subdir.startswith('_'):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
144 dirnames.remove(subdir)
154
31478eb3fb9e The default ordering of messages in generated POT files, which is based on the order those messages are found when walking the source tree, is no longer subject to differences between platforms; directory and file names are now always sorted alphabetically.
cmlenz
parents: 147
diff changeset
145 dirnames.sort()
31478eb3fb9e The default ordering of messages in generated POT files, which is based on the order those messages are found when walking the source tree, is no longer subject to differences between platforms; directory and file names are now always sorted alphabetically.
cmlenz
parents: 147
diff changeset
146 filenames.sort()
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
147 for filename in filenames:
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
148 filename = relpath(
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
149 os.path.join(root, filename).replace(os.sep, '/'),
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
150 dirname
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
151 )
62
2df27f49c320 The order of extraction methods is now preserved (see #10).
cmlenz
parents: 57
diff changeset
152 for pattern, method in method_map:
44
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
153 if pathmatch(pattern, filename):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
154 filepath = os.path.join(absname, filename)
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
155 options = {}
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
156 for opattern, odict in options_map.items():
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
157 if pathmatch(opattern, filename):
a524b547ea7e Some work towards #4.
cmlenz
parents: 36
diff changeset
158 options = odict
47
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay.
cmlenz
parents: 44
diff changeset
159 if callback:
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
160 callback(filename, method, options)
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
161 for lineno, message, comments in \
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
162 extract_from_file(method, filepath,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
163 keywords=keywords,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
164 comment_tags=comment_tags,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
165 options=options,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
166 strip_comment_tags=
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
167 strip_comment_tags):
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
168 yield filename, lineno, message, comments
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
169 break
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
170
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
171
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
172 def extract_from_file(method, filename, keywords=DEFAULT_KEYWORDS,
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
173 comment_tags=(), options=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
174 """Extract messages from a specific file.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
175
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
176 This function returns a list of tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
177
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
178 ``(lineno, funcname, message)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
179
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
180 :param filename: the path to the file to extract messages from
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
181 :param method: a string specifying the extraction method (.e.g. "python")
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
182 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
183 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
184 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
185 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
186 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
187 in the results
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
188 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
189 tags to be removed from the collected comments.
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
190 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
191 :return: the list of extracted messages
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
192 :rtype: `list`
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
193 """
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
194 fileobj = open(filename, 'U')
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
195 try:
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
196 return list(extract(method, fileobj, keywords, comment_tags, options,
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
197 strip_comment_tags))
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
198 finally:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
199 fileobj.close()
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
200
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
201
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
202 def extract(method, fileobj, keywords=DEFAULT_KEYWORDS, comment_tags=(),
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
203 options=None, strip_comment_tags=False):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
204 """Extract messages from the given file-like object using the specified
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
205 extraction method.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
206
560
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
207 This function returns tuples of the form:
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
208
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
209 ``(lineno, message, comments)``
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
210
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
211 The implementation dispatches the actual extraction to plugins, based on the
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
212 value of the ``method`` parameter.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
213
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
214 >>> source = '''# foo module
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
215 ... def run(argv):
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
216 ... print _('Hello, world!')
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
217 ... '''
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
218
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
219 >>> from StringIO import StringIO
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
220 >>> for message in extract('python', StringIO(source)):
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
221 ... print message
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
222 (3, u'Hello, world!', [])
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
223
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
224 :param method: a string specifying the extraction method (.e.g. "python");
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
225 if this is a simple name, the extraction function will be
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
226 looked up by entry point; if it is an explicit reference
329
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
227 to a function (of the form ``package.module:funcname`` or
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
228 ``package.module.funcname``), the corresponding function
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
229 will be imported and used
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
230 :param fileobj: the file-like object the messages should be extracted from
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
231 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
232 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
233 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
234 localizable strings
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
235 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
236 in the results
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
237 :param options: a dictionary of additional options (optional)
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
238 :param strip_comment_tags: a flag that if set to `True` causes all comment
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
239 tags to be removed from the collected comments.
560
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
240 :return: an iterator over ``(lineno, message, comments)`` tuples
86524be05b60 fix docstring for babel.messages.extract() so it mentions the correct return type
fschwarz
parents: 530
diff changeset
241 :rtype: `iterator`
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
242 :raise ValueError: if the extraction method is not registered
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
243 """
322
b03a6a87e4cb fix invalid message extraction methods causing:
pjenvey
parents: 258
diff changeset
244 func = None
329
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
245 if ':' in method or '.' in method:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
246 if ':' not in method:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
247 lastdot = method.rfind('.')
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
248 module, attrname = method[:lastdot], method[lastdot + 1:]
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
249 else:
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
250 module, attrname = method.split(':', 1)
35c19c01e4b5 Allow extraction method specification to use a dot instead of the colon for separating module and function names. See #105.
cmlenz
parents: 322
diff changeset
251 func = getattr(__import__(module, {}, {}, [attrname]), attrname)
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
252 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
253 try:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
254 from pkg_resources import working_set
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
255 except ImportError:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
256 # pkg_resources is not available, so we resort to looking up the
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
257 # builtin extractors directly
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
258 builtin = {'ignore': extract_nothing, 'python': extract_python}
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
259 func = builtin.get(method)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
260 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
261 for entry_point in working_set.iter_entry_points(GROUP_NAME,
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
262 method):
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
263 func = entry_point.load(require=True)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
264 break
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
265 if func is None:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
266 raise ValueError('Unknown extraction method %r' % method)
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
267
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
268 results = func(fileobj, keywords.keys(), comment_tags,
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
269 options=options or {})
366
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
270
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
271 for lineno, funcname, messages, comments in results:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
272 if funcname:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
273 spec = keywords[funcname] or (1,)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
274 else:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
275 spec = (1,)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
276 if not isinstance(messages, (list, tuple)):
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
277 messages = [messages]
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
278 if not messages:
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
279 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
280
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
281 # Validate the messages against the keyword's specification
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
282 msgs = []
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
283 invalid = False
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
284 # last_index is 1 based like the keyword spec
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
285 last_index = len(messages)
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
286 for index in spec:
258
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
287 if last_index < index:
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
288 # Not enough arguments
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
289 invalid = True
5ca5fbd47766 skip messages that have less arguments than the keyword spec calls for
pjenvey
parents: 250
diff changeset
290 break
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
291 message = messages[index - 1]
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
292 if message is None:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
293 invalid = True
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
294 break
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
295 msgs.append(message)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
296 if invalid:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
297 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
298
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
299 first_msg_index = spec[0] - 1
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
300 if not messages[first_msg_index]:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
301 # An empty string msgid isn't valid, emit a warning
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
302 where = '%s:%i' % (hasattr(fileobj, 'name') and \
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
303 fileobj.name or '(unknown)', lineno)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
304 print >> sys.stderr, empty_msgid_warning % where
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
305 continue
12
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places.
cmlenz
parents: 10
diff changeset
306
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
307 messages = tuple(msgs)
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
308 if len(messages) == 1:
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
309 messages = messages[0]
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
310
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
311 if strip_comment_tags:
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
312 _strip_comment_tags(comments, comment_tags)
250
6c06570af1b9 Soften dependency on setuptools. Extraction methods can now be referenced using a special section in the mapping configuration, mapping short names to fully-qualified function references.
cmlenz
parents: 224
diff changeset
313 yield lineno, messages, comments
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
314
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
315
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
316 def extract_nothing(fileobj, keywords, comment_tags, options):
57
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
317 """Pseudo extractor that does not actually extract anything, but simply
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
318 returns an empty list.
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
319 """
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
320 return []
d930a3dfbf3d * The `extract_messages` distutils command now operators on configurable input directories again, instead of the complete current directory. The `input_dirs` default to the package directories.
cmlenz
parents: 56
diff changeset
321
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
322
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
323 def extract_python(fileobj, keywords, comment_tags, options):
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
324 """Extract messages from Python source code.
224
0a71b675fc48 Fix for message extractors which return `None` as the gettext call.
palgarvio
parents: 223
diff changeset
325
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
326 :param fileobj: the seekable, file-like object the messages should be
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
327 extracted from
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
328 :param keywords: a list of keywords (i.e. function names) that should be
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
329 recognized as translation functions
84
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
330 :param comment_tags: a list of translator tags to search for and include
3ae316b58231 Some cosmetic changes for the new translator comments support.
cmlenz
parents: 82
diff changeset
331 in the results
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
332 :param options: a dictionary of additional options (optional)
81
85af04c72ccd Fixed and added some documentation about the translator comments implemented in [81].
palgarvio
parents: 80
diff changeset
333 :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
334 :rtype: ``iterator``
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
335 """
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
336 funcname = lineno = message_lineno = None
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
337 call_stack = -1
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
338 buf = []
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
339 messages = []
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
340 translator_comments = []
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
341 in_def = in_translator_comments = False
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
342 comment_tag = None
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
343
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
344 encoding = parse_encoding(fileobj) or options.get('encoding', 'iso-8859-1')
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
345
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
346 tokens = generate_tokens(fileobj.readline)
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
347 for tok, value, (lineno, _), _, _ in tokens:
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
348 if call_stack == -1 and tok == NAME and value in ('def', 'class'):
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
349 in_def = True
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
350 elif tok == OP and value == '(':
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
351 if in_def:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
352 # Avoid false positives for declarations such as:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
353 # def gettext(arg='message'):
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
354 in_def = False
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
355 continue
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
356 if funcname:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
357 message_lineno = lineno
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
358 call_stack += 1
223
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
359 elif in_def and tok == OP and value == ':':
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
360 # End of a class definition without parens
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
361 in_def = False
3784eb8493da fix skipping of class definitions without parens
pjenvey
parents: 222
diff changeset
362 continue
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
363 elif call_stack == -1 and tok == COMMENT:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
364 # Strip the comment token from the line
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
365 value = value.decode(encoding)[1:].strip()
147
63a93d33511a simplify
pjenvey
parents: 144
diff changeset
366 if in_translator_comments and \
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
367 translator_comments[-1][0] == lineno - 1:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
368 # We're already inside a translator comment, continue appending
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
369 translator_comments.append((lineno, value))
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
370 continue
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
371 # If execution reaches this point, let's see if comment line
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
372 # starts with one of the comment tags
85
04a2f16bdd04 Fixed de-pluralization bug introduced in [85] regarding the extraction of translator comments.
palgarvio
parents: 84
diff changeset
373 for comment_tag in comment_tags:
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
374 if value.startswith(comment_tag):
147
63a93d33511a simplify
pjenvey
parents: 144
diff changeset
375 in_translator_comments = True
338
b39145076d8a Stripping of comment tags is optional now. If enabled it will strip the tags from all lines of a comment now.
aronacher
parents: 329
diff changeset
376 translator_comments.append((lineno, value))
92
ccb9da614597 Fixed bug introduced in [92], bad use of `lstrip()`. Added a unittest to test multiple translator comment tags.
palgarvio
parents: 91
diff changeset
377 break
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
378 elif funcname and call_stack == 0:
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
379 if tok == OP and value == ')':
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
380 if buf:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
381 messages.append(''.join(buf))
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
382 del buf[:]
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
383 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
384 messages.append(None)
93
f008662b5d6e Commiting patch provided by pjenvey: Translator comments don't apply unless they immediately preceed the message.
palgarvio
parents: 92
diff changeset
385
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
386 if len(messages) > 1:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
387 messages = tuple(messages)
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
388 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
389 messages = messages[0]
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
390 # Comments don't apply unless they immediately preceed the
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
391 # message
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
392 if translator_comments and \
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
393 translator_comments[-1][0] < message_lineno - 1:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
394 translator_comments = []
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
395
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
396 yield (message_lineno, funcname, messages,
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
397 [comment[1] for comment in translator_comments])
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
398
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
399 funcname = lineno = message_lineno = None
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
400 call_stack = -1
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
401 messages = []
80
116e34b8cefa Added support for translator comments at the API and frontends levels.(See #12, item 1). Updated docs and tests accordingly.
palgarvio
parents: 75
diff changeset
402 translator_comments = []
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
403 in_translator_comments = False
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
404 elif tok == STRING:
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
405 # Unwrap quotes in a safe manner, maintaining the string's
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
406 # encoding
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
407 # https://sourceforge.net/tracker/?func=detail&atid=355470&
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
408 # aid=617979&group_id=5470
164
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
409 value = eval('# coding=%s\n%s' % (encoding, value),
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
410 {'__builtins__':{}}, {})
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
411 if isinstance(value, str):
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
412 value = value.decode(encoding)
e1199c0fb3bf made the python extractor detect source file encodings from the magic encoding
pjenvey
parents: 162
diff changeset
413 buf.append(value)
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
414 elif tok == OP and value == ',':
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
415 if buf:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
416 messages.append(''.join(buf))
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
417 del buf[:]
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
418 else:
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
419 messages.append(None)
366
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
420 if translator_comments:
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
421 # We have translator comments, and since we're on a
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
422 # comma(,) user is allowed to break into a new line
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
423 # Let's increase the last comment's lineno in order
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
424 # for the comment to still be a valid one
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
425 old_lineno, old_comment = translator_comments.pop()
6abe384584c8 Test and respective fix for gettext calls that spawn multiple lines. Fixes #119.
palgarvio
parents: 343
diff changeset
426 translator_comments.append((old_lineno+1, old_comment))
222
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
427 elif call_stack > 0 and tok == OP and value == ')':
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
428 call_stack -= 1
88caccd5da79 o extract_python fixes:
pjenvey
parents: 214
diff changeset
429 elif funcname and call_stack == -1:
1
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
430 funcname = None
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
431 elif tok == NAME and value in keywords:
7870274479f5 Import of initial code base.
cmlenz
parents:
diff changeset
432 funcname = value
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
433
340
ce83b4f77114 added some newlines to extract and jslexer to stay consistent with the rest of the sourcecode.
aronacher
parents: 339
diff changeset
434
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
435 def extract_javascript(fileobj, keywords, comment_tags, options):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
436 """Extract messages from JavaScript source code.
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
437
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
438 :param fileobj: the seekable, file-like object the messages should be
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
439 extracted from
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
440 :param keywords: a list of keywords (i.e. function names) that should be
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
441 recognized as translation functions
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
442 :param comment_tags: a list of translator tags to search for and include
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
443 in the results
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
444 :param options: a dictionary of additional options (optional)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
445 :return: an iterator over ``(lineno, funcname, message, comments)`` tuples
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
446 :rtype: ``iterator``
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
447 """
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
448 from babel.messages.jslexer import tokenize, unquote_string
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
449 funcname = message_lineno = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
450 messages = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
451 last_argument = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
452 translator_comments = []
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
453 concatenate_next = False
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
454 encoding = options.get('encoding', 'utf-8')
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
455 last_token = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
456 call_stack = -1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
457
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
458 for token in tokenize(fileobj.read().decode(encoding)):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
459 if token.type == 'operator' and token.value == '(':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
460 if funcname:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
461 message_lineno = token.lineno
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
462 call_stack += 1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
463
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
464 elif call_stack == -1 and token.type == 'linecomment':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
465 value = token.value[2:].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
466 if translator_comments and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
467 translator_comments[-1][0] == token.lineno - 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
468 translator_comments.append((token.lineno, value))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
469 continue
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
470
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
471 for comment_tag in comment_tags:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
472 if value.startswith(comment_tag):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
473 translator_comments.append((token.lineno, value.strip()))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
474 break
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
475
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
476 elif token.type == 'multilinecomment':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
477 # only one multi-line comment may preceed a translation
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
478 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
479 value = token.value[2:-2].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
480 for comment_tag in comment_tags:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
481 if value.startswith(comment_tag):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
482 lines = value.splitlines()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
483 if lines:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
484 lines[0] = lines[0].strip()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
485 lines[1:] = dedent('\n'.join(lines[1:])).splitlines()
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
486 for offset, line in enumerate(lines):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
487 translator_comments.append((token.lineno + offset,
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
488 line))
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
489 break
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
490
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
491 elif funcname and call_stack == 0:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
492 if token.type == 'operator' and token.value == ')':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
493 if last_argument is not None:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
494 messages.append(last_argument)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
495 if len(messages) > 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
496 messages = tuple(messages)
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
497 elif messages:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
498 messages = messages[0]
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
499 else:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
500 messages = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
501
426
palgarvio
parents: 414
diff changeset
502 # Comments don't apply unless they immediately precede the
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
503 # message
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
504 if translator_comments and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
505 translator_comments[-1][0] < message_lineno - 1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
506 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
507
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
508 if messages is not None:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
509 yield (message_lineno, funcname, messages,
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
510 [comment[1] for comment in translator_comments])
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
511
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
512 funcname = message_lineno = last_argument = None
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
513 concatenate_next = False
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
514 translator_comments = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
515 messages = []
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
516 call_stack = -1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
517
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
518 elif token.type == 'string':
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
519 new_value = unquote_string(token.value)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
520 if concatenate_next:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
521 last_argument = (last_argument or '') + new_value
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
522 concatenate_next = False
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
523 else:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
524 last_argument = new_value
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
525
405
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
526 elif token.type == 'operator':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
527 if token.value == ',':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
528 if last_argument is not None:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
529 messages.append(last_argument)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
530 last_argument = None
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
531 else:
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
532 messages.append(None)
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
533 concatenate_next = False
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
534 elif token.value == '+':
0686aa8e7e36 Added support for string concatenation to javascript lexer. _("foo" + "bar") is now equivalent to _("foobar")
aronacher
parents: 366
diff changeset
535 concatenate_next = True
339
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
536
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
537 elif call_stack > 0 and token.type == 'operator' \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
538 and token.value == ')':
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
539 call_stack -= 1
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
540
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
541 elif funcname and call_stack == -1:
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
542 funcname = None
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
543
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
544 elif call_stack == -1 and token.type == 'name' and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
545 token.value in keywords and \
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
546 (last_token is None or last_token.type != 'name' or
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
547 last_token.value != 'function'):
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
548 funcname = token.value
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
549
93a896111488 Added !JavaScript extractor
aronacher
parents: 338
diff changeset
550 last_token = token
Copyright (C) 2012-2017 Edgewall Software