babel/mirror: babel/messages/extract.py annotate

annotate babel/messages/extract.py @ 54:7dbcbc3f07e0 trunk

Rename the `babel.catalog` package to `babel.messages` for consistency with the other package names.

author	cmlenz
date	Fri, 08 Jun 2007 09:16:32 +0000
parents	babel/catalog/extract.py@f8469ab4b257
children	f40fc143439c

rev	line source
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	1 # -- coding: utf-8 --
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	2 #
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	3 # Copyright (C) 2007 Edgewall Software
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	4 # All rights reserved.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	5 #
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	6 # This software is licensed as described in the file COPYING, which
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	7 # you should have received as part of this distribution. The terms
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	8 # are also available at http://babel.edgewall.org/wiki/License.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	9 #
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	10 # This software consists of voluntary contributions made by many
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	11 # individuals. For the exact contribution history, see the revision
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	12 # history and logs, available at http://babel.edgewall.org/log/.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	13
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	14 """Basic infrastructure for extracting localizable messages from source files.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	15
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	16 This module defines an extensible system for collecting localizable message
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	17 strings from a variety of sources. A native extractor for Python source files
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	18 is builtin, extractors for other sources can be added using very simple plugins.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	19
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	20 The main entry points into the extraction functionality are the functions
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	21 `extract_from_dir` and `extract_from_file`.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	22 """
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	23
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	24 import os
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	25 try:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	26 set
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	27 except NameError:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	28 from sets import Set as set
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	29 import sys
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	30 from tokenize import generate_tokens, NAME, OP, STRING
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	31
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	32 from babel.util import pathmatch, relpath
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	33
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	34 __all__ = ['extract', 'extract_from_dir', 'extract_from_file']
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	35 __docformat__ = 'restructuredtext en'
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	36
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	37 GROUP_NAME = 'babel.extractors'
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	38
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	39 DEFAULT_KEYWORDS = {
10 4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	40 '_': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	41 'gettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	42 'ngettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	43 'ugettext': None,
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	44 'ungettext': (1, 2),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	45 'dgettext': (2,),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	46 'dngettext': (2, 3),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	47 }
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	48
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	49 DEFAULT_MAPPING = {
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	50 '**.html': 'genshi',
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	51 '**.py': 'python'
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	52 }
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	53
47 f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	54 def extract_from_dir(dirname=os.getcwd(), method_map=DEFAULT_MAPPING,
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	55 options_map=None, keywords=DEFAULT_KEYWORDS,
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	56 callback=None):
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	57 """Extract messages from any source files found in the given directory.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	58
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	59 This function generates tuples of the form:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	60
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	61 ``(filename, lineno, funcname, message)``
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	62
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	63 Which extraction method is used per file is determined by the `method_map`
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	64 parameter, which maps extended glob patterns to extraction method names.
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	65 For example, the following is the default mapping:
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	66
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	67 >>> method_map = {
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	68 ... '**.py': 'python'
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	69 ... }
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	70
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	71 This basically says that files with the filename extension ".py" at any
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	72 level inside the directory should be processed by the "python" extraction
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	73 method. Files that don't match any of the mapping patterns are ignored. See
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	74 the documentation of the `pathmatch` function for details on the pattern
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	75 syntax.
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	76
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	77 The following extended mapping would also use the "genshi" extraction method
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	78 on any file in "templates" subdirectory:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	79
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	80 >>> method_map = {
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	81 ... '/templates/.*': 'genshi',
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	82 ... '**.py': 'python'
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	83 ... }
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	84
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	85 The dictionary provided by the optional `options_map` parameter augments
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	86 the mapping data. It too uses extended glob patterns as keys, but the values
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	87 are dictionaries mapping options names to option values (both strings).
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	88
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	89 The glob patterns of the `options_map` do not necessarily need to be the
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	90 same as those used in the pattern. For example, while all files in the
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	91 ``templates`` folders in an application may be Genshi applications, the
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	92 options for those files may differ based on extension:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	93
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	94 >>> options_map = {
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	95 ... '/templates/.txt': {
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	96 ... 'template_class': 'genshi.template.text.TextTemplate',
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	97 ... 'encoding': 'latin-1'
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	98 ... },
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	99 ... '/templates/.html': {
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	100 ... 'include_attrs': ''
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	101 ... }
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	102 ... }
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	103
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	104 :param dirname: the path to the directory to extract messages from
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	105 :param method_map: a mapping of extraction method names to extended glob
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	106 patterns
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	107 :param options_map: a dictionary of additional options (optional)
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	108 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	109 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	110 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	111 localizable strings
47 f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	112 :param callback: a function that is called for every file that message are
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	113 extracted from, just before the extraction itself is
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	114 performed; the function is passed the filename and the
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	115 options dictionary as positional arguments, in that order
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	116 :return: an iterator over ``(filename, lineno, funcname, message)`` tuples
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	117 :rtype: ``iterator``
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	118 :see: `pathmatch`
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	119 """
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	120 if options_map is None:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	121 options_map = {}
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	122 absname = os.path.abspath(dirname)
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	123 for root, dirnames, filenames in os.walk(absname):
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	124 for subdir in dirnames:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	125 if subdir.startswith('.') or subdir.startswith('_'):
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	126 dirnames.remove(subdir)
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	127 for filename in filenames:
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	128 filename = relpath(
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	129 os.path.join(root, filename).replace(os.sep, '/'),
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	130 dirname
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	131 )
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	132 for pattern, method in method_map.items():
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	133 if pathmatch(pattern, filename):
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	134 filepath = os.path.join(absname, filename)
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	135 options = {}
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	136 for opattern, odict in options_map.items():
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	137 if pathmatch(opattern, filename):
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	138 options = odict
47 f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	139 if callback:
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	140 callback(filename, options)
44 a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	141 for line, func, key in extract_from_file(method, filepath,
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	142 keywords=keywords,
a524b547ea7e Some work towards #4. cmlenz parents: 36 diff changeset	143 options=options):
47 f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	144 yield filename, line, func, key
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	145
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	146 def extract_from_file(method, filename, keywords=DEFAULT_KEYWORDS,
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	147 options=None):
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	148 """Extract messages from a specific file.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	149
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	150 This function returns a list of tuples of the form:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	151
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	152 ``(lineno, funcname, message)``
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	153
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	154 :param filename: the path to the file to extract messages from
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	155 :param method: a string specifying the extraction method (.e.g. "python")
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	156 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	157 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	158 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	159 localizable strings
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	160 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	161 :return: the list of extracted messages
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	162 :rtype: `list`
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	163 """
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	164 fileobj = open(filename, 'U')
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	165 try:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	166 return list(extract(method, fileobj, keywords, options=options))
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	167 finally:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	168 fileobj.close()
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	169
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	170 def extract(method, fileobj, keywords=DEFAULT_KEYWORDS, options=None):
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	171 """Extract messages from the given file-like object using the specified
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	172 extraction method.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	173
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	174 This function returns a list of tuples of the form:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	175
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	176 ``(lineno, funcname, message)``
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	177
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	178 The implementation dispatches the actual extraction to plugins, based on the
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	179 value of the ``method`` parameter.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	180
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	181 >>> source = '''# foo module
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	182 ... def run(argv):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	183 ... print _('Hello, world!')
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	184 ... '''
10 4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	185
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	186 >>> from StringIO import StringIO
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	187 >>> for message in extract('python', StringIO(source)):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	188 ... print message
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	189 (3, '_', 'Hello, world!')
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	190
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	191 :param method: a string specifying the extraction method (.e.g. "python")
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	192 :param fileobj: the file-like object the messages should be extracted from
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	193 :param keywords: a dictionary mapping keywords (i.e. names of functions
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	194 that should be recognized as translation functions) to
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	195 tuples that specify which of their arguments contain
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	196 localizable strings
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	197 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	198 :return: the list of extracted messages
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	199 :rtype: `list`
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	200 :raise ValueError: if the extraction method is not registered
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	201 """
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	202 from pkg_resources import working_set
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	203
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	204 for entry_point in working_set.iter_entry_points(GROUP_NAME, method):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	205 func = entry_point.load(require=True)
10 4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	206 m = []
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	207 for lineno, funcname, messages in func(fileobj, keywords.keys(),
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	208 options=options or {}):
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	209 if isinstance(messages, (list, tuple)):
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	210 msgs = []
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	211 for index in keywords[funcname]:
e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	212 msgs.append(messages[index - 1])
10 4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	213 messages = tuple(msgs)
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	214 if len(messages) == 1:
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	215 messages = messages[0]
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	216 yield lineno, funcname, messages
4130d9c6cb34 Both Babel's [source:trunk/babel/catalog/frontend.py frontend] and [source:trunk/babel/catalog/extract.py extract] now handle keyword indices. Also added an extra boolean flag so that the default keywords defined by Babel are not included in the keywords to search for when extracting strings. palgarvio parents: 1 diff changeset	217 return
12 e6ba3e878b10 * Removed pkg_resources/setuptools requirement from various places. cmlenz parents: 10 diff changeset	218
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	219 raise ValueError('Unknown extraction method %r' % method)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	220
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	221 def extract_genshi(fileobj, keywords, options):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	222 """Extract messages from Genshi templates.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	223
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	224 :param fileobj: the file-like object the messages should be extracted from
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	225 :param keywords: a list of keywords (i.e. function names) that should be
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	226 recognized as translation functions
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	227 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	228 :return: an iterator over ``(lineno, funcname, message)`` tuples
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	229 :rtype: ``iterator``
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	230 """
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	231 from genshi.filters.i18n import Translator
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	232 from genshi.template import MarkupTemplate
47 f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	233
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	234 template_class = options.get('template_class', MarkupTemplate)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	235 if isinstance(template_class, basestring):
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	236 module, clsname = template_class.split(':', 1)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	237 template_class = getattr(__import__(module, {}, {}, [clsname]), clsname)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	238 encoding = options.get('encoding', None)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	239
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	240 ignore_tags = options.get('ignore_tags', Translator.IGNORE_TAGS)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	241 if isinstance(ignore_tags, basestring):
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	242 ignore_tags = ignore_tags.split()
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	243 include_attrs = options.get('include_attrs', Translator.INCLUDE_ATTRS)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	244 if isinstance(include_attrs, basestring):
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	245 include_attrs = include_attrs.split()
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	246
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	247 tmpl = template_class(fileobj, filename=getattr(fileobj, 'name'),
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	248 encoding=encoding)
f8469ab4b257 Support passing extraction method mapping and options from the frontends (see #4). No distutils/setuptools keyword supported yet, but the rest seems to be working okay. cmlenz parents: 44 diff changeset	249 translator = Translator(None, ignore_tags, include_attrs)
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	250 for message in translator.extract(tmpl.stream, gettext_functions=keywords):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	251 yield message
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	252
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	253 def extract_python(fileobj, keywords, options):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	254 """Extract messages from Python source code.
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	255
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	256 :param fileobj: the file-like object the messages should be extracted from
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	257 :param keywords: a list of keywords (i.e. function names) that should be
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	258 recognized as translation functions
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	259 :param options: a dictionary of additional options (optional)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	260 :return: an iterator over ``(lineno, funcname, message)`` tuples
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	261 :rtype: ``iterator``
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	262 """
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	263 funcname = None
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	264 lineno = None
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	265 buf = []
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	266 messages = []
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	267 in_args = False
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	268
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	269 tokens = generate_tokens(fileobj.readline)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	270 for tok, value, (lineno, _), _, _ in tokens:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	271 if funcname and tok == OP and value == '(':
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	272 in_args = True
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	273 elif funcname and in_args:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	274 if tok == OP and value == ')':
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	275 in_args = False
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	276 if buf:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	277 messages.append(''.join(buf))
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	278 del buf[:]
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	279 if filter(None, messages):
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	280 if len(messages) > 1:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	281 messages = tuple(messages)
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	282 else:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	283 messages = messages[0]
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	284 yield lineno, funcname, messages
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	285 funcname = lineno = None
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	286 messages = []
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	287 elif tok == STRING:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	288 if lineno is None:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	289 lineno = stup[0]
36 cfd15b7921f9 Fix for #8: fix extraction of strings from Python source using prefixes ('u' or 'r') or triple quotes. cmlenz parents: 12 diff changeset	290 # Unwrap quotes in a safe manner
cfd15b7921f9 Fix for #8: fix extraction of strings from Python source using prefixes ('u' or 'r') or triple quotes. cmlenz parents: 12 diff changeset	291 buf.append(eval(value, {'__builtins__':{}}, {}))
1 7870274479f5 Import of initial code base. cmlenz parents: diff changeset	292 elif tok == OP and value == ',':
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	293 messages.append(''.join(buf))
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	294 del buf[:]
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	295 elif funcname:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	296 funcname = None
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	297 elif tok == NAME and value in keywords:
7870274479f5 Import of initial code base. cmlenz parents: diff changeset	298 funcname = value

Mercurial > babel > mirror

annotate babel/messages/extract.py @ 54:7dbcbc3f07e0 trunk