Mercurial > bitten > bitten-test
annotate bitten/build/pythontools.py @ 915:e36f9b446976
Wrap lint report messages in a msg tag (includes a test and a documentation update). Fixes #547.
author | hodgestar |
---|---|
date | Mon, 23 May 2011 09:28:19 +0000 |
parents | 7c80375d4817 |
children |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
5 | 2 # |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
404
diff
changeset
|
3 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
482 | 4 # Copyright (C) 2008 Matt Good <matt@matt-good.net> |
832 | 5 # Copyright (C) 2008-2010 Edgewall Software |
163 | 6 # All rights reserved. |
5 | 7 # |
163 | 8 # This software is licensed as described in the file COPYING, which |
9 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
404
diff
changeset
|
10 # are also available at http://bitten.edgewall.org/wiki/License. |
5 | 11 |
313 | 12 """Recipe commands for tools commonly used by Python projects.""" |
13 | |
482 | 14 from __future__ import division |
15 | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
16 import logging |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
17 import os |
482 | 18 import cPickle as pickle |
4 | 19 import re |
172
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
20 try: |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
21 set |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
22 except NameError: |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
23 from sets import Set as set |
403 | 24 import shlex |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
25 import sys |
4 | 26 |
166
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
27 from bitten.build import CommandLine, FileSet |
169
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
28 from bitten.util import loc, xmlio |
4 | 29 |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
30 log = logging.getLogger('bitten.build.pythontools') |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
31 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
32 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
33 |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
34 def _python_path(ctxt): |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
35 """Return the path to the Python interpreter. |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
36 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
37 If the configuration has a ``python.path`` property, the value of that |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
38 option is returned; otherwise the path to the current Python interpreter is |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
39 returned. |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
40 """ |
240
24e91cbae6e0
New recipe command `<java:ant>` for running Ant builds.
cmlenz
parents:
233
diff
changeset
|
41 python_path = ctxt.config.get_filepath('python.path') |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
42 if python_path: |
240
24e91cbae6e0
New recipe command `<java:ant>` for running Ant builds.
cmlenz
parents:
233
diff
changeset
|
43 return python_path |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
44 return sys.executable |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
45 |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
46 def distutils(ctxt, file_='setup.py', command='build', |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
47 options=None, timeout=None): |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
48 """Execute a ``distutils`` command. |
313 | 49 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
50 :param ctxt: the build context |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
51 :type ctxt: `Context` |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
52 :param file\_: name of the file defining the distutils setup |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
53 :param command: the setup command to execute |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
54 :param options: additional options to pass to the command |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
55 :param timeout: the number of seconds before the external process should |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
56 be aborted (has same constraints as CommandLine) |
313 | 57 """ |
403 | 58 if options: |
59 if isinstance(options, basestring): | |
404
0bd6243b350c
Add `options` attribute to `<py:distutils>` recipe command.
cmlenz
parents:
403
diff
changeset
|
60 options = shlex.split(options) |
403 | 61 else: |
62 options = [] | |
63 | |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
64 if timeout: |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
65 timeout = int(timeout) |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
66 |
403 | 67 cmdline = CommandLine(_python_path(ctxt), |
68 [ctxt.resolve(file_), command] + options, | |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
69 cwd=ctxt.basedir) |
115
16d69eb6e047
Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}).
cmlenz
parents:
109
diff
changeset
|
70 log_elem = xmlio.Fragment() |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
71 error_logged = False |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
72 for out, err in cmdline.execute(timeout): |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
73 if out is not None: |
150
553784dccf97
Factored out common program execution logic into <sh:exec> command, which is used by <python:exec>.
cmlenz
parents:
146
diff
changeset
|
74 log.info(out) |
221 | 75 log_elem.append(xmlio.Element('message', level='info')[out]) |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
76 if err is not None: |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
77 level = 'error' |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
78 if err.startswith('warning: '): |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
79 err = err[9:] |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
80 level = 'warning' |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
81 log.warning(err) |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
82 elif err.startswith('error: '): |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
83 ctxt.error(err[7:]) |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
84 error_logged = True |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
85 else: |
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
86 log.error(err) |
221 | 87 log_elem.append(xmlio.Element('message', level=level)[err]) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
88 ctxt.log(log_elem) |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
89 |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
244
diff
changeset
|
90 if not error_logged and cmdline.returncode != 0: |
131
3ed8f568f60a
Fix error handling so that reports are still generated even if a command has failed.
cmlenz
parents:
129
diff
changeset
|
91 ctxt.error('distutils failed (%s)' % cmdline.returncode) |
4 | 92 |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
93 def exec_(ctxt, file_=None, module=None, function=None, |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
94 output=None, args=None, timeout=None): |
313 | 95 """Execute a Python script. |
96 | |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
97 Either the `file_` or the `module` parameter must be provided. If |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
98 specified using the `file_` parameter, the file must be inside the project |
313 | 99 directory. If specified as a module, the module must either be resolvable |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
100 to a file, or the `function` parameter must be provided |
313 | 101 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
102 :param ctxt: the build context |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
103 :type ctxt: `Context` |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
104 :param file\_: name of the script file to execute |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
105 :param module: name of the Python module to execute |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
106 :param function: name of the Python function to run |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
107 :param output: name of the file to which output should be written |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
108 :param args: extra arguments to pass to the script |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
109 :param timeout: the number of seconds before the external process should |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
110 be aborted (has same constraints as CommandLine) |
313 | 111 """ |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
112 assert file_ or module, 'Either "file" or "module" attribute required' |
244
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
113 if function: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
114 assert module and not file_, '"module" attribute required for use of ' \ |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
115 '"function" attribute' |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
116 |
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
117 if module: |
244
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
118 # Script specified as module name, need to resolve that to a file, |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
119 # or use the function name if provided |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
120 if function: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
121 args = '-c "import sys; from %s import %s; %s(sys.argv)" %s' % ( |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
122 module, function, function, args) |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
123 else: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
124 try: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
125 mod = __import__(module, globals(), locals(), []) |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
126 components = module.split('.') |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
127 for comp in components[1:]: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
128 mod = getattr(mod, comp) |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
129 file_ = mod.__file__.replace('\\', '/') |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
130 except ImportError, e: |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
131 ctxt.error('Cannot execute Python module %s: %s' % (module, e)) |
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
240
diff
changeset
|
132 return |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
133 |
150
553784dccf97
Factored out common program execution logic into <sh:exec> command, which is used by <python:exec>.
cmlenz
parents:
146
diff
changeset
|
134 from bitten.build import shtools |
454
ed81c13a5cac
`py:exec` recipe command now reports an error for process exit codes <> 0. Should close #174.
cmlenz
parents:
442
diff
changeset
|
135 returncode = shtools.execute(ctxt, executable=_python_path(ctxt), |
752
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
136 file_=file_, output=output, args=args, |
673ec182679d
Allow ''timeout'' parameter to be set on sh:exec, python:exec and
wbell
parents:
744
diff
changeset
|
137 timeout=timeout) |
454
ed81c13a5cac
`py:exec` recipe command now reports an error for process exit codes <> 0. Should close #174.
cmlenz
parents:
442
diff
changeset
|
138 if returncode != 0: |
744
0bec85f087e6
Give more information on failed Python commands that are specified by function/module.
wbell
parents:
721
diff
changeset
|
139 ctxt.error('Executing %s failed (error code %s)' % \ |
0bec85f087e6
Give more information on failed Python commands that are specified by function/module.
wbell
parents:
721
diff
changeset
|
140 (file_ or function or module, returncode)) |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
141 |
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
142 def pylint(ctxt, file_=None): |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
143 """Extract data from a ``pylint`` run written to a file. |
313 | 144 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
145 :param ctxt: the build context |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
146 :type ctxt: `Context` |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
147 :param file\_: name of the file containing the Pylint output |
313 | 148 """ |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
149 assert file_, 'Missing required attribute "file"' |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
150 msg_re = re.compile(r'^(?P<file>.+):(?P<line>\d+): ' |
145
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
151 r'\[(?P<type>[A-Z]\d*)(?:, (?P<tag>[\w\.]+))?\] ' |
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
152 r'(?P<msg>.*)$') |
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
153 msg_categories = dict(W='warning', E='error', C='convention', R='refactor') |
4 | 154 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
172
diff
changeset
|
155 problems = xmlio.Fragment() |
60
055a6c666fa8
* Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents:
57
diff
changeset
|
156 try: |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
157 fd = open(ctxt.resolve(file_), 'r') |
68 | 158 try: |
159 for line in fd: | |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
160 match = msg_re.search(line) |
68 | 161 if match: |
145
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
162 msg_type = match.group('type') |
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
163 category = msg_categories.get(msg_type[0]) |
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
164 if len(msg_type) == 1: |
0221d7cdf59a
Support for `--include-ids` option in Pylint. Closes #43.
cmlenz
parents:
144
diff
changeset
|
165 msg_type = None |
595
538e4f975505
0.6dev: Fix for filenames in pylint report that made incorrect absolute pathnames. Filenames should now be properly shortened, and link correctly to source browser from lint report.
osimons
parents:
486
diff
changeset
|
166 filename = match.group('file') |
538e4f975505
0.6dev: Fix for filenames in pylint report that made incorrect absolute pathnames. Filenames should now be properly shortened, and link correctly to source browser from lint report.
osimons
parents:
486
diff
changeset
|
167 if os.path.isabs(filename) \ |
538e4f975505
0.6dev: Fix for filenames in pylint report that made incorrect absolute pathnames. Filenames should now be properly shortened, and link correctly to source browser from lint report.
osimons
parents:
486
diff
changeset
|
168 and filename.startswith(ctxt.basedir): |
68 | 169 filename = filename[len(ctxt.basedir) + 1:] |
595
538e4f975505
0.6dev: Fix for filenames in pylint report that made incorrect absolute pathnames. Filenames should now be properly shortened, and link correctly to source browser from lint report.
osimons
parents:
486
diff
changeset
|
170 filename = filename.replace('\\', '/') |
68 | 171 lineno = int(match.group('line')) |
109
5bf22bb87915
Transmit build log and generated data back to the build master in XML format. Closes #23.
cmlenz
parents:
105
diff
changeset
|
172 tag = match.group('tag') |
221 | 173 problems.append(xmlio.Element('problem', category=category, |
174 type=msg_type, tag=tag, | |
175 line=lineno, file=filename)[ | |
915
e36f9b446976
Wrap lint report messages in a msg tag (includes a test and a documentation update). Fixes #547.
hodgestar
parents:
832
diff
changeset
|
176 xmlio.Element('msg')[match.group('msg') or ''] |
221 | 177 ]) |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
172
diff
changeset
|
178 ctxt.report('lint', problems) |
68 | 179 finally: |
180 fd.close() | |
181 except IOError, e: | |
131
3ed8f568f60a
Fix error handling so that reports are still generated even if a command has failed.
cmlenz
parents:
129
diff
changeset
|
182 log.warning('Error opening pylint results file (%s)', e) |
60
055a6c666fa8
* Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents:
57
diff
changeset
|
183 |
429 | 184 def coverage(ctxt, summary=None, coverdir=None, include=None, exclude=None): |
185 """Extract data from a ``coverage.py`` run. | |
186 | |
187 :param ctxt: the build context | |
188 :type ctxt: `Context` | |
189 :param summary: path to the file containing the coverage summary | |
190 :param coverdir: name of the directory containing the per-module coverage | |
191 details | |
192 :param include: patterns of files or directories to include in the report | |
193 :param exclude: patterns of files or directories to exclude from the report | |
194 """ | |
195 assert summary, 'Missing required attribute "summary"' | |
196 | |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
197 summary_line_re = re.compile(r'^(?P<module>.*?)\s+(?P<stmts>\d+)\s+' |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
198 r'(?P<exec>\d+)\s+(?P<cov>\d+)%\s+' |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
199 r'(?:(?P<missing>(?:\d+(?:-\d+)?(?:, )?)*)\s+)?' |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
200 r'(?P<file>.+)$') |
429 | 201 |
202 fileset = FileSet(ctxt.basedir, include, exclude) | |
203 missing_files = [] | |
204 for filename in fileset: | |
205 if os.path.splitext(filename)[1] != '.py': | |
206 continue | |
207 missing_files.append(filename) | |
208 covered_modules = set() | |
209 | |
210 try: | |
211 summary_file = open(ctxt.resolve(summary), 'r') | |
212 try: | |
213 coverage = xmlio.Fragment() | |
214 for summary_line in summary_file: | |
215 match = summary_line_re.search(summary_line) | |
216 if match: | |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
217 modname = match.group(1) |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
218 filename = match.group(6) |
429 | 219 if not os.path.isabs(filename): |
220 filename = os.path.normpath(os.path.join(ctxt.basedir, | |
221 filename)) | |
222 else: | |
223 filename = os.path.realpath(filename) | |
224 if not filename.startswith(ctxt.basedir): | |
225 continue | |
226 filename = filename[len(ctxt.basedir) + 1:] | |
227 if not filename in fileset: | |
228 continue | |
229 | |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
230 percentage = int(match.group(4).rstrip('%')) |
464
894b0b0721da
add line counts for coverage.py and some initial unit tests
mgood
parents:
454
diff
changeset
|
231 num_lines = int(match.group(2)) |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
232 |
429 | 233 missing_files.remove(filename) |
234 covered_modules.add(modname) | |
235 module = xmlio.Element('coverage', name=modname, | |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
429
diff
changeset
|
236 file=filename.replace(os.sep, '/'), |
464
894b0b0721da
add line counts for coverage.py and some initial unit tests
mgood
parents:
454
diff
changeset
|
237 percentage=percentage, |
894b0b0721da
add line counts for coverage.py and some initial unit tests
mgood
parents:
454
diff
changeset
|
238 lines=num_lines) |
429 | 239 coverage.append(module) |
240 | |
241 for filename in missing_files: | |
242 modname = os.path.splitext(filename.replace(os.sep, '.'))[0] | |
243 if modname in covered_modules: | |
244 continue | |
245 covered_modules.add(modname) | |
246 module = xmlio.Element('coverage', name=modname, | |
247 file=filename.replace(os.sep, '/'), | |
248 percentage=0) | |
249 coverage.append(module) | |
250 | |
251 ctxt.report('coverage', coverage) | |
252 finally: | |
253 summary_file.close() | |
254 except IOError, e: | |
255 log.warning('Error opening coverage summary file (%s)', e) | |
256 | |
60
055a6c666fa8
* Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents:
57
diff
changeset
|
257 def trace(ctxt, summary=None, coverdir=None, include=None, exclude=None): |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
258 """Extract data from a ``trace.py`` run. |
313 | 259 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
260 :param ctxt: the build context |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
261 :type ctxt: `Context` |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
262 :param summary: path to the file containing the coverage summary |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
263 :param coverdir: name of the directory containing the per-module coverage |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
264 details |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
265 :param include: patterns of files or directories to include in the report |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
266 :param exclude: patterns of files or directories to exclude from the report |
313 | 267 """ |
4 | 268 assert summary, 'Missing required attribute "summary"' |
269 assert coverdir, 'Missing required attribute "coverdir"' | |
270 | |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
271 summary_line_re = re.compile(r'^\s*(?P<lines>\d+)\s+(?P<cov>\d+)%\s+' |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
272 r'(?P<module>.*?)\s+\((?P<filename>.*?)\)') |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
273 coverage_line_re = re.compile(r'\s*(?:(?P<hits>\d+): )?(?P<line>.*)') |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
274 |
166
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
275 fileset = FileSet(ctxt.basedir, include, exclude) |
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
276 missing_files = [] |
169
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
277 for filename in fileset: |
172
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
278 if os.path.splitext(filename)[1] != '.py': |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
279 continue |
169
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
280 missing_files.append(filename) |
172
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
281 covered_modules = set() |
166
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
282 |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
283 def handle_file(elem, sourcefile, coverfile=None): |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
284 code_lines = set() |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
285 for lineno, linetype, line in loc.count(sourcefile): |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
286 if linetype == loc.CODE: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
287 code_lines.add(lineno) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
288 num_covered = 0 |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
289 lines = [] |
292
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
290 |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
291 if coverfile: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
292 prev_hits = '0' |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
293 for idx, coverline in enumerate(coverfile): |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
294 match = coverage_line_re.search(coverline) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
295 if match: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
296 hits = match.group(1) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
297 if hits: # Line covered |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
298 if hits != '0': |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
299 num_covered += 1 |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
300 lines.append(hits) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
301 prev_hits = hits |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
302 elif coverline.startswith('>'): # Line not covered |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
303 lines.append('0') |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
304 prev_hits = '0' |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
305 elif idx not in code_lines: # Not a code line |
355
c67532c7d0d2
Use "-" instead of "0" to mark a non-code line in `<py:trace>` reports, just as the new `<java:cobertura>` command does.
cmlenz
parents:
340
diff
changeset
|
306 lines.append('-') |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
307 prev_hits = '0' |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
308 else: # A code line not flagged by trace.py |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
309 if prev_hits != '0': |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
310 num_covered += 1 |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
311 lines.append(prev_hits) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
312 |
340 | 313 elem.append(xmlio.Element('line_hits')[' '.join(lines)]) |
292
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
314 |
608
3e018dcb1b91
0.6dev: Minor issue with `python:trace` where percentage was calculated based on covered lines as read from .cover file divided by code line count as read by `bitten.util.loc`. They obviously count code lines with some minor differences.
osimons
parents:
595
diff
changeset
|
315 num_lines = not lines and len(code_lines) or \ |
3e018dcb1b91
0.6dev: Minor issue with `python:trace` where percentage was calculated based on covered lines as read from .cover file divided by code line count as read by `bitten.util.loc`. They obviously count code lines with some minor differences.
osimons
parents:
595
diff
changeset
|
316 len([l for l in lines if l != '-']) |
292
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
317 if num_lines: |
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
318 percentage = int(round(num_covered * 100 / num_lines)) |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
319 else: |
292
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
320 percentage = 0 |
340 | 321 elem.attr['percentage'] = percentage |
322 elem.attr['lines'] = num_lines | |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
323 |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
324 try: |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
325 summary_file = open(ctxt.resolve(summary), 'r') |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
326 try: |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
327 coverage = xmlio.Fragment() |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
328 for summary_line in summary_file: |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
329 match = summary_line_re.search(summary_line) |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
330 if match: |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
331 modname = match.group(3) |
217
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
332 filename = match.group(4) |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
333 if not os.path.isabs(filename): |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
334 filename = os.path.normpath(os.path.join(ctxt.basedir, |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
335 filename)) |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
336 else: |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
337 filename = os.path.realpath(filename) |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
338 if not filename.startswith(ctxt.basedir): |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
339 continue |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
340 filename = filename[len(ctxt.basedir) + 1:] |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
341 if not filename in fileset: |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
342 continue |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
343 |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
344 missing_files.remove(filename) |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
345 covered_modules.add(modname) |
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
346 module = xmlio.Element('coverage', name=modname, |
292
3b9d79ef560f
Fix regression in test coverage collection introduced in [303]. Closes #67 (again).
cmlenz
parents:
291
diff
changeset
|
347 file=filename.replace(os.sep, '/')) |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
348 sourcefile = file(ctxt.resolve(filename)) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
349 try: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
350 coverpath = ctxt.resolve(coverdir, modname + '.cover') |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
351 if os.path.isfile(coverpath): |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
352 coverfile = file(coverpath, 'r') |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
353 else: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
354 log.warning('No coverage file for module %s at %s', |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
355 modname, coverpath) |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
356 coverfile = None |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
357 try: |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
358 handle_file(module, sourcefile, coverfile) |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
359 finally: |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
360 if coverfile: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
361 coverfile.close() |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
362 finally: |
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
363 sourcefile.close() |
217
44e91849ca43
Handle relative file paths in `trace.py` output. Closes #51.
cmlenz
parents:
213
diff
changeset
|
364 coverage.append(module) |
166
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
365 |
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
366 for filename in missing_files: |
172
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
367 modname = os.path.splitext(filename.replace(os.sep, '.'))[0] |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
368 if modname in covered_modules: |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
369 continue |
d7c8d4375374
More improvements to the `<python:trace>` code coverage report:
cmlenz
parents:
169
diff
changeset
|
370 covered_modules.add(modname) |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
172
diff
changeset
|
371 module = xmlio.Element('coverage', name=modname, |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
372 file=filename.replace(os.sep, '/'), |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
373 percentage=0) |
169
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
374 filepath = ctxt.resolve(filename) |
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
375 fileobj = file(filepath, 'r') |
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
376 try: |
291
d83208ed2db9
Change the way coverage statistics are collected, so that line number counting in covered and not covered files matches. Closes #67.
cmlenz
parents:
267
diff
changeset
|
377 handle_file(module, fileobj) |
169
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
378 finally: |
722215d17899
For more accurate code coverage reporting, include the statistics for modules that haven't been run at all during the tests. To do this, we need to count the lines of code in those modules. This is done by the `bitten.util.loc` module, which is based on [http://starship.python.net/crew/gherman/playground/pycount/ pycount.py] (but heavily modified).
cmlenz
parents:
166
diff
changeset
|
379 fileobj.close() |
166
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
380 coverage.append(module) |
60af98d66f11
* Move the `CommandLine` class from `bitten.util.cmdline` to `bitten.build.api`.
cmlenz
parents:
163
diff
changeset
|
381 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
172
diff
changeset
|
382 ctxt.report('coverage', coverage) |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
383 finally: |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
384 summary_file.close() |
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
385 except IOError, e: |
134
1a1a294ce10e
Catch XML parse errors in the {{{<python:unittest>}}} command.
cmlenz
parents:
133
diff
changeset
|
386 log.warning('Error opening coverage summary file (%s)', e) |
125
92e29e6b4c16
* The {{{python:trace>}}} recipe command now transmits coverage statistics to the build master. Closes #33.
cmlenz
parents:
118
diff
changeset
|
387 |
482 | 388 def figleaf(ctxt, summary=None, include=None, exclude=None): |
662
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
389 """Extract data from a ``Figleaf`` run. |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
390 |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
391 :param ctxt: the build context |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
392 :type ctxt: `Context` |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
393 :param summary: path to the file containing the coverage summary |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
394 :param include: patterns of files or directories to include in the report |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
395 :param exclude: patterns of files or directories to exclude from the report |
b00da52e942f
0.6dev: Adding docs for `<python:figleaf>` command. Closes #438.
osimons
parents:
608
diff
changeset
|
396 """ |
482 | 397 from figleaf import get_lines |
398 coverage = xmlio.Fragment() | |
399 try: | |
400 fileobj = open(ctxt.resolve(summary)) | |
401 except IOError, e: | |
402 log.warning('Error opening coverage summary file (%s)', e) | |
403 return | |
404 coverage_data = pickle.load(fileobj) | |
405 fileset = FileSet(ctxt.basedir, include, exclude) | |
406 for filename in fileset: | |
407 base, ext = os.path.splitext(filename) | |
408 if ext != '.py': | |
409 continue | |
486 | 410 modname = base.replace(os.path.sep, '.') |
482 | 411 realfilename = ctxt.resolve(filename) |
412 interesting_lines = get_lines(open(realfilename)) | |
721
8ca05f45ed2e
Fix a potential `ZeroDivisionError` in figleaf command. Closes #509.
osimons
parents:
666
diff
changeset
|
413 if not interesting_lines: |
8ca05f45ed2e
Fix a potential `ZeroDivisionError` in figleaf command. Closes #509.
osimons
parents:
666
diff
changeset
|
414 continue |
482 | 415 covered_lines = coverage_data.get(realfilename, set()) |
416 percentage = int(round(len(covered_lines) * 100 / len(interesting_lines))) | |
417 line_hits = [] | |
418 for lineno in xrange(1, max(interesting_lines)+1): | |
419 if lineno not in interesting_lines: | |
420 line_hits.append('-') | |
421 elif lineno in covered_lines: | |
422 line_hits.append('1') | |
423 else: | |
424 line_hits.append('0') | |
425 module = xmlio.Element('coverage', name=modname, | |
666
3c1c52f5e8ad
0.6dev: Fix for python:figleaf coverage annotation linking from Windows builds.
osimons
parents:
662
diff
changeset
|
426 file=filename.replace(os.sep, '/'), |
482 | 427 percentage=percentage, |
428 lines=len(interesting_lines), | |
429 line_hits=' '.join(line_hits)) | |
430 coverage.append(module) | |
431 ctxt.report('coverage', coverage) | |
432 | |
433 def _normalize_filenames(ctxt, filenames, fileset): | |
434 for filename in filenames: | |
435 if not os.path.isabs(filename): | |
436 filename = os.path.normpath(os.path.join(ctxt.basedir, | |
437 filename)) | |
438 else: | |
439 filename = os.path.realpath(filename) | |
440 if not filename.startswith(ctxt.basedir): | |
441 continue | |
442 filename = filename[len(ctxt.basedir) + 1:] | |
443 if filename not in fileset: | |
444 continue | |
445 yield filename.replace(os.sep, '/') | |
446 | |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
447 def unittest(ctxt, file_=None): |
313 | 448 """Extract data from a unittest results file in XML format. |
449 | |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
450 :param ctxt: the build context |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
451 :type ctxt: `Context` |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
452 :param file\_: name of the file containing the test results |
313 | 453 """ |
146
affd91b4c6fb
Add a `<python:exec>` recipe command so that things like Pylint can be executed without using a Makefile.
cmlenz
parents:
145
diff
changeset
|
454 assert file_, 'Missing required attribute "file"' |
8 | 455 |
60
055a6c666fa8
* Pass a {{{Context}}} object to recipe commands as the first argument. Currently this only has the basedir, but will be extended to also provide output recording etc.
cmlenz
parents:
57
diff
changeset
|
456 try: |
267 | 457 fileobj = file(ctxt.resolve(file_), 'r') |
68 | 458 try: |
249
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
459 total, failed = 0, 0 |
118
77dd69f7c405
* Make the recipe command {{{<python:unittest>}}} transmit the test results to the build master. Closes #32.
cmlenz
parents:
116
diff
changeset
|
460 results = xmlio.Fragment() |
267 | 461 for child in xmlio.parse(fileobj).children(): |
144
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
462 test = xmlio.Element('test') |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
463 for name, value in child.attr.items(): |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
464 if name == 'file': |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
465 value = os.path.realpath(value) |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
466 if value.startswith(ctxt.basedir): |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
467 value = value[len(ctxt.basedir) + 1:] |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
468 value = value.replace(os.sep, '/') |
144
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
469 else: |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
470 continue |
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
471 test.attr[name] = value |
249
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
472 if name == 'status' and value in ('error', 'failure'): |
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
473 failed += 1 |
144
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
474 for grandchild in child.children(): |
219 | 475 test.append(xmlio.Element(grandchild.name)[ |
476 grandchild.gettext() | |
477 ]) | |
144
76dea27af878
* Make the `<python:unittest>` command strip the base dir from file names in the report. Fixes #42.
cmlenz
parents:
134
diff
changeset
|
478 results.append(test) |
249
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
479 total += 1 |
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
480 if failed: |
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
481 ctxt.error('%d of %d test%s failed' % (failed, total, |
dcba83c01266
If `<python:unittest>` sees test failures, it logs an appropriate error containing the number of failures versus the total number of tests.
cmlenz
parents:
245
diff
changeset
|
482 total != 1 and 's' or '')) |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
172
diff
changeset
|
483 ctxt.report('test', results) |
68 | 484 finally: |
267 | 485 fileobj.close() |
68 | 486 except IOError, e: |
131
3ed8f568f60a
Fix error handling so that reports are still generated even if a command has failed.
cmlenz
parents:
129
diff
changeset
|
487 log.warning('Error opening unittest results file (%s)', e) |
134
1a1a294ce10e
Catch XML parse errors in the {{{<python:unittest>}}} command.
cmlenz
parents:
133
diff
changeset
|
488 except xmlio.ParseError, e: |
1a1a294ce10e
Catch XML parse errors in the {{{<python:unittest>}}} command.
cmlenz
parents:
133
diff
changeset
|
489 log.warning('Error parsing unittest results file (%s)', e) |