Mercurial > bitten > bitten-test

annotate doc/reports.txt @ 883:dfbf2f857a50

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fixed handling of active configurations that points to deleted branches. Configurations and builds can now be checked and displayed even though the repository path cannot be found for last revision. It uses configuration 'last' (`max_rev`) to "close" the configuration. Closes #606. Thanks to falkb for extensive testing.
author osimons
date Fri, 10 Dec 2010 09:23:12 +0000
parents fc90ef7b2b86
children e36f9b446976
rev   line source
818
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
1 .. -*- mode: rst; encoding: utf-8 -*-
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
2
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
3 ==============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
4 Report Formats
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
5 ==============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
6
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
7 The base element of the report must be "report" and have an attribute
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
8 "category" that is one of "test", "coverage" or "lint":
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
9
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
10 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
11
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
12 <report category="test|coverage|lint">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
13 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
14
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
15 Inside the report there must be elements for each report type. The way the
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
16 data is captured is pretty flexible because it can either be in attributes
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
17 or in child elements.
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
18
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
19 Test Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
20 ============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
21
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
22 Test reports must have sub-elements of report of type ``<test />``. These
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
23 elements can have any of these attributes (or subelements with contained
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
24 cdata):
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
25
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
26 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
27 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
28 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
29 | ``duration`` | Duration of test (float) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
30 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
31 | ``status`` | "success", "failure", "error", or "ignore" (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
32 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
33 | ``name`` | Name of the test (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
34 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
35 | ``fixture`` | Name of the test fixture (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
36 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
37 | ``file`` | Path to test file relative to the base path for the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
38 | | build configuration (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
39 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
40 | ``stdout`` | The output from the test (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
41 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
42 | ``traceback`` | The traceback from any error or failure (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
43 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
44
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
45 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
46
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
47 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
48
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
49 <report category="test">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
50 <test duration="0.073"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
51 status="success"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
52 fixture="bitten.tests.model.BuildConfigTestCase"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
53 name="test_config_update_name"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
54 file="bitten/tests/model.py"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
55 stdout="Renaming build configuration"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
56 traceback="None">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
57 </test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
58 <test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
59 <duration>0.073</duration>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
60 <status>success</status>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
61 <fixture>bitten.tests.model.BuildConfigTestCase</fixture>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
62 <name>test_config_update_name</name>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
63 <file>bitten/tests/model.py</file>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
64 <stdout>Renaming build configuration</stdout>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
65 </test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
66 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
67
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
68 Coverage Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
69 ================
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
70
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
71 Coverage reports must be inside ``<coverage />`` elements. The
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
72 allowable attributes are:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
73
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
74 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
75 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
76 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
77 | ``name`` | The name of the module being tested for coverage |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
78 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
79 | ``file`` | The name of the file relative to the base path in the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
80 | | build configuration |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
81 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
82 | ``percentage`` | The percentage of lines in that file covered |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
83 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
84 | ``lines`` | The number of lines covered |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
85 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
86 | ``line_hits`` | Line-by-line coverage of the file, where code lines have |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
87 | | 0 or more times covered and non-code lines are marked |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
88 | | as `'-'` (optional) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
89 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
90
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
91 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
92
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
93 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
94
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
95 <report category="coverage">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
96 <coverage name="my_module"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
97 file="my_module.py"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
98 percentage="75"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
99 lines="4"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
100 line_hits="2 0 - 1 1">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
101 </coverage>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
102 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
103
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
104 Lint Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
105 ============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
106
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
107 Lint issues are placed inside ``<problem />`` elements, with allowed attributes of:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
108
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
109 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
110 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
111 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
112 | ``file`` | The name of the file relative to the base path in the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
113 | | build configuration |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
114 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
115 | ``tag`` | Class, method or other useful identifiable location |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
116 | | inside the file |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
117 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
118 | ``line`` | Line number |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
119 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
120 | ``category`` | Category for problem; |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
121 | | ``convention \| warning \| refactor \| error`` |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
122 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
123
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
124 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
125
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
126 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
127
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
128 <report category="lint">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
129 <problem category="convention"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
130 line="17"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
131 tag="TestResultsChartGenerator"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
132 file="bitten/report/testing.py">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
133 Missing docstring
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
134 </problem>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
135 </report>
Copyright (C) 2012-2017 Edgewall Software