Mercurial > bitten > bitten-test

annotate doc/reports.txt @ 899:6ac02ad71cc4

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Update ChangeLog before 0.6 release.
author hodgestar
date Fri, 11 Mar 2011 11:10:41 +0000
parents fc90ef7b2b86
children e36f9b446976
rev   line source
818
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
1 .. -*- mode: rst; encoding: utf-8 -*-
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
2
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
3 ==============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
4 Report Formats
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
5 ==============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
6
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
7 The base element of the report must be "report" and have an attribute
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
8 "category" that is one of "test", "coverage" or "lint":
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
9
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
10 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
11
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
12 <report category="test|coverage|lint">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
13 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
14
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
15 Inside the report there must be elements for each report type. The way the
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
16 data is captured is pretty flexible because it can either be in attributes
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
17 or in child elements.
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
18
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
19 Test Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
20 ============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
21
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
22 Test reports must have sub-elements of report of type ``<test />``. These
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
23 elements can have any of these attributes (or subelements with contained
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
24 cdata):
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
25
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
26 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
27 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
28 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
29 | ``duration`` | Duration of test (float) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
30 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
31 | ``status`` | "success", "failure", "error", or "ignore" (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
32 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
33 | ``name`` | Name of the test (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
34 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
35 | ``fixture`` | Name of the test fixture (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
36 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
37 | ``file`` | Path to test file relative to the base path for the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
38 | | build configuration (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
39 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
40 | ``stdout`` | The output from the test (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
41 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
42 | ``traceback`` | The traceback from any error or failure (string) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
43 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
44
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
45 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
46
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
47 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
48
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
49 <report category="test">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
50 <test duration="0.073"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
51 status="success"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
52 fixture="bitten.tests.model.BuildConfigTestCase"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
53 name="test_config_update_name"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
54 file="bitten/tests/model.py"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
55 stdout="Renaming build configuration"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
56 traceback="None">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
57 </test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
58 <test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
59 <duration>0.073</duration>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
60 <status>success</status>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
61 <fixture>bitten.tests.model.BuildConfigTestCase</fixture>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
62 <name>test_config_update_name</name>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
63 <file>bitten/tests/model.py</file>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
64 <stdout>Renaming build configuration</stdout>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
65 </test>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
66 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
67
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
68 Coverage Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
69 ================
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
70
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
71 Coverage reports must be inside ``<coverage />`` elements. The
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
72 allowable attributes are:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
73
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
74 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
75 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
76 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
77 | ``name`` | The name of the module being tested for coverage |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
78 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
79 | ``file`` | The name of the file relative to the base path in the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
80 | | build configuration |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
81 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
82 | ``percentage`` | The percentage of lines in that file covered |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
83 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
84 | ``lines`` | The number of lines covered |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
85 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
86 | ``line_hits`` | Line-by-line coverage of the file, where code lines have |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
87 | | 0 or more times covered and non-code lines are marked |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
88 | | as `'-'` (optional) |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
89 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
90
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
91 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
92
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
93 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
94
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
95 <report category="coverage">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
96 <coverage name="my_module"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
97 file="my_module.py"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
98 percentage="75"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
99 lines="4"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
100 line_hits="2 0 - 1 1">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
101 </coverage>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
102 </report>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
103
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
104 Lint Reports
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
105 ============
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
106
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
107 Lint issues are placed inside ``<problem />`` elements, with allowed attributes of:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
108
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
109 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
110 | Attribute | Description |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
111 +=================+==========================================================+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
112 | ``file`` | The name of the file relative to the base path in the |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
113 | | build configuration |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
114 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
115 | ``tag`` | Class, method or other useful identifiable location |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
116 | | inside the file |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
117 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
118 | ``line`` | Line number |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
119 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
120 | ``category`` | Category for problem; |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
121 | | ``convention \| warning \| refactor \| error`` |
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
122 +-----------------+----------------------------------------------------------+
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
123
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
124 Example:
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
125
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
126 .. code-block:: xml
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
127
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
128 <report category="lint">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
129 <problem category="convention"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
130 line="17"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
131 tag="TestResultsChartGenerator"
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
132 file="bitten/report/testing.py">
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
133 Missing docstring
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
134 </problem>
fc90ef7b2b86 Moving the wiki:ReportFormats page into the repository.
osimons
parents:
diff changeset
135 </report>
Copyright (C) 2012-2017 Edgewall Software