Mercurial > bitten > bitten-test
annotate bitten/upgrades.py @ 860:ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
author | hodgestar |
---|---|
date | Sun, 17 Oct 2010 23:20:44 +0000 |
parents | f606ee47b8ca |
children | 14010fa88b4b |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
2 # |
832 | 3 # Copyright (C) 2007-2010 Edgewall Software |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
379
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
163 | 5 # All rights reserved. |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
6 # |
163 | 7 # This software is licensed as described in the file COPYING, which |
8 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
379
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
10 |
316 | 11 """Automated upgrades for the Bitten database tables, and other data stored |
775
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
12 in the Trac environment. |
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
13 |
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
14 **Do not import and call directly!**""" |
316 | 15 |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
16 import os |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
17 import sys |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
18 |
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
19 from trac.core import TracError |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
20 from trac.db import DatabaseManager, Table, Column, Index |
519
384e59137bf8
Support unicode by converting everything to UTF-8 on write and back to unicode on read - should fix #369
dfraser
parents:
518
diff
changeset
|
21 from trac.util.text import to_unicode |
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
22 import codecs |
320 | 23 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
24 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
25 |
708 | 26 # database abstraction functions |
27 | |
846
ee95915922dd
Add checks that (PostgreSQL) sequences have ended up in the correct state at the end of the upgrade scripts. Test currently fails on PostgreSQL as a result of #632.
hodgestar
parents:
841
diff
changeset
|
28 def parse_scheme(env): |
708 | 29 """Retrieve the environment database scheme.""" |
30 connection_uri = DatabaseManager(env).connection_uri | |
31 parts = connection_uri.split(':', 1) | |
32 scheme = parts[0].lower() | |
33 return scheme | |
34 | |
35 def update_sequence(env, db, tbl, col): | |
36 """Update a sequence associated with an autoincrement column.""" | |
37 # Hopefully Trac will eventually implement its own version | |
38 # of this function. | |
846
ee95915922dd
Add checks that (PostgreSQL) sequences have ended up in the correct state at the end of the upgrade scripts. Test currently fails on PostgreSQL as a result of #632.
hodgestar
parents:
841
diff
changeset
|
39 scheme = parse_scheme(env) |
708 | 40 if scheme == "postgres": |
41 seq = '%s_%s_seq' % (tbl, col) | |
42 cursor = db.cursor() | |
43 cursor.execute("SELECT setval('%s', (SELECT MAX(%s) FROM %s))" | |
44 % (seq, col, tbl)) | |
45 | |
46 def drop_index(env, db, tbl, idx): | |
47 """Drop an index associated with a table.""" | |
48 # Hopefully Trac will eventually implement its own version | |
49 # of this function. | |
846
ee95915922dd
Add checks that (PostgreSQL) sequences have ended up in the correct state at the end of the upgrade scripts. Test currently fails on PostgreSQL as a result of #632.
hodgestar
parents:
841
diff
changeset
|
50 scheme = parse_scheme(env) |
708 | 51 cursor = db.cursor() |
52 if scheme == "mysql": | |
53 cursor.execute("DROP INDEX %s ON %s" % (idx, tbl)) | |
54 else: | |
55 cursor.execute("DROP INDEX %s" % (idx,)) | |
56 | |
57 # upgrade scripts | |
58 | |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
59 def add_log_table(env, db): |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
60 """Add a table for storing the builds logs.""" |
708 | 61 INFO_LEVEL = 'I' |
62 | |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
63 cursor = db.cursor() |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
64 |
708 | 65 build_log_schema_v3 = [ |
66 Table('bitten_log', key='id')[ | |
67 Column('id', auto_increment=True), Column('build', type='int'), | |
68 Column('step'), Column('type') | |
69 ], | |
70 Table('bitten_log_message', key=('log', 'line'))[ | |
71 Column('log', type='int'), Column('line', type='int'), | |
72 Column('level', size=1), Column('message') | |
73 ] | |
74 ] | |
75 | |
76 build_step_schema_v3 = [ | |
77 Table('bitten_step', key=('build', 'name'))[ | |
78 Column('build', type='int'), Column('name'), Column('description'), | |
79 Column('status', size=1), Column('started', type='int'), | |
80 Column('stopped', type='int') | |
81 ] | |
82 ] | |
83 | |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
84 connector, _ = DatabaseManager(env)._get_connector() |
708 | 85 for table in build_log_schema_v3: |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
86 for stmt in connector.to_sql(table): |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
87 cursor.execute(stmt) |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
88 |
708 | 89 update_cursor = db.cursor() |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
90 cursor.execute("SELECT build,name,log FROM bitten_step " |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
91 "WHERE log IS NOT NULL") |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
92 for build, step, log in cursor: |
708 | 93 update_cursor.execute("INSERT INTO bitten_log (build, step) " |
94 "VALUES (%s,%s)", (build, step)) | |
95 log_id = db.get_last_id(update_cursor, 'bitten_log') | |
96 messages = [(log_id, line, INFO_LEVEL, msg) | |
97 for line, msg in enumerate(log.splitlines())] | |
98 update_cursor.executemany("INSERT INTO bitten_log_message (log, line, level, message) " | |
99 "VALUES (%s, %s, %s, %s)", messages) | |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
100 |
524
c022478bc111
Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370
dfraser
parents:
521
diff
changeset
|
101 cursor.execute("CREATE TEMPORARY TABLE old_step AS SELECT * FROM bitten_step") |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
102 cursor.execute("DROP TABLE bitten_step") |
708 | 103 for table in build_step_schema_v3: |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
104 for stmt in connector.to_sql(table): |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
105 cursor.execute(stmt) |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
106 cursor.execute("INSERT INTO bitten_step (build,name,description,status," |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
107 "started,stopped) SELECT build,name,description,status," |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
108 "started,stopped FROM old_step") |
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
109 |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
110 def add_recipe_to_config(env, db): |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
111 """Add a column for storing the build recipe to the build configuration |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
112 table.""" |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
113 cursor = db.cursor() |
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
114 |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
115 build_config_schema_v3 = Table('bitten_config', key='name')[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
116 Column('name'), Column('path'), Column('active', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
117 Column('recipe'), Column('min_rev'), Column('max_rev'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
118 Column('label'), Column('description') |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
119 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
120 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
121 cursor.execute("CREATE TEMPORARY TABLE old_config_v2 AS " |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
122 "SELECT * FROM bitten_config") |
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
123 cursor.execute("DROP TABLE bitten_config") |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
124 |
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
125 connector, _ = DatabaseManager(env)._get_connector() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
126 for stmt in connector.to_sql(build_config_schema_v3): |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
127 cursor.execute(stmt) |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
128 |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
129 cursor.execute("INSERT INTO bitten_config (name,path,active,recipe,min_rev," |
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
130 "max_rev,label,description) SELECT name,path,0,'',NULL," |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
131 "NULL,label,description FROM old_config_v2") |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
132 |
762
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
133 def add_last_activity_to_build(env, db): |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
134 """Add a column for storing the last activity to the build table.""" |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
135 cursor = db.cursor() |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
136 |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
137 build_table_schema_v12 = Table('bitten_build', key='id')[ |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
138 Column('id', auto_increment=True), Column('config'), Column('rev'), |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
139 Column('rev_time', type='int'), Column('platform', type='int'), |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
140 Column('slave'), Column('started', type='int'), |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
141 Column('stopped', type='int'), Column('status', size=1), |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
142 Column('last_activity', type='int'), |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
143 Index(['config', 'rev', 'platform'], unique=True) |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
144 ] |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
145 |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
146 cursor.execute("CREATE TEMPORARY TABLE old_build_v11 AS " |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
147 "SELECT * FROM bitten_build") |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
148 cursor.execute("DROP TABLE bitten_build") |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
149 |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
150 connector, _ = DatabaseManager(env)._get_connector() |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
151 for stmt in connector.to_sql(build_table_schema_v12): |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
152 cursor.execute(stmt) |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
153 |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
154 # it's safe to make the last activity the stop time of the build |
788 | 155 cursor.execute("INSERT INTO bitten_build (id,config,rev,rev_time,platform," |
762
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
156 "slave,started,stopped,last_activity,status) " |
788 | 157 "SELECT id,config,rev,rev_time,platform," |
762
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
158 "slave,started,stopped,stopped,status FROM old_build_v11") |
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
159 |
847
f606ee47b8ca
Update bitten_build id sequence on PostgreSQL after re-creating bitten_build table when adding last_activity column. See #632.
hodgestar
parents:
846
diff
changeset
|
160 update_sequence(env, db, 'bitten_build', 'id') |
f606ee47b8ca
Update bitten_build id sequence on PostgreSQL after re-creating bitten_build table when adding last_activity column. See #632.
hodgestar
parents:
846
diff
changeset
|
161 |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
162 def add_config_to_reports(env, db): |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
163 """Add the name of the build configuration as metadata to report documents |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
164 stored in the BDB XML database.""" |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
165 try: |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
166 from bsddb3 import db as bdb |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
167 import dbxml |
219 | 168 except ImportError: |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
169 return |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
170 |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
171 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml') |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
172 if not os.path.isfile(dbfile): |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
173 return |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
174 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
175 dbenv = bdb.DBEnv() |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
176 dbenv.open(os.path.dirname(dbfile), |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
177 bdb.DB_CREATE | bdb.DB_INIT_LOCK | bdb.DB_INIT_LOG | |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
178 bdb.DB_INIT_MPOOL | bdb.DB_INIT_TXN, 0) |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
179 |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
180 mgr = dbxml.XmlManager(dbenv, 0) |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
181 xtn = mgr.createTransaction() |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
182 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL) |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
183 uc = mgr.createUpdateContext() |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
184 |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
185 container.addIndex(xtn, '', 'config', 'node-metadata-equality-string', uc) |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
186 |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
187 qc = mgr.createQueryContext() |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
188 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc): |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
189 doc = value.asDocument() |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
190 metaval = dbxml.XmlValue() |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
191 if doc.getMetaData('', 'build', metaval): |
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
192 build_id = int(metaval.asNumber()) |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
193 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
194 cursor = db.cursor() |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
195 cursor.execute("SELECT config FROM bitten_build WHERE id=%s", (build_id,)) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
196 row = cursor.fetchone() |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
197 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
198 if row: |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
199 doc.setMetaData('', 'config', dbxml.XmlValue(row[0])) |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
200 container.updateDocument(xtn, doc, uc) |
175
f7c2f112afe6
Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it.
cmlenz
parents:
174
diff
changeset
|
201 else: |
f7c2f112afe6
Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it.
cmlenz
parents:
174
diff
changeset
|
202 # an orphaned report, for whatever reason... just remove it |
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
203 container.deleteDocument(xtn, doc, uc) |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
204 |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
205 xtn.commit() |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
206 container.close() |
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
207 dbenv.close(0) |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
208 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
209 def add_order_to_log(env, db): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
210 """Add order column to log table to make sure that build logs are displayed |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
211 in the order they were generated.""" |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
212 cursor = db.cursor() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
213 |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
214 log_table_schema_v6 = Table('bitten_log', key='id')[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
215 Column('id', auto_increment=True), Column('build', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
216 Column('step'), Column('generator'), Column('orderno', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
217 Index(['build', 'step']) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
218 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
219 |
708 | 220 cursor.execute("CREATE TEMPORARY TABLE old_log_v5 AS " |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
221 "SELECT * FROM bitten_log") |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
222 cursor.execute("DROP TABLE bitten_log") |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
223 |
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
224 connector, _ = DatabaseManager(env)._get_connector() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
225 for stmt in connector.to_sql(log_table_schema_v6): |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
226 cursor.execute(stmt) |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
227 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
228 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno) " |
708 | 229 "SELECT id,build,step,type,0 FROM old_log_v5") |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
230 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
231 def add_report_tables(env, db): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
232 """Add database tables for report storage.""" |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
233 cursor = db.cursor() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
234 |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
235 report_schema_v6 = Table('bitten_report', key='id')[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
236 Column('id', auto_increment=True), Column('build', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
237 Column('step'), Column('category'), Column('generator'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
238 Index(['build', 'step', 'category']) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
239 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
240 report_item_schema_v6 = Table('bitten_report_item', key=('report', 'item', 'name'))[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
241 Column('report', type='int'), Column('item', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
242 Column('name'), Column('value') |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
243 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
244 |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
245 connector, _ = DatabaseManager(env)._get_connector() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
246 for table in [report_schema_v6, report_item_schema_v6]: |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
247 for stmt in connector.to_sql(table): |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
248 cursor.execute(stmt) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
249 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
250 def xmldb_to_db(env, db): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
251 """Migrate report data from Berkeley DB XML to SQL database. |
762
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
252 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
253 Depending on the number of reports stored, this might take rather long. |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
254 After the upgrade is done, the bitten.dbxml file (and any BDB XML log files) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
255 may be deleted. BDB XML is no longer used by Bitten. |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
256 """ |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
257 from bitten.util import xmlio |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
258 try: |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
259 from bsddb3 import db as bdb |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
260 import dbxml |
219 | 261 except ImportError: |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
262 return |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
263 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
264 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml') |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
265 if not os.path.isfile(dbfile): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
266 return |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
267 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
268 dbenv = bdb.DBEnv() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
269 dbenv.open(os.path.dirname(dbfile), |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
270 bdb.DB_CREATE | bdb.DB_INIT_LOCK | bdb.DB_INIT_LOG | |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
271 bdb.DB_INIT_MPOOL | bdb.DB_INIT_TXN, 0) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
272 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
273 mgr = dbxml.XmlManager(dbenv, 0) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
274 xtn = mgr.createTransaction() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
275 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
276 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
277 def get_pylint_items(xml): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
278 for problems_elem in xml.children('problems'): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
279 for problem_elem in problems_elem.children('problem'): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
280 item = {'type': 'problem'} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
281 item.update(problem_elem.attr) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
282 yield item |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
283 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
284 def get_trace_items(xml): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
285 for cov_elem in xml.children('coverage'): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
286 item = {'type': 'coverage', 'name': cov_elem.attr['module'], |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
287 'file': cov_elem.attr['file'], |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
288 'percentage': cov_elem.attr['percentage']} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
289 lines = 0 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
290 line_hits = [] |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
291 for line_elem in cov_elem.children('line'): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
292 lines += 1 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
293 line_hits.append(line_elem.attr['hits']) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
294 item['lines'] = lines |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
295 item['line_hits'] = ' '.join(line_hits) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
296 yield item |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
297 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
298 def get_unittest_items(xml): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
299 for test_elem in xml.children('test'): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
300 item = {'type': 'test'} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
301 item.update(test_elem.attr) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
302 for child_elem in test_elem.children(): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
303 item[child_elem.name] = child_elem.gettext() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
304 yield item |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
305 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
306 qc = mgr.createQueryContext() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
307 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc, 0): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
308 doc = value.asDocument() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
309 metaval = dbxml.XmlValue() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
310 build, step = None, None |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
311 if doc.getMetaData('', 'build', metaval): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
312 build = metaval.asNumber() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
313 if doc.getMetaData('', 'step', metaval): |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
314 step = metaval.asString() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
315 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
316 report_types = {'pylint': ('lint', get_pylint_items), |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
317 'trace': ('coverage', get_trace_items), |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
318 'unittest': ('test', get_unittest_items)} |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
319 xml = xmlio.parse(value.asString()) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
320 report_type = xml.attr['type'] |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
321 category, get_items = report_types[report_type] |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
322 sys.stderr.write('.') |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
323 sys.stderr.flush() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
324 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
325 items = list(get_items(xml)) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
326 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
327 cursor = db.cursor() |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
328 cursor.execute("SELECT bitten_report.id FROM bitten_report " |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
329 "WHERE build=%s AND step=%s AND category=%s", |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
330 (build, step, category)) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
331 rows = cursor.fetchall() |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
332 if rows: |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
333 # Duplicate report, skip |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
334 continue |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
335 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
336 cursor.execute("INSERT INTO bitten_report " |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
337 "(build,step,category,generator) VALUES (%s,%s,%s,%s)", |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
338 (build, step, category, report_type)) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
339 id = db.get_last_id(cursor, 'bitten_report') |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
340 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
341 for idx, item in enumerate(items): |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
342 cursor.executemany("INSERT INTO bitten_report_item " |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
343 "(report,item,name,value) VALUES (%s,%s,%s,%s)", |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
344 [(id, idx, key, value) for key, value |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
345 in item.items()]) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
346 |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
347 sys.stderr.write('\n') |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
348 sys.stderr.flush() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
349 |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
350 xtn.abort() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
351 container.close() |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
352 dbenv.close(0) |
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
353 |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
354 def normalize_file_paths(env, db): |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
355 """Normalize the file separator in file names in reports.""" |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
356 cursor = db.cursor() |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
357 cursor.execute("SELECT report,item,value FROM bitten_report_item " |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
358 "WHERE name='file'") |
223
067bde207c23
Fix upgrade for installs with no stored reports. Thanks to Matt Good for catching this.
cmlenz
parents:
219
diff
changeset
|
359 rows = cursor.fetchall() or [] |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
360 for report, item, value in rows: |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
361 if '\\' in value: |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
362 cursor.execute("UPDATE bitten_report_item SET value=%s " |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
363 "WHERE report=%s AND item=%s AND name='file'", |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
364 (value.replace('\\', '/'), report, item)) |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
365 |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
366 def fixup_generators(env, db): |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
367 """Upgrade the identifiers for the recipe commands that generated log |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
368 messages and report data.""" |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
369 |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
370 mapping = { |
683
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
371 'pipe': 'http://bitten.edgewall.org/tools/sh#pipe', |
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
372 'make': 'http://bitten.edgewall.org/tools/c#make', |
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
373 'distutils': 'http://bitten.edgewall.org/tools/python#distutils', |
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
374 'exec_': 'http://bitten.edgewall.org/tools/python#exec' # Ambigious |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
375 } |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
376 cursor = db.cursor() |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
377 cursor.execute("SELECT id,generator FROM bitten_log " |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
378 "WHERE generator IN (%s)" |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
379 % ','.join([repr(key) for key in mapping.keys()])) |
219 | 380 for log_id, generator in cursor: |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
381 cursor.execute("UPDATE bitten_log SET generator=%s " |
219 | 382 "WHERE id=%s", (mapping[generator], log_id)) |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
383 |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
384 mapping = { |
683
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
385 'unittest': 'http://bitten.edgewall.org/tools/python#unittest', |
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
386 'trace': 'http://bitten.edgewall.org/tools/python#trace', |
0c4fec90c8e2
0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though.
osimons
parents:
656
diff
changeset
|
387 'pylint': 'http://bitten.edgewall.org/tools/python#pylint' |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
388 } |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
389 cursor.execute("SELECT id,generator FROM bitten_report " |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
390 "WHERE generator IN (%s)" |
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
391 % ','.join([repr(key) for key in mapping.keys()])) |
219 | 392 for report_id, generator in cursor: |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
393 cursor.execute("UPDATE bitten_report SET generator=%s " |
219 | 394 "WHERE id=%s", (mapping[generator], report_id)) |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
395 |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
396 def add_error_table(env, db): |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
397 """Add the bitten_error table for recording step failure reasons.""" |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
398 table = Table('bitten_error', key=('build', 'step', 'orderno'))[ |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
399 Column('build', type='int'), Column('step'), Column('message'), |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
400 Column('orderno', type='int') |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
401 ] |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
402 cursor = db.cursor() |
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
403 |
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
404 connector, _ = DatabaseManager(env)._get_connector() |
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
405 for stmt in connector.to_sql(table): |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
406 cursor.execute(stmt) |
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
407 |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
408 def add_filename_to_logs(env, db): |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
409 """Add filename column to log table to save where log files are stored.""" |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
410 cursor = db.cursor() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
411 |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
412 build_log_schema_v9 = Table('bitten_log', key='id')[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
413 Column('id', auto_increment=True), Column('build', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
414 Column('step'), Column('generator'), Column('orderno', type='int'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
415 Column('filename'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
416 Index(['build', 'step']) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
417 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
418 |
708 | 419 cursor.execute("CREATE TEMPORARY TABLE old_log_v8 AS " |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
420 "SELECT * FROM bitten_log") |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
421 cursor.execute("DROP TABLE bitten_log") |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
422 |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
423 connector, _ = DatabaseManager(env)._get_connector() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
424 for stmt in connector.to_sql(build_log_schema_v9): |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
425 cursor.execute(stmt) |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
426 |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
427 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno,filename) " |
708 | 428 "SELECT id,build,step,generator,orderno,'' FROM old_log_v8") |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
429 |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
430 def migrate_logs_to_files(env, db): |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
431 """Migrates logs that are stored in the bitten_log_messages table into files.""" |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
432 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
433 if not os.path.isabs(logs_dir): |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
434 logs_dir = os.path.join(env.path, logs_dir) |
860
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
435 |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
436 if os.path.exists(logs_dir): |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
437 print "Bitten log folder %r already exists" % (logs_dir,) |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
438 print "Upgrade cannot be performed until the existing folder is moved." |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
439 print "The upgrade script will now exit with an error:\n" |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
440 raise TracError("") |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
441 |
ac28bc3feb2a
Refuse to create log folder during database upgrade if the folder already exists. The existence of such a folder is an indication that a previous attempt to migrate the logs into files has failed and that the folder should be removed before continuing. See #462 for further discussion.
hodgestar
parents:
847
diff
changeset
|
442 os.makedirs(logs_dir) |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
443 |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
444 cursor = db.cursor() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
445 message_cursor = db.cursor() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
446 update_cursor = db.cursor() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
447 cursor.execute("SELECT id FROM bitten_log") |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
448 for log_id, in cursor.fetchall(): |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
449 filename = "%s.log" % (log_id,) |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
450 message_cursor.execute("SELECT message, level FROM bitten_log_message WHERE log=%s ORDER BY line", (log_id,)) |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
451 full_filename = os.path.join(logs_dir, filename) |
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
452 message_file = codecs.open(full_filename, "wb", "UTF-8") |
727
508636ffbf11
Added a constant so we don't have any more confusion about what files are used for levels - see #517
dfraser
parents:
708
diff
changeset
|
453 # Note: the original version of this code erroneously wrote to filename + ".level" instead of ".levels", producing unused level files |
735
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
454 level_file = codecs.open(full_filename + '.levels', "wb", "UTF-8") |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
455 for message, level in message_cursor.fetchall() or []: |
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
456 message_file.write(to_unicode(message) + "\n") |
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
457 level_file.write(to_unicode(level) + "\n") |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
458 message_file.close() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
459 level_file.close() |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
460 update_cursor.execute("UPDATE bitten_log SET filename=%s WHERE id=%s", (filename, log_id)) |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
461 env.log.info("Migrated log %s", log_id) |
518 | 462 env.log.warning("Logs have been migrated from the database to files in %s. " |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
463 "Ensure permissions are set correctly on this file. " |
619
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
464 "Since we presume that the migration worked correctly, " |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
465 "we are now dropping the bitten_log_message table in the database (aren't you glad you backed up)", logs_dir) |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
466 cursor.close() |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
467 cursor = db.cursor() |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
468 cursor.execute("DROP TABLE bitten_log_message") |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
469 cursor.close() |
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
470 env.log.warning("We have dropped the bitten_log_message table - you may want to vaccuum/compress your database to save space") |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
471 |
735
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
472 def fix_log_levels_misnaming(env, db): |
775
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
473 """Renames or removes \*.log.level files created by older versions of migrate_logs_to_files.""" |
735
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
474 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
475 if not os.path.isabs(logs_dir): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
476 logs_dir = os.path.join(env.path, logs_dir) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
477 if not os.path.isdir(logs_dir): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
478 return |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
479 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
480 rename_count = 0 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
481 rename_error_count = 0 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
482 delete_count = 0 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
483 delete_error_count = 0 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
484 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
485 for wrong_filename in os.listdir(logs_dir): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
486 if not wrong_filename.endswith('.log.level'): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
487 continue |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
488 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
489 log_filename = os.path.splitext(wrong_filename)[0] |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
490 right_filename = log_filename + '.levels' |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
491 full_log_filename = os.path.join(logs_dir, log_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
492 full_wrong_filename = os.path.join(logs_dir, wrong_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
493 full_right_filename = os.path.join(logs_dir, right_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
494 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
495 if not os.path.exists(full_log_filename): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
496 try: |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
497 os.remove(full_wrong_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
498 delete_count += 1 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
499 env.log.info("Deleted stray log level file %s", wrong_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
500 except Exception, e: |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
501 delete_error_count += 1 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
502 env.log.warning("Error removing stray log level file %s: %s", wrong_filename, e) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
503 else: |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
504 if os.path.exists(full_right_filename): |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
505 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: new filename already exists", |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
506 full_wrong_filename, full_right_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
507 rename_error_count += 1 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
508 continue |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
509 try: |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
510 os.rename(full_wrong_filename, full_right_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
511 rename_count += 1 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
512 env.log.info("Renamed incorrectly named log level file %s to %s", wrong_filename, right_filename) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
513 except Exception, e: |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
514 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: %s", full_wrong_filename, full_right_filename, e) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
515 rename_error_count += 1 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
516 |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
517 env.log.info("Renamed %d incorrectly named log level files from previous migrate (%d errors)", rename_count, rename_error_count) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
518 env.log.info("Deleted %d stray log level (%d errors)", delete_count, delete_error_count) |
90903d6cc932
Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.
hodgestar
parents:
727
diff
changeset
|
519 |
737
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
520 def remove_stray_log_levels_files(env, db): |
775
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
521 """Remove \*.log.levels files without a matching \*.log file (old Bitten |
025b3e889321
Doc fixes. Making docs compile without errors + some minor tweaks to output.
osimons
parents:
762
diff
changeset
|
522 versions did not delete .log.levels files when builds were deleted)""" |
737
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
523 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
524 if not os.path.isabs(logs_dir): |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
525 logs_dir = os.path.join(env.path, logs_dir) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
526 if not os.path.isdir(logs_dir): |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
527 return |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
528 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
529 delete_count = 0 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
530 delete_error_count = 0 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
531 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
532 for filename in os.listdir(logs_dir): |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
533 if not filename.endswith('.log.levels'): |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
534 continue |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
535 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
536 log_filename = os.path.splitext(filename)[0] |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
537 full_log_filename = os.path.join(logs_dir, log_filename) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
538 full_filename = os.path.join(logs_dir, filename) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
539 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
540 if not os.path.exists(full_log_filename): |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
541 try: |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
542 os.remove(full_filename) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
543 delete_count += 1 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
544 env.log.info("Deleted stray log levels file %s", filename) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
545 except Exception, e: |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
546 delete_error_count += 1 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
547 env.log.warning("Error removing stray log levels file %s: %s", filename, e) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
548 |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
549 env.log.info("Deleted %d stray log levels (%d errors)", delete_count, delete_error_count) |
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
550 |
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
551 def recreate_rule_with_int_id(env, db): |
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
552 """Recreates the bitten_rule table with an integer id column rather than a text one.""" |
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
553 cursor = db.cursor() |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
554 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
555 rule_schema_v9 = Table('bitten_rule', key=('id', 'propname'))[ |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
556 Column('id', type='int'), Column('propname'), Column('pattern'), |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
557 Column('orderno', type='int') |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
558 ] |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
559 |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
560 env.log.info("Migrating bitten_rule table to integer ids") |
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
561 connector, _ = DatabaseManager(env)._get_connector() |
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
562 |
841
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
563 cursor.execute("CREATE TEMPORARY TABLE old_rule_v9 AS SELECT * FROM bitten_rule") |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
564 cursor.execute("DROP TABLE bitten_rule") |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
565 for stmt in connector.to_sql(rule_schema_v9): |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
566 cursor.execute(stmt) |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
567 cursor.execute("INSERT INTO bitten_rule (id,propname,pattern,orderno) " |
2ea7bb126a96
Remove references to bitten.model from bitten.upgrades. See #629 for detailed explanation.
hodgestar
parents:
832
diff
changeset
|
568 "SELECT %s,propname,pattern,orderno FROM old_rule_v9" % db.cast('id', 'int')) |
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
569 |
650
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
570 def add_config_platform_rev_index_to_build(env, db): |
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
571 """Adds a unique index on (config, platform, rev) to the bitten_build table. |
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
572 Also drops the old index on bitten_build that serves no real purpose anymore.""" |
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
573 # check for existing duplicates |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
574 duplicates_cursor = db.cursor() |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
575 build_cursor = db.cursor() |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
576 |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
577 duplicates_cursor.execute("SELECT config, rev, platform FROM bitten_build GROUP BY config, rev, platform HAVING COUNT(config) > 1") |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
578 duplicates_exist = False |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
579 for config, rev, platform in duplicates_cursor.fetchall(): |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
580 if not duplicates_exist: |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
581 duplicates_exist = True |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
582 print "\nConfig Name, Revision, Platform :: [<list of build ids>]" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
583 print "--------------------------------------------------------" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
584 |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
585 build_cursor.execute("SELECT id FROM bitten_build WHERE config='%s' AND rev='%s' AND platform='%s'" % (config, rev, platform)) |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
586 build_ids = [row[0] for row in build_cursor.fetchall()] |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
587 print "%s, %s, %s :: %s" % (config, rev, platform, build_ids) |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
588 |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
589 if duplicates_exist: |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
590 print "--------------------------------------------------------\n" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
591 print "Duplicate builds found. You can remove the builds you don't want to" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
592 print "keep by using this one-line command:\n" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
593 print "$ python -c \"from bitten.model import Build; from trac.env import Environment; " \ |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
594 "Build.fetch(Environment('%s'), <buildid>).delete()\"" % env.path |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
595 print "\n...where <buildid> is the id of the build to remove.\n" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
596 print "Upgrades cannot be performed until conflicts are resolved." |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
597 print "The upgrade script will now exit with an error:\n" |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
598 |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
599 duplicates_cursor.close() |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
600 build_cursor.close() |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
601 |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
602 if not duplicates_exist: |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
603 cursor = db.cursor() |
846
ee95915922dd
Add checks that (PostgreSQL) sequences have ended up in the correct state at the end of the upgrade scripts. Test currently fails on PostgreSQL as a result of #632.
hodgestar
parents:
841
diff
changeset
|
604 scheme = parse_scheme(env) |
708 | 605 if scheme == "mysql": |
606 # 111 = 333 / len(columns in index) -- this is the Trac default | |
607 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config(111), rev(111), platform)") | |
608 else: | |
609 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config,rev,platform)") | |
610 drop_index(env, db, 'bitten_build', 'bitten_build_config_rev_slave_idx') | |
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
611 else: |
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
612 raise TracError('') |
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
613 |
708 | 614 def fix_sequences(env, db): |
615 """Fixes any auto increment sequences that might have been left in an inconsistent state. | |
616 | |
617 Upgrade scripts for schema versions > 10 should handle sequence updates correctly themselves. | |
618 """ | |
619 update_sequence(env, db, 'bitten_build', 'id') | |
620 update_sequence(env, db, 'bitten_log', 'id') | |
621 update_sequence(env, db, 'bitten_platform', 'id') | |
622 update_sequence(env, db, 'bitten_report', 'id') | |
623 | |
624 | |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
625 map = { |
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
626 2: [add_log_table], |
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
627 3: [add_recipe_to_config], |
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
628 4: [add_config_to_reports], |
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
629 5: [add_order_to_log, add_report_tables, xmldb_to_db], |
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
630 6: [normalize_file_paths, fixup_generators], |
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
631 7: [add_error_table], |
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
632 8: [add_filename_to_logs,migrate_logs_to_files], |
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
633 9: [recreate_rule_with_int_id], |
708 | 634 10: [add_config_platform_rev_index_to_build, fix_sequences], |
737
5f95a6f38490
Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary).
hodgestar
parents:
735
diff
changeset
|
635 11: [fix_log_levels_misnaming, remove_stray_log_levels_files], |
762
5f0cfee44540
Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go.
wbell
parents:
737
diff
changeset
|
636 12: [add_last_activity_to_build], |
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
637 } |