Mercurial > bitten > bitten-test
annotate bitten/upgrades.py @ 861:a08acf72a846 0.6.x
0.6dev: Merged [r938] from trunk.
| author | hodgestar |
|---|---|
| date | Sun, 17 Oct 2010 23:24:07 +0000 |
| parents | 3d4f61044b42 |
| children | 467e993df52d |
| rev | line source |
|---|---|
| 379 | 1 # -*- coding: utf-8 -*- |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
2 # |
| 833 | 3 # Copyright (C) 2007-2010 Edgewall Software |
|
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
379
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
| 163 | 5 # All rights reserved. |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
6 # |
| 163 | 7 # This software is licensed as described in the file COPYING, which |
| 8 # you should have received as part of this distribution. The terms | |
|
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
379
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
10 |
| 316 | 11 """Automated upgrades for the Bitten database tables, and other data stored |
| 776 | 12 in the Trac environment. |
| 13 | |
| 14 **Do not import and call directly!**""" | |
| 316 | 15 |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
16 import os |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
17 import sys |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
18 |
|
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
19 from trac.core import TracError |
| 842 | 20 from trac.db import DatabaseManager, Table, Column, Index |
|
519
384e59137bf8
Support unicode by converting everything to UTF-8 on write and back to unicode on read - should fix #369
dfraser
parents:
518
diff
changeset
|
21 from trac.util.text import to_unicode |
|
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
22 import codecs |
| 320 | 23 |
|
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
24 __docformat__ = 'restructuredtext en' |
|
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
410
diff
changeset
|
25 |
| 709 | 26 # database abstraction functions |
| 27 | |
| 848 | 28 def parse_scheme(env): |
| 709 | 29 """Retrieve the environment database scheme.""" |
| 30 connection_uri = DatabaseManager(env).connection_uri | |
| 31 parts = connection_uri.split(':', 1) | |
| 32 scheme = parts[0].lower() | |
| 33 return scheme | |
| 34 | |
| 35 def update_sequence(env, db, tbl, col): | |
| 36 """Update a sequence associated with an autoincrement column.""" | |
| 37 # Hopefully Trac will eventually implement its own version | |
| 38 # of this function. | |
| 848 | 39 scheme = parse_scheme(env) |
| 709 | 40 if scheme == "postgres": |
| 41 seq = '%s_%s_seq' % (tbl, col) | |
| 42 cursor = db.cursor() | |
| 43 cursor.execute("SELECT setval('%s', (SELECT MAX(%s) FROM %s))" | |
| 44 % (seq, col, tbl)) | |
| 45 | |
| 46 def drop_index(env, db, tbl, idx): | |
| 47 """Drop an index associated with a table.""" | |
| 48 # Hopefully Trac will eventually implement its own version | |
| 49 # of this function. | |
| 848 | 50 scheme = parse_scheme(env) |
| 709 | 51 cursor = db.cursor() |
| 52 if scheme == "mysql": | |
| 53 cursor.execute("DROP INDEX %s ON %s" % (idx, tbl)) | |
| 54 else: | |
| 55 cursor.execute("DROP INDEX %s" % (idx,)) | |
| 56 | |
| 57 # upgrade scripts | |
| 58 | |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
59 def add_log_table(env, db): |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
60 """Add a table for storing the builds logs.""" |
| 709 | 61 INFO_LEVEL = 'I' |
| 62 | |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
63 cursor = db.cursor() |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
64 |
| 709 | 65 build_log_schema_v3 = [ |
| 66 Table('bitten_log', key='id')[ | |
| 67 Column('id', auto_increment=True), Column('build', type='int'), | |
| 68 Column('step'), Column('type') | |
| 69 ], | |
| 70 Table('bitten_log_message', key=('log', 'line'))[ | |
| 71 Column('log', type='int'), Column('line', type='int'), | |
| 72 Column('level', size=1), Column('message') | |
| 73 ] | |
| 74 ] | |
| 75 | |
| 76 build_step_schema_v3 = [ | |
| 77 Table('bitten_step', key=('build', 'name'))[ | |
| 78 Column('build', type='int'), Column('name'), Column('description'), | |
| 79 Column('status', size=1), Column('started', type='int'), | |
| 80 Column('stopped', type='int') | |
| 81 ] | |
| 82 ] | |
| 83 | |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
84 connector, _ = DatabaseManager(env)._get_connector() |
| 709 | 85 for table in build_log_schema_v3: |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
86 for stmt in connector.to_sql(table): |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
87 cursor.execute(stmt) |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
88 |
| 709 | 89 update_cursor = db.cursor() |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
90 cursor.execute("SELECT build,name,log FROM bitten_step " |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
91 "WHERE log IS NOT NULL") |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
92 for build, step, log in cursor: |
| 709 | 93 update_cursor.execute("INSERT INTO bitten_log (build, step) " |
| 94 "VALUES (%s,%s)", (build, step)) | |
| 95 log_id = db.get_last_id(update_cursor, 'bitten_log') | |
| 96 messages = [(log_id, line, INFO_LEVEL, msg) | |
| 97 for line, msg in enumerate(log.splitlines())] | |
| 98 update_cursor.executemany("INSERT INTO bitten_log_message (log, line, level, message) " | |
| 99 "VALUES (%s, %s, %s, %s)", messages) | |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
100 |
|
524
c022478bc111
Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370
dfraser
parents:
521
diff
changeset
|
101 cursor.execute("CREATE TEMPORARY TABLE old_step AS SELECT * FROM bitten_step") |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
102 cursor.execute("DROP TABLE bitten_step") |
| 709 | 103 for table in build_step_schema_v3: |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
104 for stmt in connector.to_sql(table): |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
105 cursor.execute(stmt) |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
106 cursor.execute("INSERT INTO bitten_step (build,name,description,status," |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
107 "started,stopped) SELECT build,name,description,status," |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
108 "started,stopped FROM old_step") |
|
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
109 |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
110 def add_recipe_to_config(env, db): |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
111 """Add a column for storing the build recipe to the build configuration |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
112 table.""" |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
113 cursor = db.cursor() |
|
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
114 |
| 842 | 115 build_config_schema_v3 = Table('bitten_config', key='name')[ |
| 116 Column('name'), Column('path'), Column('active', type='int'), | |
| 117 Column('recipe'), Column('min_rev'), Column('max_rev'), | |
| 118 Column('label'), Column('description') | |
| 119 ] | |
| 120 | |
| 121 cursor.execute("CREATE TEMPORARY TABLE old_config_v2 AS " | |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
122 "SELECT * FROM bitten_config") |
|
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
123 cursor.execute("DROP TABLE bitten_config") |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
124 |
|
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
125 connector, _ = DatabaseManager(env)._get_connector() |
| 842 | 126 for stmt in connector.to_sql(build_config_schema_v3): |
| 127 cursor.execute(stmt) | |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
128 |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
129 cursor.execute("INSERT INTO bitten_config (name,path,active,recipe,min_rev," |
|
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
130 "max_rev,label,description) SELECT name,path,0,'',NULL," |
| 842 | 131 "NULL,label,description FROM old_config_v2") |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
132 |
| 763 | 133 def add_last_activity_to_build(env, db): |
| 134 """Add a column for storing the last activity to the build table.""" | |
| 135 cursor = db.cursor() | |
| 136 | |
| 137 build_table_schema_v12 = Table('bitten_build', key='id')[ | |
| 138 Column('id', auto_increment=True), Column('config'), Column('rev'), | |
| 139 Column('rev_time', type='int'), Column('platform', type='int'), | |
| 140 Column('slave'), Column('started', type='int'), | |
| 141 Column('stopped', type='int'), Column('status', size=1), | |
| 142 Column('last_activity', type='int'), | |
| 143 Index(['config', 'rev', 'platform'], unique=True) | |
| 144 ] | |
| 145 | |
| 146 cursor.execute("CREATE TEMPORARY TABLE old_build_v11 AS " | |
| 147 "SELECT * FROM bitten_build") | |
| 148 cursor.execute("DROP TABLE bitten_build") | |
| 149 | |
| 150 connector, _ = DatabaseManager(env)._get_connector() | |
| 151 for stmt in connector.to_sql(build_table_schema_v12): | |
| 152 cursor.execute(stmt) | |
| 153 | |
| 154 # it's safe to make the last activity the stop time of the build | |
| 789 | 155 cursor.execute("INSERT INTO bitten_build (id,config,rev,rev_time,platform," |
| 763 | 156 "slave,started,stopped,last_activity,status) " |
| 789 | 157 "SELECT id,config,rev,rev_time,platform," |
| 763 | 158 "slave,started,stopped,stopped,status FROM old_build_v11") |
| 159 | |
| 848 | 160 update_sequence(env, db, 'bitten_build', 'id') |
| 161 | |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
162 def add_config_to_reports(env, db): |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
163 """Add the name of the build configuration as metadata to report documents |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
164 stored in the BDB XML database.""" |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
165 try: |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
166 from bsddb3 import db as bdb |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
167 import dbxml |
| 219 | 168 except ImportError: |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
169 return |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
170 |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
171 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml') |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
172 if not os.path.isfile(dbfile): |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
173 return |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
174 |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
175 dbenv = bdb.DBEnv() |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
176 dbenv.open(os.path.dirname(dbfile), |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
177 bdb.DB_CREATE | bdb.DB_INIT_LOCK | bdb.DB_INIT_LOG | |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
178 bdb.DB_INIT_MPOOL | bdb.DB_INIT_TXN, 0) |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
179 |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
180 mgr = dbxml.XmlManager(dbenv, 0) |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
181 xtn = mgr.createTransaction() |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
182 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL) |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
183 uc = mgr.createUpdateContext() |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
184 |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
185 container.addIndex(xtn, '', 'config', 'node-metadata-equality-string', uc) |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
186 |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
187 qc = mgr.createQueryContext() |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
188 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc): |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
189 doc = value.asDocument() |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
190 metaval = dbxml.XmlValue() |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
191 if doc.getMetaData('', 'build', metaval): |
|
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
192 build_id = int(metaval.asNumber()) |
| 842 | 193 |
| 194 cursor = db.cursor() | |
| 195 cursor.execute("SELECT config FROM bitten_build WHERE id=%s", (build_id,)) | |
| 196 row = cursor.fetchone() | |
| 197 | |
| 198 if row: | |
| 199 doc.setMetaData('', 'config', dbxml.XmlValue(row[0])) | |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
200 container.updateDocument(xtn, doc, uc) |
|
175
f7c2f112afe6
Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it.
cmlenz
parents:
174
diff
changeset
|
201 else: |
|
f7c2f112afe6
Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it.
cmlenz
parents:
174
diff
changeset
|
202 # an orphaned report, for whatever reason... just remove it |
|
200
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
203 container.deleteDocument(xtn, doc, uc) |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
204 |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
205 xtn.commit() |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
206 container.close() |
|
692924ffed80
Changes to the BDB XML report store to support transactions. Closes #47.
cmlenz
parents:
196
diff
changeset
|
207 dbenv.close(0) |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
208 |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
209 def add_order_to_log(env, db): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
210 """Add order column to log table to make sure that build logs are displayed |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
211 in the order they were generated.""" |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
212 cursor = db.cursor() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
213 |
| 842 | 214 log_table_schema_v6 = Table('bitten_log', key='id')[ |
| 215 Column('id', auto_increment=True), Column('build', type='int'), | |
| 216 Column('step'), Column('generator'), Column('orderno', type='int'), | |
| 217 Index(['build', 'step']) | |
| 218 ] | |
| 219 | |
| 709 | 220 cursor.execute("CREATE TEMPORARY TABLE old_log_v5 AS " |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
221 "SELECT * FROM bitten_log") |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
222 cursor.execute("DROP TABLE bitten_log") |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
223 |
|
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
224 connector, _ = DatabaseManager(env)._get_connector() |
| 842 | 225 for stmt in connector.to_sql(log_table_schema_v6): |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
226 cursor.execute(stmt) |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
227 |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
228 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno) " |
| 709 | 229 "SELECT id,build,step,type,0 FROM old_log_v5") |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
230 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
231 def add_report_tables(env, db): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
232 """Add database tables for report storage.""" |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
233 cursor = db.cursor() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
234 |
| 842 | 235 report_schema_v6 = Table('bitten_report', key='id')[ |
| 236 Column('id', auto_increment=True), Column('build', type='int'), | |
| 237 Column('step'), Column('category'), Column('generator'), | |
| 238 Index(['build', 'step', 'category']) | |
| 239 ] | |
| 240 report_item_schema_v6 = Table('bitten_report_item', key=('report', 'item', 'name'))[ | |
| 241 Column('report', type='int'), Column('item', type='int'), | |
| 242 Column('name'), Column('value') | |
| 243 ] | |
| 244 | |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
245 connector, _ = DatabaseManager(env)._get_connector() |
| 842 | 246 for table in [report_schema_v6, report_item_schema_v6]: |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
247 for stmt in connector.to_sql(table): |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
248 cursor.execute(stmt) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
249 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
250 def xmldb_to_db(env, db): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
251 """Migrate report data from Berkeley DB XML to SQL database. |
| 763 | 252 |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
253 Depending on the number of reports stored, this might take rather long. |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
254 After the upgrade is done, the bitten.dbxml file (and any BDB XML log files) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
255 may be deleted. BDB XML is no longer used by Bitten. |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
256 """ |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
257 from bitten.util import xmlio |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
258 try: |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
259 from bsddb3 import db as bdb |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
260 import dbxml |
| 219 | 261 except ImportError: |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
262 return |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
263 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
264 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml') |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
265 if not os.path.isfile(dbfile): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
266 return |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
267 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
268 dbenv = bdb.DBEnv() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
269 dbenv.open(os.path.dirname(dbfile), |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
270 bdb.DB_CREATE | bdb.DB_INIT_LOCK | bdb.DB_INIT_LOG | |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
271 bdb.DB_INIT_MPOOL | bdb.DB_INIT_TXN, 0) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
272 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
273 mgr = dbxml.XmlManager(dbenv, 0) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
274 xtn = mgr.createTransaction() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
275 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
276 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
277 def get_pylint_items(xml): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
278 for problems_elem in xml.children('problems'): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
279 for problem_elem in problems_elem.children('problem'): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
280 item = {'type': 'problem'} |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
281 item.update(problem_elem.attr) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
282 yield item |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
283 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
284 def get_trace_items(xml): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
285 for cov_elem in xml.children('coverage'): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
286 item = {'type': 'coverage', 'name': cov_elem.attr['module'], |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
287 'file': cov_elem.attr['file'], |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
288 'percentage': cov_elem.attr['percentage']} |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
289 lines = 0 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
290 line_hits = [] |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
291 for line_elem in cov_elem.children('line'): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
292 lines += 1 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
293 line_hits.append(line_elem.attr['hits']) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
294 item['lines'] = lines |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
295 item['line_hits'] = ' '.join(line_hits) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
296 yield item |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
297 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
298 def get_unittest_items(xml): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
299 for test_elem in xml.children('test'): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
300 item = {'type': 'test'} |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
301 item.update(test_elem.attr) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
302 for child_elem in test_elem.children(): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
303 item[child_elem.name] = child_elem.gettext() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
304 yield item |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
305 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
306 qc = mgr.createQueryContext() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
307 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc, 0): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
308 doc = value.asDocument() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
309 metaval = dbxml.XmlValue() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
310 build, step = None, None |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
311 if doc.getMetaData('', 'build', metaval): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
312 build = metaval.asNumber() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
313 if doc.getMetaData('', 'step', metaval): |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
314 step = metaval.asString() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
315 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
316 report_types = {'pylint': ('lint', get_pylint_items), |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
317 'trace': ('coverage', get_trace_items), |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
318 'unittest': ('test', get_unittest_items)} |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
319 xml = xmlio.parse(value.asString()) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
320 report_type = xml.attr['type'] |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
321 category, get_items = report_types[report_type] |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
322 sys.stderr.write('.') |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
323 sys.stderr.flush() |
| 842 | 324 |
| 325 items = list(get_items(xml)) | |
| 326 | |
| 327 cursor = db.cursor() | |
| 328 cursor.execute("SELECT bitten_report.id FROM bitten_report " | |
| 329 "WHERE build=%s AND step=%s AND category=%s", | |
| 330 (build, step, category)) | |
| 331 rows = cursor.fetchall() | |
| 332 if rows: | |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
333 # Duplicate report, skip |
| 842 | 334 continue |
| 335 | |
| 336 cursor.execute("INSERT INTO bitten_report " | |
| 337 "(build,step,category,generator) VALUES (%s,%s,%s,%s)", | |
| 338 (build, step, category, report_type)) | |
| 339 id = db.get_last_id(cursor, 'bitten_report') | |
| 340 | |
| 341 for idx, item in enumerate(items): | |
| 342 cursor.executemany("INSERT INTO bitten_report_item " | |
| 343 "(report,item,name,value) VALUES (%s,%s,%s,%s)", | |
| 344 [(id, idx, key, value) for key, value | |
| 345 in item.items()]) | |
| 346 | |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
347 sys.stderr.write('\n') |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
348 sys.stderr.flush() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
349 |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
350 xtn.abort() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
351 container.close() |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
352 dbenv.close(0) |
|
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
353 |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
354 def normalize_file_paths(env, db): |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
355 """Normalize the file separator in file names in reports.""" |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
356 cursor = db.cursor() |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
357 cursor.execute("SELECT report,item,value FROM bitten_report_item " |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
358 "WHERE name='file'") |
|
223
067bde207c23
Fix upgrade for installs with no stored reports. Thanks to Matt Good for catching this.
cmlenz
parents:
219
diff
changeset
|
359 rows = cursor.fetchall() or [] |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
360 for report, item, value in rows: |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
361 if '\\' in value: |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
362 cursor.execute("UPDATE bitten_report_item SET value=%s " |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
363 "WHERE report=%s AND item=%s AND name='file'", |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
364 (value.replace('\\', '/'), report, item)) |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
365 |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
366 def fixup_generators(env, db): |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
367 """Upgrade the identifiers for the recipe commands that generated log |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
368 messages and report data.""" |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
369 |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
370 mapping = { |
| 684 | 371 'pipe': 'http://bitten.edgewall.org/tools/sh#pipe', |
| 372 'make': 'http://bitten.edgewall.org/tools/c#make', | |
| 373 'distutils': 'http://bitten.edgewall.org/tools/python#distutils', | |
| 374 'exec_': 'http://bitten.edgewall.org/tools/python#exec' # Ambigious | |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
375 } |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
376 cursor = db.cursor() |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
377 cursor.execute("SELECT id,generator FROM bitten_log " |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
378 "WHERE generator IN (%s)" |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
379 % ','.join([repr(key) for key in mapping.keys()])) |
| 219 | 380 for log_id, generator in cursor: |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
381 cursor.execute("UPDATE bitten_log SET generator=%s " |
| 219 | 382 "WHERE id=%s", (mapping[generator], log_id)) |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
383 |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
384 mapping = { |
| 684 | 385 'unittest': 'http://bitten.edgewall.org/tools/python#unittest', |
| 386 'trace': 'http://bitten.edgewall.org/tools/python#trace', | |
| 387 'pylint': 'http://bitten.edgewall.org/tools/python#pylint' | |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
388 } |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
389 cursor.execute("SELECT id,generator FROM bitten_report " |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
390 "WHERE generator IN (%s)" |
|
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
391 % ','.join([repr(key) for key in mapping.keys()])) |
| 219 | 392 for report_id, generator in cursor: |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
393 cursor.execute("UPDATE bitten_report SET generator=%s " |
| 219 | 394 "WHERE id=%s", (mapping[generator], report_id)) |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
395 |
|
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
396 def add_error_table(env, db): |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
397 """Add the bitten_error table for recording step failure reasons.""" |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
398 table = Table('bitten_error', key=('build', 'step', 'orderno'))[ |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
399 Column('build', type='int'), Column('step'), Column('message'), |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
400 Column('orderno', type='int') |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
401 ] |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
402 cursor = db.cursor() |
|
410
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
403 |
|
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
404 connector, _ = DatabaseManager(env)._get_connector() |
|
7930cdd83d13
More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac.
cmlenz
parents:
408
diff
changeset
|
405 for stmt in connector.to_sql(table): |
|
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
406 cursor.execute(stmt) |
|
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
407 |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
408 def add_filename_to_logs(env, db): |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
409 """Add filename column to log table to save where log files are stored.""" |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
410 cursor = db.cursor() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
411 |
| 842 | 412 build_log_schema_v9 = Table('bitten_log', key='id')[ |
| 413 Column('id', auto_increment=True), Column('build', type='int'), | |
| 414 Column('step'), Column('generator'), Column('orderno', type='int'), | |
| 415 Column('filename'), | |
| 416 Index(['build', 'step']) | |
| 417 ] | |
| 418 | |
| 709 | 419 cursor.execute("CREATE TEMPORARY TABLE old_log_v8 AS " |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
420 "SELECT * FROM bitten_log") |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
421 cursor.execute("DROP TABLE bitten_log") |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
422 |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
423 connector, _ = DatabaseManager(env)._get_connector() |
| 842 | 424 for stmt in connector.to_sql(build_log_schema_v9): |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
425 cursor.execute(stmt) |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
426 |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
427 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno,filename) " |
| 709 | 428 "SELECT id,build,step,generator,orderno,'' FROM old_log_v8") |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
429 |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
430 def migrate_logs_to_files(env, db): |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
431 """Migrates logs that are stored in the bitten_log_messages table into files.""" |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
432 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
433 if not os.path.isabs(logs_dir): |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
434 logs_dir = os.path.join(env.path, logs_dir) |
| 861 | 435 |
| 436 if os.path.exists(logs_dir): | |
| 437 print "Bitten log folder %r already exists" % (logs_dir,) | |
| 438 print "Upgrade cannot be performed until the existing folder is moved." | |
| 439 print "The upgrade script will now exit with an error:\n" | |
| 440 raise TracError("") | |
| 441 | |
| 442 os.makedirs(logs_dir) | |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
443 |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
444 cursor = db.cursor() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
445 message_cursor = db.cursor() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
446 update_cursor = db.cursor() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
447 cursor.execute("SELECT id FROM bitten_log") |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
448 for log_id, in cursor.fetchall(): |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
449 filename = "%s.log" % (log_id,) |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
450 message_cursor.execute("SELECT message, level FROM bitten_log_message WHERE log=%s ORDER BY line", (log_id,)) |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
451 full_filename = os.path.join(logs_dir, filename) |
|
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
452 message_file = codecs.open(full_filename, "wb", "UTF-8") |
| 736 | 453 # Note: the original version of this code erroneously wrote to filename + ".level" instead of ".levels", producing unused level files |
| 454 level_file = codecs.open(full_filename + '.levels', "wb", "UTF-8") | |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
455 for message, level in message_cursor.fetchall() or []: |
|
521
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
456 message_file.write(to_unicode(message) + "\n") |
|
b661ea254972
Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused):
dfraser
parents:
519
diff
changeset
|
457 level_file.write(to_unicode(level) + "\n") |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
458 message_file.close() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
459 level_file.close() |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
460 update_cursor.execute("UPDATE bitten_log SET filename=%s WHERE id=%s", (filename, log_id)) |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
461 env.log.info("Migrated log %s", log_id) |
| 518 | 462 env.log.warning("Logs have been migrated from the database to files in %s. " |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
463 "Ensure permissions are set correctly on this file. " |
|
619
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
464 "Since we presume that the migration worked correctly, " |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
465 "we are now dropping the bitten_log_message table in the database (aren't you glad you backed up)", logs_dir) |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
466 cursor.close() |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
467 cursor = db.cursor() |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
468 cursor.execute("DROP TABLE bitten_log_message") |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
469 cursor.close() |
|
3dd1b2d4b2a7
After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329
dfraser
parents:
565
diff
changeset
|
470 env.log.warning("We have dropped the bitten_log_message table - you may want to vaccuum/compress your database to save space") |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
471 |
| 736 | 472 def fix_log_levels_misnaming(env, db): |
| 776 | 473 """Renames or removes \*.log.level files created by older versions of migrate_logs_to_files.""" |
| 736 | 474 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
| 475 if not os.path.isabs(logs_dir): | |
| 476 logs_dir = os.path.join(env.path, logs_dir) | |
| 477 if not os.path.isdir(logs_dir): | |
| 478 return | |
| 479 | |
| 480 rename_count = 0 | |
| 481 rename_error_count = 0 | |
| 482 delete_count = 0 | |
| 483 delete_error_count = 0 | |
| 484 | |
| 485 for wrong_filename in os.listdir(logs_dir): | |
| 486 if not wrong_filename.endswith('.log.level'): | |
| 487 continue | |
| 488 | |
| 489 log_filename = os.path.splitext(wrong_filename)[0] | |
| 490 right_filename = log_filename + '.levels' | |
| 491 full_log_filename = os.path.join(logs_dir, log_filename) | |
| 492 full_wrong_filename = os.path.join(logs_dir, wrong_filename) | |
| 493 full_right_filename = os.path.join(logs_dir, right_filename) | |
| 494 | |
| 495 if not os.path.exists(full_log_filename): | |
| 496 try: | |
| 497 os.remove(full_wrong_filename) | |
| 498 delete_count += 1 | |
| 499 env.log.info("Deleted stray log level file %s", wrong_filename) | |
| 500 except Exception, e: | |
| 501 delete_error_count += 1 | |
| 502 env.log.warning("Error removing stray log level file %s: %s", wrong_filename, e) | |
| 503 else: | |
| 504 if os.path.exists(full_right_filename): | |
| 505 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: new filename already exists", | |
| 506 full_wrong_filename, full_right_filename) | |
| 507 rename_error_count += 1 | |
| 508 continue | |
| 509 try: | |
| 510 os.rename(full_wrong_filename, full_right_filename) | |
| 511 rename_count += 1 | |
| 512 env.log.info("Renamed incorrectly named log level file %s to %s", wrong_filename, right_filename) | |
| 513 except Exception, e: | |
| 514 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: %s", full_wrong_filename, full_right_filename, e) | |
| 515 rename_error_count += 1 | |
| 516 | |
| 517 env.log.info("Renamed %d incorrectly named log level files from previous migrate (%d errors)", rename_count, rename_error_count) | |
| 518 env.log.info("Deleted %d stray log level (%d errors)", delete_count, delete_error_count) | |
| 519 | |
| 738 | 520 def remove_stray_log_levels_files(env, db): |
| 776 | 521 """Remove \*.log.levels files without a matching \*.log file (old Bitten |
| 522 versions did not delete .log.levels files when builds were deleted)""" | |
| 738 | 523 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten") |
| 524 if not os.path.isabs(logs_dir): | |
| 525 logs_dir = os.path.join(env.path, logs_dir) | |
| 526 if not os.path.isdir(logs_dir): | |
| 527 return | |
| 528 | |
| 529 delete_count = 0 | |
| 530 delete_error_count = 0 | |
| 531 | |
| 532 for filename in os.listdir(logs_dir): | |
| 533 if not filename.endswith('.log.levels'): | |
| 534 continue | |
| 535 | |
| 536 log_filename = os.path.splitext(filename)[0] | |
| 537 full_log_filename = os.path.join(logs_dir, log_filename) | |
| 538 full_filename = os.path.join(logs_dir, filename) | |
| 539 | |
| 540 if not os.path.exists(full_log_filename): | |
| 541 try: | |
| 542 os.remove(full_filename) | |
| 543 delete_count += 1 | |
| 544 env.log.info("Deleted stray log levels file %s", filename) | |
| 545 except Exception, e: | |
| 546 delete_error_count += 1 | |
| 547 env.log.warning("Error removing stray log levels file %s: %s", filename, e) | |
| 548 | |
| 549 env.log.info("Deleted %d stray log levels (%d errors)", delete_count, delete_error_count) | |
| 550 | |
|
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
551 def recreate_rule_with_int_id(env, db): |
|
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
552 """Recreates the bitten_rule table with an integer id column rather than a text one.""" |
|
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
553 cursor = db.cursor() |
| 842 | 554 |
| 555 rule_schema_v9 = Table('bitten_rule', key=('id', 'propname'))[ | |
| 556 Column('id', type='int'), Column('propname'), Column('pattern'), | |
| 557 Column('orderno', type='int') | |
| 558 ] | |
| 559 | |
| 560 env.log.info("Migrating bitten_rule table to integer ids") | |
|
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
561 connector, _ = DatabaseManager(env)._get_connector() |
|
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
562 |
| 842 | 563 cursor.execute("CREATE TEMPORARY TABLE old_rule_v9 AS SELECT * FROM bitten_rule") |
| 564 cursor.execute("DROP TABLE bitten_rule") | |
| 565 for stmt in connector.to_sql(rule_schema_v9): | |
| 566 cursor.execute(stmt) | |
| 567 cursor.execute("INSERT INTO bitten_rule (id,propname,pattern,orderno) " | |
| 568 "SELECT %s,propname,pattern,orderno FROM old_rule_v9" % db.cast('id', 'int')) | |
|
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
569 |
|
650
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
570 def add_config_platform_rev_index_to_build(env, db): |
|
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
571 """Adds a unique index on (config, platform, rev) to the bitten_build table. |
|
b1a50f2d92eb
0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included.
osimons
parents:
619
diff
changeset
|
572 Also drops the old index on bitten_build that serves no real purpose anymore.""" |
|
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
573 # check for existing duplicates |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
574 duplicates_cursor = db.cursor() |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
575 build_cursor = db.cursor() |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
576 |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
577 duplicates_cursor.execute("SELECT config, rev, platform FROM bitten_build GROUP BY config, rev, platform HAVING COUNT(config) > 1") |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
578 duplicates_exist = False |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
579 for config, rev, platform in duplicates_cursor.fetchall(): |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
580 if not duplicates_exist: |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
581 duplicates_exist = True |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
582 print "\nConfig Name, Revision, Platform :: [<list of build ids>]" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
583 print "--------------------------------------------------------" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
584 |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
585 build_cursor.execute("SELECT id FROM bitten_build WHERE config='%s' AND rev='%s' AND platform='%s'" % (config, rev, platform)) |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
586 build_ids = [row[0] for row in build_cursor.fetchall()] |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
587 print "%s, %s, %s :: %s" % (config, rev, platform, build_ids) |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
588 |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
589 if duplicates_exist: |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
590 print "--------------------------------------------------------\n" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
591 print "Duplicate builds found. You can remove the builds you don't want to" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
592 print "keep by using this one-line command:\n" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
593 print "$ python -c \"from bitten.model import Build; from trac.env import Environment; " \ |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
594 "Build.fetch(Environment('%s'), <buildid>).delete()\"" % env.path |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
595 print "\n...where <buildid> is the id of the build to remove.\n" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
596 print "Upgrades cannot be performed until conflicts are resolved." |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
597 print "The upgrade script will now exit with an error:\n" |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
598 |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
599 duplicates_cursor.close() |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
600 build_cursor.close() |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
601 |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
602 if not duplicates_exist: |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
603 cursor = db.cursor() |
| 848 | 604 scheme = parse_scheme(env) |
| 709 | 605 if scheme == "mysql": |
| 606 # 111 = 333 / len(columns in index) -- this is the Trac default | |
| 607 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config(111), rev(111), platform)") | |
| 608 else: | |
| 609 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config,rev,platform)") | |
| 610 drop_index(env, db, 'bitten_build', 'bitten_build_config_rev_slave_idx') | |
|
656
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
611 else: |
|
5467873fc0dc
0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually.
osimons
parents:
650
diff
changeset
|
612 raise TracError('') |
|
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
613 |
| 709 | 614 def fix_sequences(env, db): |
| 615 """Fixes any auto increment sequences that might have been left in an inconsistent state. | |
| 616 | |
| 617 Upgrade scripts for schema versions > 10 should handle sequence updates correctly themselves. | |
| 618 """ | |
| 619 update_sequence(env, db, 'bitten_build', 'id') | |
| 620 update_sequence(env, db, 'bitten_log', 'id') | |
| 621 update_sequence(env, db, 'bitten_platform', 'id') | |
| 622 update_sequence(env, db, 'bitten_report', 'id') | |
| 623 | |
| 624 | |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
625 map = { |
|
147
395b67aa072e
Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41.
cmlenz
parents:
112
diff
changeset
|
626 2: [add_log_table], |
|
174
79c61c26a4e1
* Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics.
cmlenz
parents:
163
diff
changeset
|
627 3: [add_recipe_to_config], |
|
203
e6ddca1e5712
Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL).
cmlenz
parents:
200
diff
changeset
|
628 4: [add_config_to_reports], |
|
213
25f84dd9f159
* Refactoring of build recipes, the file format has changed slightly:
cmlenz
parents:
203
diff
changeset
|
629 5: [add_order_to_log, add_report_tables, xmldb_to_db], |
|
245
a22ec8fce6c9
Store the reason(s) for build step failure in the database.
cmlenz
parents:
223
diff
changeset
|
630 6: [normalize_file_paths, fixup_generators], |
|
516
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
631 7: [add_error_table], |
|
2f3b7c17d3c3
Switch to storing log messages in files rather than in database rows:
dfraser
parents:
411
diff
changeset
|
632 8: [add_filename_to_logs,migrate_logs_to_files], |
|
565
78109beea395
Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later.
wbell
parents:
524
diff
changeset
|
633 9: [recreate_rule_with_int_id], |
| 709 | 634 10: [add_config_platform_rev_index_to_build, fix_sequences], |
| 738 | 635 11: [fix_log_levels_misnaming, remove_stray_log_levels_files], |
| 763 | 636 12: [add_last_activity_to_build], |
|
112
a38eabd4b6e1
* Store build logs in a structured way, for example to highlight messages on the error stream.
cmlenz
parents:
diff
changeset
|
637 } |