bitten/bitten-test: bitten/upgrades.py annotate

annotate bitten/upgrades.py @ 735:90903d6cc932

Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517.

author	hodgestar
date	Fri, 19 Feb 2010 11:51:46 +0000
parents	508636ffbf11
children	5f95a6f38490

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 323 diff changeset	1 # -- coding: utf-8 --
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	2 #
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	3 # Copyright (C) 2007 Edgewall Software
933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	5 # All rights reserved.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	6 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	8 # you should have received as part of this distribution. The terms
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	9 # are also available at http://bitten.edgewall.org/wiki/License.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	10
316 87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	11 """Automated upgrades for the Bitten database tables, and other data stored
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	12 in the Trac environment."""
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	13
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	14 import os
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	15 import sys
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	16
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	17 from trac.core import TracError
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	18 from trac.db import DatabaseManager
519 384e59137bf8 Support unicode by converting everything to UTF-8 on write and back to unicode on read - should fix #369 dfraser parents: 518 diff changeset	19 from trac.util.text import to_unicode
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	20 import codecs
320 a8b713254286 Fixes for compatibility with Trac trunk and 0.9.3. cmlenz parents: 316 diff changeset	21
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	22 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	23
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	24 # database abstraction functions
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	25
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	26 def _parse_scheme(env):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	27 """Retrieve the environment database scheme."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	28 connection_uri = DatabaseManager(env).connection_uri
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	29 parts = connection_uri.split(':', 1)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	30 scheme = parts[0].lower()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	31 return scheme
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	32
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	33 def update_sequence(env, db, tbl, col):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	34 """Update a sequence associated with an autoincrement column."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	35 # Hopefully Trac will eventually implement its own version
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	36 # of this function.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	37 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	38 if scheme == "postgres":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	39 seq = '%s_%s_seq' % (tbl, col)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	40 cursor = db.cursor()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	41 cursor.execute("SELECT setval('%s', (SELECT MAX(%s) FROM %s))"
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	42 % (seq, col, tbl))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	43
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	44 def drop_index(env, db, tbl, idx):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	45 """Drop an index associated with a table."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	46 # Hopefully Trac will eventually implement its own version
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	47 # of this function.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	48 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	49 cursor = db.cursor()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	50 if scheme == "mysql":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	51 cursor.execute("DROP INDEX %s ON %s" % (idx, tbl))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	52 else:
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	53 cursor.execute("DROP INDEX %s" % (idx,))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	54
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	55 # upgrade scripts
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	56
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	57 def add_log_table(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	58 """Add a table for storing the builds logs."""
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	59 from trac.db import Table, Column
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	60 INFO_LEVEL = 'I'
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	61
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	62 cursor = db.cursor()
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	63
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	64 build_log_schema_v3 = [
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	65 Table('bitten_log', key='id')[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	66 Column('id', auto_increment=True), Column('build', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	67 Column('step'), Column('type')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	68 ],
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	69 Table('bitten_log_message', key=('log', 'line'))[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	70 Column('log', type='int'), Column('line', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	71 Column('level', size=1), Column('message')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	72 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	73 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	74
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	75 build_step_schema_v3 = [
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	76 Table('bitten_step', key=('build', 'name'))[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	77 Column('build', type='int'), Column('name'), Column('description'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	78 Column('status', size=1), Column('started', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	79 Column('stopped', type='int')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	80 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	81 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	82
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	83 connector, _ = DatabaseManager(env)._get_connector()
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	84 for table in build_log_schema_v3:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	85 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	86 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	87
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	88 update_cursor = db.cursor()
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	89 cursor.execute("SELECT build,name,log FROM bitten_step "
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	90 "WHERE log IS NOT NULL")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	91 for build, step, log in cursor:
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	92 update_cursor.execute("INSERT INTO bitten_log (build, step) "
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	93 "VALUES (%s,%s)", (build, step))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	94 log_id = db.get_last_id(update_cursor, 'bitten_log')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	95 messages = [(log_id, line, INFO_LEVEL, msg)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	96 for line, msg in enumerate(log.splitlines())]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	97 update_cursor.executemany("INSERT INTO bitten_log_message (log, line, level, message) "
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	98 "VALUES (%s, %s, %s, %s)", messages)
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	99
524 c022478bc111 Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370 dfraser parents: 521 diff changeset	100 cursor.execute("CREATE TEMPORARY TABLE old_step AS SELECT * FROM bitten_step")
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	101 cursor.execute("DROP TABLE bitten_step")
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	102 for table in build_step_schema_v3:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	103 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	104 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	105 cursor.execute("INSERT INTO bitten_step (build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	106 "started,stopped) SELECT build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	107 "started,stopped FROM old_step")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	108
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	109 def add_recipe_to_config(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	110 """Add a column for storing the build recipe to the build configuration
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	111 table."""
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	112 from bitten.model import BuildConfig
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	113 cursor = db.cursor()
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	114
524 c022478bc111 Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370 dfraser parents: 521 diff changeset	115 cursor.execute("CREATE TEMPORARY TABLE old_config AS "
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	116 "SELECT * FROM bitten_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	117 cursor.execute("DROP TABLE bitten_config")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	118
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	119 connector, _ = DatabaseManager(env)._get_connector()
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	120 for table in BuildConfig._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	121 for stmt in connector.to_sql(table):
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	122 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	123
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	124 cursor.execute("INSERT INTO bitten_config (name,path,active,recipe,min_rev,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	125 "max_rev,label,description) SELECT name,path,0,'',NULL,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	126 "NULL,label,description FROM old_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	127
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	128 def add_config_to_reports(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	129 """Add the name of the build configuration as metadata to report documents
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	130 stored in the BDB XML database."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	131
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	132 from bitten.model import Build
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	133 try:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	134 from bsddb3 import db as bdb
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	135 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	136 except ImportError:
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	137 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	138
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	139 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	140 if not os.path.isfile(dbfile):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	141 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	142
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	143 dbenv = bdb.DBEnv()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	144 dbenv.open(os.path.dirname(dbfile),
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	145 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	146 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	147
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	148 mgr = dbxml.XmlManager(dbenv, 0)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	149 xtn = mgr.createTransaction()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	150 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	151 uc = mgr.createUpdateContext()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	152
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	153 container.addIndex(xtn, '', 'config', 'node-metadata-equality-string', uc)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	154
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	155 qc = mgr.createQueryContext()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	156 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc):
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	157 doc = value.asDocument()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	158 metaval = dbxml.XmlValue()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	159 if doc.getMetaData('', 'build', metaval):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	160 build_id = int(metaval.asNumber())
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	161 build = Build.fetch(env, id=build_id, db=db)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	162 if build:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	163 doc.setMetaData('', 'config', dbxml.XmlValue(build.config))
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	164 container.updateDocument(xtn, doc, uc)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	165 else:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	166 # an orphaned report, for whatever reason... just remove it
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	167 container.deleteDocument(xtn, doc, uc)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	168
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	169 xtn.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	170 container.close()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	171 dbenv.close(0)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	172
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	173 def add_order_to_log(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	174 """Add order column to log table to make sure that build logs are displayed
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	175 in the order they were generated."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	176 from bitten.model import BuildLog
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	177 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	178
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	179 cursor.execute("CREATE TEMPORARY TABLE old_log_v5 AS "
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	180 "SELECT * FROM bitten_log")
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	181 cursor.execute("DROP TABLE bitten_log")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	182
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	183 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	184 for stmt in connector.to_sql(BuildLog._schema[0]):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	185 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	186
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	187 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno) "
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	188 "SELECT id,build,step,type,0 FROM old_log_v5")
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	189
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	190 def add_report_tables(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	191 """Add database tables for report storage."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	192 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	193 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	194
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	195 connector, _ = DatabaseManager(env)._get_connector()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	196 for table in Report._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	197 for stmt in connector.to_sql(table):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	198 cursor.execute(stmt)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	199
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	200 def xmldb_to_db(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	201 """Migrate report data from Berkeley DB XML to SQL database.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	202
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	203 Depending on the number of reports stored, this might take rather long.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	204 After the upgrade is done, the bitten.dbxml file (and any BDB XML log files)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	205 may be deleted. BDB XML is no longer used by Bitten.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	206 """
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	207 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	208 from bitten.util import xmlio
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	209 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	210 from bsddb3 import db as bdb
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	211 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	212 except ImportError:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	213 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	214
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	215 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	216 if not os.path.isfile(dbfile):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	217 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	218
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	219 dbenv = bdb.DBEnv()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	220 dbenv.open(os.path.dirname(dbfile),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	221 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	222 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	223
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	224 mgr = dbxml.XmlManager(dbenv, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	225 xtn = mgr.createTransaction()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	226 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	227
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	228 def get_pylint_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	229 for problems_elem in xml.children('problems'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	230 for problem_elem in problems_elem.children('problem'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	231 item = {'type': 'problem'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	232 item.update(problem_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	233 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	234
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	235 def get_trace_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	236 for cov_elem in xml.children('coverage'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	237 item = {'type': 'coverage', 'name': cov_elem.attr['module'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	238 'file': cov_elem.attr['file'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	239 'percentage': cov_elem.attr['percentage']}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	240 lines = 0
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	241 line_hits = []
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	242 for line_elem in cov_elem.children('line'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	243 lines += 1
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	244 line_hits.append(line_elem.attr['hits'])
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	245 item['lines'] = lines
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	246 item['line_hits'] = ' '.join(line_hits)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	247 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	248
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	249 def get_unittest_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	250 for test_elem in xml.children('test'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	251 item = {'type': 'test'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	252 item.update(test_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	253 for child_elem in test_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	254 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	255 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	256
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	257 qc = mgr.createQueryContext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	258 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc, 0):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	259 doc = value.asDocument()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	260 metaval = dbxml.XmlValue()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	261 build, step = None, None
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	262 if doc.getMetaData('', 'build', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	263 build = metaval.asNumber()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	264 if doc.getMetaData('', 'step', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	265 step = metaval.asString()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	266
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	267 report_types = {'pylint': ('lint', get_pylint_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	268 'trace': ('coverage', get_trace_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	269 'unittest': ('test', get_unittest_items)}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	270 xml = xmlio.parse(value.asString())
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	271 report_type = xml.attr['type']
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	272 category, get_items = report_types[report_type]
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	273 sys.stderr.write('.')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	274 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	275 report = Report(env, build, step, category=category,
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	276 generator=report_type)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	277 report.items = list(get_items(xml))
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	278 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	279 report.insert(db=db)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	280 except AssertionError:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	281 # Duplicate report, skip
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	282 pass
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	283 sys.stderr.write('\n')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	284 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	285
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	286 xtn.abort()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	287 container.close()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	288 dbenv.close(0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	289
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	290 def normalize_file_paths(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	291 """Normalize the file separator in file names in reports."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	292 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	293 cursor.execute("SELECT report,item,value FROM bitten_report_item "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	294 "WHERE name='file'")
223 067bde207c23 Fix upgrade for installs with no stored reports. Thanks to Matt Good for catching this. cmlenz parents: 219 diff changeset	295 rows = cursor.fetchall() or []
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	296 for report, item, value in rows:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	297 if '\\' in value:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	298 cursor.execute("UPDATE bitten_report_item SET value=%s "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	299 "WHERE report=%s AND item=%s AND name='file'",
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	300 (value.replace('\\', '/'), report, item))
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	301
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	302 def fixup_generators(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	303 """Upgrade the identifiers for the recipe commands that generated log
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	304 messages and report data."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	305
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	306 mapping = {
683 0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	307 'pipe': 'http://bitten.edgewall.org/tools/sh#pipe',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	308 'make': 'http://bitten.edgewall.org/tools/c#make',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	309 'distutils': 'http://bitten.edgewall.org/tools/python#distutils',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	310 'exec_': 'http://bitten.edgewall.org/tools/python#exec' # Ambigious
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	311 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	312 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	313 cursor.execute("SELECT id,generator FROM bitten_log "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	314 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	315 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	316 for log_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	317 cursor.execute("UPDATE bitten_log SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	318 "WHERE id=%s", (mapping[generator], log_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	319
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	320 mapping = {
683 0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	321 'unittest': 'http://bitten.edgewall.org/tools/python#unittest',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	322 'trace': 'http://bitten.edgewall.org/tools/python#trace',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	323 'pylint': 'http://bitten.edgewall.org/tools/python#pylint'
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	324 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	325 cursor.execute("SELECT id,generator FROM bitten_report "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	326 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	327 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	328 for report_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	329 cursor.execute("UPDATE bitten_report SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	330 "WHERE id=%s", (mapping[generator], report_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	331
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	332 def add_error_table(env, db):
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	333 """Add the bitten_error table for recording step failure reasons."""
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	334 from trac.db import Table, Column
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	335
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	336 table = Table('bitten_error', key=('build', 'step', 'orderno'))[
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	337 Column('build', type='int'), Column('step'), Column('message'),
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	338 Column('orderno', type='int')
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	339 ]
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	340 cursor = db.cursor()
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	341
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	342 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	343 for stmt in connector.to_sql(table):
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	344 cursor.execute(stmt)
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	345
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	346 def add_filename_to_logs(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	347 """Add filename column to log table to save where log files are stored."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	348 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	349 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	350
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	351 cursor.execute("CREATE TEMPORARY TABLE old_log_v8 AS "
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	352 "SELECT * FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	353 cursor.execute("DROP TABLE bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	354
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	355 connector, _ = DatabaseManager(env)._get_connector()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	356 for stmt in connector.to_sql(BuildLog._schema[0]):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	357 cursor.execute(stmt)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	358
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	359 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno,filename) "
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	360 "SELECT id,build,step,generator,orderno,'' FROM old_log_v8")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	361
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	362 def migrate_logs_to_files(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	363 """Migrates logs that are stored in the bitten_log_messages table into files."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	364 from trac.db import Table, Column
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	365 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	366 log_table = BuildLog._schema[0]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	367 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	368 if not os.path.isabs(logs_dir):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	369 logs_dir = os.path.join(env.path, logs_dir)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	370 if not os.path.exists(logs_dir):
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	371 os.makedirs(logs_dir)
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	372
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	373 message_table = Table('bitten_log_message', key=('log', 'line'))[
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	374 Column('log', type='int'), Column('line', type='int'),
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	375 Column('level', size=1), Column('message')
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	376 ]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	377
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	378 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	379 message_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	380 update_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	381 cursor.execute("SELECT id FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	382 for log_id, in cursor.fetchall():
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	383 filename = "%s.log" % (log_id,)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	384 message_cursor.execute("SELECT message, level FROM bitten_log_message WHERE log=%s ORDER BY line", (log_id,))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	385 full_filename = os.path.join(logs_dir, filename)
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	386 message_file = codecs.open(full_filename, "wb", "UTF-8")
727 508636ffbf11 Added a constant so we don't have any more confusion about what files are used for levels - see #517 dfraser parents: 708 diff changeset	387 # Note: the original version of this code erroneously wrote to filename + ".level" instead of ".levels", producing unused level files
735 90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	388 level_file = codecs.open(full_filename + '.levels', "wb", "UTF-8")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	389 for message, level in message_cursor.fetchall() or []:
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	390 message_file.write(to_unicode(message) + "\n")
b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	391 level_file.write(to_unicode(level) + "\n")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	392 message_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	393 level_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	394 update_cursor.execute("UPDATE bitten_log SET filename=%s WHERE id=%s", (filename, log_id))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	395 env.log.info("Migrated log %s", log_id)
518 18485105d1c3 Fix typo - see #329 dfraser parents: 516 diff changeset	396 env.log.warning("Logs have been migrated from the database to files in %s. "
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	397 "Ensure permissions are set correctly on this file. "
619 3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	398 "Since we presume that the migration worked correctly, "
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	399 "we are now dropping the bitten_log_message table in the database (aren't you glad you backed up)", logs_dir)
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	400 cursor.close()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	401 cursor = db.cursor()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	402 cursor.execute("DROP TABLE bitten_log_message")
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	403 cursor.close()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	404 env.log.warning("We have dropped the bitten_log_message table - you may want to vaccuum/compress your database to save space")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	405
735 90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	406 def fix_log_levels_misnaming(env, db):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	407 """Renames or removes *.log.level files created by older versions of migrate_logs_to_files."""
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	408 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	409 if not os.path.isabs(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	410 logs_dir = os.path.join(env.path, logs_dir)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	411 if not os.path.isdir(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	412 return
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	413
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	414 rename_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	415 rename_error_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	416 delete_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	417 delete_error_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	418
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	419 for wrong_filename in os.listdir(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	420 if not wrong_filename.endswith('.log.level'):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	421 continue
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	422
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	423 log_filename = os.path.splitext(wrong_filename)[0]
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	424 right_filename = log_filename + '.levels'
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	425 full_log_filename = os.path.join(logs_dir, log_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	426 full_wrong_filename = os.path.join(logs_dir, wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	427 full_right_filename = os.path.join(logs_dir, right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	428
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	429 if not os.path.exists(full_log_filename):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	430 try:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	431 os.remove(full_wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	432 delete_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	433 env.log.info("Deleted stray log level file %s", wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	434 except Exception, e:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	435 delete_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	436 env.log.warning("Error removing stray log level file %s: %s", wrong_filename, e)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	437 else:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	438 if os.path.exists(full_right_filename):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	439 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: new filename already exists",
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	440 full_wrong_filename, full_right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	441 rename_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	442 continue
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	443 try:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	444 os.rename(full_wrong_filename, full_right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	445 rename_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	446 env.log.info("Renamed incorrectly named log level file %s to %s", wrong_filename, right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	447 except Exception, e:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	448 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: %s", full_wrong_filename, full_right_filename, e)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	449 rename_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	450
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	451 env.log.info("Renamed %d incorrectly named log level files from previous migrate (%d errors)", rename_count, rename_error_count)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	452 env.log.info("Deleted %d stray log level (%d errors)", delete_count, delete_error_count)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	453
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	454 def recreate_rule_with_int_id(env, db):
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	455 """Recreates the bitten_rule table with an integer id column rather than a text one."""
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	456 from bitten.model import TargetPlatform
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	457 cursor = db.cursor()
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	458 connector, _ = DatabaseManager(env)._get_connector()
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	459
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	460 for table in TargetPlatform._schema:
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	461 if table.name is 'bitten_rule':
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	462 env.log.info("Migrating bitten_rule table to integer ids")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	463 cursor.execute("CREATE TEMPORARY TABLE old_rule AS SELECT * FROM bitten_rule")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	464 cursor.execute("DROP TABLE bitten_rule")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	465 for stmt in connector.to_sql(table):
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	466 cursor.execute(stmt)
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	467 cursor.execute("INSERT INTO bitten_rule (id,propname,pattern,orderno) SELECT %s,propname,pattern,orderno FROM old_rule" % db.cast('id', 'int'))
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	468
650 b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	469 def add_config_platform_rev_index_to_build(env, db):
b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	470 """Adds a unique index on (config, platform, rev) to the bitten_build table.
b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	471 Also drops the old index on bitten_build that serves no real purpose anymore."""
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	472 # check for existing duplicates
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	473 duplicates_cursor = db.cursor()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	474 build_cursor = db.cursor()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	475
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	476 duplicates_cursor.execute("SELECT config, rev, platform FROM bitten_build GROUP BY config, rev, platform HAVING COUNT(config) > 1")
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	477 duplicates_exist = False
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	478 for config, rev, platform in duplicates_cursor.fetchall():
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	479 if not duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	480 duplicates_exist = True
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	481 print "\nConfig Name, Revision, Platform :: [<list of build ids>]"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	482 print "--------------------------------------------------------"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	483
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	484 build_cursor.execute("SELECT id FROM bitten_build WHERE config='%s' AND rev='%s' AND platform='%s'" % (config, rev, platform))
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	485 build_ids = [row[0] for row in build_cursor.fetchall()]
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	486 print "%s, %s, %s :: %s" % (config, rev, platform, build_ids)
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	487
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	488 if duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	489 print "--------------------------------------------------------\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	490 print "Duplicate builds found. You can remove the builds you don't want to"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	491 print "keep by using this one-line command:\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	492 print "$ python -c \"from bitten.model import Build; from trac.env import Environment; " \
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	493 "Build.fetch(Environment('%s'), <buildid>).delete()\"" % env.path
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	494 print "\n...where <buildid> is the id of the build to remove.\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	495 print "Upgrades cannot be performed until conflicts are resolved."
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	496 print "The upgrade script will now exit with an error:\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	497
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	498 duplicates_cursor.close()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	499 build_cursor.close()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	500
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	501 if not duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	502 cursor = db.cursor()
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	503 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	504 if scheme == "mysql":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	505 # 111 = 333 / len(columns in index) -- this is the Trac default
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	506 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config(111), rev(111), platform)")
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	507 else:
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	508 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config,rev,platform)")
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	509 drop_index(env, db, 'bitten_build', 'bitten_build_config_rev_slave_idx')
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	510 else:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	511 raise TracError('')
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	512
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	513 def fix_sequences(env, db):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	514 """Fixes any auto increment sequences that might have been left in an inconsistent state.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	515
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	516 Upgrade scripts for schema versions > 10 should handle sequence updates correctly themselves.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	517 """
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	518 update_sequence(env, db, 'bitten_build', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	519 update_sequence(env, db, 'bitten_log', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	520 update_sequence(env, db, 'bitten_platform', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	521 update_sequence(env, db, 'bitten_report', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	522
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	523
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	524 map = {
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	525 2: [add_log_table],
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	526 3: [add_recipe_to_config],
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	527 4: [add_config_to_reports],
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	528 5: [add_order_to_log, add_report_tables, xmldb_to_db],
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	529 6: [normalize_file_paths, fixup_generators],
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	530 7: [add_error_table],
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	531 8: [add_filename_to_logs,migrate_logs_to_files],
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	532 9: [recreate_rule_with_int_id],
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	533 10: [add_config_platform_rev_index_to_build, fix_sequences],
735 90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	534 11: [fix_log_levels_misnaming],
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	535 }

Mercurial > bitten > bitten-test

annotate bitten/upgrades.py @ 735:90903d6cc932