bitten/bitten-test: bitten/upgrades.py annotate

annotate bitten/upgrades.py @ 762:5f0cfee44540

Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. On the completion of every step, last activity is updated to the current time. All orphaning/invalidation based on slave_timeout happens based on that time, not started. Previously, builds were orphaned if it had been more than slave_timeout (config setting) seconds since they had started, now it's since the last interaction (which is actually what the documentation on the field already says.) This requires a database upgrade. Thanks to Hodgestar for comments. Closes #222. Refs #411.

author	wbell
date	Sat, 24 Apr 2010 15:11:23 +0000
parents	5f95a6f38490
children	025b3e889321

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 323 diff changeset	1 # -- coding: utf-8 --
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	2 #
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	3 # Copyright (C) 2007 Edgewall Software
933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	5 # All rights reserved.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	6 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	8 # you should have received as part of this distribution. The terms
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	9 # are also available at http://bitten.edgewall.org/wiki/License.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	10
316 87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	11 """Automated upgrades for the Bitten database tables, and other data stored
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	12 in the Trac environment."""
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	13
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	14 import os
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	15 import sys
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	16
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	17 from trac.core import TracError
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	18 from trac.db import DatabaseManager
519 384e59137bf8 Support unicode by converting everything to UTF-8 on write and back to unicode on read - should fix #369 dfraser parents: 518 diff changeset	19 from trac.util.text import to_unicode
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	20 import codecs
320 a8b713254286 Fixes for compatibility with Trac trunk and 0.9.3. cmlenz parents: 316 diff changeset	21
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	22 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	23
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	24 # database abstraction functions
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	25
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	26 def _parse_scheme(env):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	27 """Retrieve the environment database scheme."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	28 connection_uri = DatabaseManager(env).connection_uri
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	29 parts = connection_uri.split(':', 1)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	30 scheme = parts[0].lower()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	31 return scheme
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	32
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	33 def update_sequence(env, db, tbl, col):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	34 """Update a sequence associated with an autoincrement column."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	35 # Hopefully Trac will eventually implement its own version
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	36 # of this function.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	37 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	38 if scheme == "postgres":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	39 seq = '%s_%s_seq' % (tbl, col)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	40 cursor = db.cursor()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	41 cursor.execute("SELECT setval('%s', (SELECT MAX(%s) FROM %s))"
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	42 % (seq, col, tbl))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	43
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	44 def drop_index(env, db, tbl, idx):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	45 """Drop an index associated with a table."""
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	46 # Hopefully Trac will eventually implement its own version
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	47 # of this function.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	48 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	49 cursor = db.cursor()
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	50 if scheme == "mysql":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	51 cursor.execute("DROP INDEX %s ON %s" % (idx, tbl))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	52 else:
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	53 cursor.execute("DROP INDEX %s" % (idx,))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	54
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	55 # upgrade scripts
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	56
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	57 def add_log_table(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	58 """Add a table for storing the builds logs."""
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	59 from trac.db import Table, Column
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	60 INFO_LEVEL = 'I'
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	61
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	62 cursor = db.cursor()
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	63
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	64 build_log_schema_v3 = [
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	65 Table('bitten_log', key='id')[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	66 Column('id', auto_increment=True), Column('build', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	67 Column('step'), Column('type')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	68 ],
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	69 Table('bitten_log_message', key=('log', 'line'))[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	70 Column('log', type='int'), Column('line', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	71 Column('level', size=1), Column('message')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	72 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	73 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	74
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	75 build_step_schema_v3 = [
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	76 Table('bitten_step', key=('build', 'name'))[
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	77 Column('build', type='int'), Column('name'), Column('description'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	78 Column('status', size=1), Column('started', type='int'),
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	79 Column('stopped', type='int')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	80 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	81 ]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	82
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	83 connector, _ = DatabaseManager(env)._get_connector()
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	84 for table in build_log_schema_v3:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	85 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	86 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	87
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	88 update_cursor = db.cursor()
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	89 cursor.execute("SELECT build,name,log FROM bitten_step "
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	90 "WHERE log IS NOT NULL")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	91 for build, step, log in cursor:
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	92 update_cursor.execute("INSERT INTO bitten_log (build, step) "
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	93 "VALUES (%s,%s)", (build, step))
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	94 log_id = db.get_last_id(update_cursor, 'bitten_log')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	95 messages = [(log_id, line, INFO_LEVEL, msg)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	96 for line, msg in enumerate(log.splitlines())]
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	97 update_cursor.executemany("INSERT INTO bitten_log_message (log, line, level, message) "
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	98 "VALUES (%s, %s, %s, %s)", messages)
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	99
524 c022478bc111 Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370 dfraser parents: 521 diff changeset	100 cursor.execute("CREATE TEMPORARY TABLE old_step AS SELECT * FROM bitten_step")
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	101 cursor.execute("DROP TABLE bitten_step")
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	102 for table in build_step_schema_v3:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	103 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	104 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	105 cursor.execute("INSERT INTO bitten_step (build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	106 "started,stopped) SELECT build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	107 "started,stopped FROM old_step")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	108
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	109 def add_recipe_to_config(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	110 """Add a column for storing the build recipe to the build configuration
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	111 table."""
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	112 from bitten.model import BuildConfig
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	113 cursor = db.cursor()
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	114
524 c022478bc111 Replace `CREATE TEMP TABLE` with the more database-independent `CREATE TEMP TABLE` (thanks Manfred) - Fixes #370 dfraser parents: 521 diff changeset	115 cursor.execute("CREATE TEMPORARY TABLE old_config AS "
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	116 "SELECT * FROM bitten_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	117 cursor.execute("DROP TABLE bitten_config")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	118
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	119 connector, _ = DatabaseManager(env)._get_connector()
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	120 for table in BuildConfig._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	121 for stmt in connector.to_sql(table):
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	122 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	123
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	124 cursor.execute("INSERT INTO bitten_config (name,path,active,recipe,min_rev,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	125 "max_rev,label,description) SELECT name,path,0,'',NULL,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	126 "NULL,label,description FROM old_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	127
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	128 def add_last_activity_to_build(env, db):
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	129 """Add a column for storing the last activity to the build table."""
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	130 from trac.db import Table, Column, Index
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	131 cursor = db.cursor()
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	132
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	133 build_table_schema_v12 = Table('bitten_build', key='id')[
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	134 Column('id', auto_increment=True), Column('config'), Column('rev'),
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	135 Column('rev_time', type='int'), Column('platform', type='int'),
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	136 Column('slave'), Column('started', type='int'),
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	137 Column('stopped', type='int'), Column('status', size=1),
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	138 Column('last_activity', type='int'),
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	139 Index(['config', 'rev', 'platform'], unique=True)
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	140 ]
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	141
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	142 cursor.execute("CREATE TEMPORARY TABLE old_build_v11 AS "
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	143 "SELECT * FROM bitten_build")
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	144 cursor.execute("DROP TABLE bitten_build")
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	145
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	146 connector, _ = DatabaseManager(env)._get_connector()
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	147 for stmt in connector.to_sql(build_table_schema_v12):
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	148 cursor.execute(stmt)
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	149
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	150 # it's safe to make the last activity the stop time of the build
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	151 cursor.execute("INSERT INTO bitten_build (config,rev,rev_time,platform,"
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	152 "slave,started,stopped,last_activity,status) "
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	153 "SELECT config,rev,rev_time,platform,"
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	154 "slave,started,stopped,stopped,status FROM old_build_v11")
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	155
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	156 def add_config_to_reports(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	157 """Add the name of the build configuration as metadata to report documents
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	158 stored in the BDB XML database."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	159
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	160 from bitten.model import Build
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	161 try:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	162 from bsddb3 import db as bdb
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	163 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	164 except ImportError:
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	165 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	166
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	167 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	168 if not os.path.isfile(dbfile):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	169 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	170
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	171 dbenv = bdb.DBEnv()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	172 dbenv.open(os.path.dirname(dbfile),
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	173 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	174 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	175
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	176 mgr = dbxml.XmlManager(dbenv, 0)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	177 xtn = mgr.createTransaction()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	178 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	179 uc = mgr.createUpdateContext()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	180
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	181 container.addIndex(xtn, '', 'config', 'node-metadata-equality-string', uc)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	182
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	183 qc = mgr.createQueryContext()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	184 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc):
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	185 doc = value.asDocument()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	186 metaval = dbxml.XmlValue()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	187 if doc.getMetaData('', 'build', metaval):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	188 build_id = int(metaval.asNumber())
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	189 build = Build.fetch(env, id=build_id, db=db)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	190 if build:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	191 doc.setMetaData('', 'config', dbxml.XmlValue(build.config))
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	192 container.updateDocument(xtn, doc, uc)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	193 else:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	194 # an orphaned report, for whatever reason... just remove it
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	195 container.deleteDocument(xtn, doc, uc)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	196
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	197 xtn.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	198 container.close()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	199 dbenv.close(0)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	200
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	201 def add_order_to_log(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	202 """Add order column to log table to make sure that build logs are displayed
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	203 in the order they were generated."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	204 from bitten.model import BuildLog
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	205 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	206
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	207 cursor.execute("CREATE TEMPORARY TABLE old_log_v5 AS "
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	208 "SELECT * FROM bitten_log")
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	209 cursor.execute("DROP TABLE bitten_log")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	210
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	211 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	212 for stmt in connector.to_sql(BuildLog._schema[0]):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	213 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	214
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	215 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno) "
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	216 "SELECT id,build,step,type,0 FROM old_log_v5")
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	217
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	218 def add_report_tables(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	219 """Add database tables for report storage."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	220 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	221 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	222
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	223 connector, _ = DatabaseManager(env)._get_connector()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	224 for table in Report._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	225 for stmt in connector.to_sql(table):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	226 cursor.execute(stmt)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	227
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	228 def xmldb_to_db(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	229 """Migrate report data from Berkeley DB XML to SQL database.
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	230
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	231 Depending on the number of reports stored, this might take rather long.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	232 After the upgrade is done, the bitten.dbxml file (and any BDB XML log files)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	233 may be deleted. BDB XML is no longer used by Bitten.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	234 """
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	235 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	236 from bitten.util import xmlio
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	237 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	238 from bsddb3 import db as bdb
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	239 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	240 except ImportError:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	241 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	242
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	243 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	244 if not os.path.isfile(dbfile):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	245 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	246
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	247 dbenv = bdb.DBEnv()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	248 dbenv.open(os.path.dirname(dbfile),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	249 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	250 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	251
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	252 mgr = dbxml.XmlManager(dbenv, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	253 xtn = mgr.createTransaction()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	254 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	255
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	256 def get_pylint_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	257 for problems_elem in xml.children('problems'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	258 for problem_elem in problems_elem.children('problem'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	259 item = {'type': 'problem'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	260 item.update(problem_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	261 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	262
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	263 def get_trace_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	264 for cov_elem in xml.children('coverage'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	265 item = {'type': 'coverage', 'name': cov_elem.attr['module'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	266 'file': cov_elem.attr['file'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	267 'percentage': cov_elem.attr['percentage']}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	268 lines = 0
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	269 line_hits = []
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	270 for line_elem in cov_elem.children('line'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	271 lines += 1
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	272 line_hits.append(line_elem.attr['hits'])
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	273 item['lines'] = lines
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	274 item['line_hits'] = ' '.join(line_hits)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	275 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	276
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	277 def get_unittest_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	278 for test_elem in xml.children('test'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	279 item = {'type': 'test'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	280 item.update(test_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	281 for child_elem in test_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	282 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	283 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	284
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	285 qc = mgr.createQueryContext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	286 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc, 0):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	287 doc = value.asDocument()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	288 metaval = dbxml.XmlValue()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	289 build, step = None, None
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	290 if doc.getMetaData('', 'build', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	291 build = metaval.asNumber()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	292 if doc.getMetaData('', 'step', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	293 step = metaval.asString()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	294
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	295 report_types = {'pylint': ('lint', get_pylint_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	296 'trace': ('coverage', get_trace_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	297 'unittest': ('test', get_unittest_items)}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	298 xml = xmlio.parse(value.asString())
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	299 report_type = xml.attr['type']
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	300 category, get_items = report_types[report_type]
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	301 sys.stderr.write('.')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	302 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	303 report = Report(env, build, step, category=category,
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	304 generator=report_type)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	305 report.items = list(get_items(xml))
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	306 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	307 report.insert(db=db)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	308 except AssertionError:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	309 # Duplicate report, skip
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	310 pass
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	311 sys.stderr.write('\n')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	312 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	313
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	314 xtn.abort()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	315 container.close()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	316 dbenv.close(0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	317
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	318 def normalize_file_paths(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	319 """Normalize the file separator in file names in reports."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	320 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	321 cursor.execute("SELECT report,item,value FROM bitten_report_item "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	322 "WHERE name='file'")
223 067bde207c23 Fix upgrade for installs with no stored reports. Thanks to Matt Good for catching this. cmlenz parents: 219 diff changeset	323 rows = cursor.fetchall() or []
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	324 for report, item, value in rows:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	325 if '\\' in value:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	326 cursor.execute("UPDATE bitten_report_item SET value=%s "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	327 "WHERE report=%s AND item=%s AND name='file'",
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	328 (value.replace('\\', '/'), report, item))
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	329
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	330 def fixup_generators(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	331 """Upgrade the identifiers for the recipe commands that generated log
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	332 messages and report data."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	333
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	334 mapping = {
683 0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	335 'pipe': 'http://bitten.edgewall.org/tools/sh#pipe',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	336 'make': 'http://bitten.edgewall.org/tools/c#make',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	337 'distutils': 'http://bitten.edgewall.org/tools/python#distutils',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	338 'exec_': 'http://bitten.edgewall.org/tools/python#exec' # Ambigious
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	339 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	340 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	341 cursor.execute("SELECT id,generator FROM bitten_log "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	342 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	343 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	344 for log_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	345 cursor.execute("UPDATE bitten_log SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	346 "WHERE id=%s", (mapping[generator], log_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	347
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	348 mapping = {
683 0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	349 'unittest': 'http://bitten.edgewall.org/tools/python#unittest',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	350 'trace': 'http://bitten.edgewall.org/tools/python#trace',
0c4fec90c8e2 0.6dev: Update all tools and docs to use the new `http://bitten.edgewall.org/tools/` namespace as default. Old namespace will still work, but a notice appears when editing config if deprecated namespace is in use. Both will work for now, though. osimons parents: 656 diff changeset	351 'pylint': 'http://bitten.edgewall.org/tools/python#pylint'
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	352 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	353 cursor.execute("SELECT id,generator FROM bitten_report "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	354 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	355 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	356 for report_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	357 cursor.execute("UPDATE bitten_report SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	358 "WHERE id=%s", (mapping[generator], report_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	359
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	360 def add_error_table(env, db):
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	361 """Add the bitten_error table for recording step failure reasons."""
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	362 from trac.db import Table, Column
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	363
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	364 table = Table('bitten_error', key=('build', 'step', 'orderno'))[
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	365 Column('build', type='int'), Column('step'), Column('message'),
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	366 Column('orderno', type='int')
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	367 ]
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	368 cursor = db.cursor()
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	369
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	370 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	371 for stmt in connector.to_sql(table):
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	372 cursor.execute(stmt)
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	373
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	374 def add_filename_to_logs(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	375 """Add filename column to log table to save where log files are stored."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	376 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	377 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	378
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	379 cursor.execute("CREATE TEMPORARY TABLE old_log_v8 AS "
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	380 "SELECT * FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	381 cursor.execute("DROP TABLE bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	382
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	383 connector, _ = DatabaseManager(env)._get_connector()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	384 for stmt in connector.to_sql(BuildLog._schema[0]):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	385 cursor.execute(stmt)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	386
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	387 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno,filename) "
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	388 "SELECT id,build,step,generator,orderno,'' FROM old_log_v8")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	389
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	390 def migrate_logs_to_files(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	391 """Migrates logs that are stored in the bitten_log_messages table into files."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	392 from trac.db import Table, Column
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	393 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	394 log_table = BuildLog._schema[0]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	395 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	396 if not os.path.isabs(logs_dir):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	397 logs_dir = os.path.join(env.path, logs_dir)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	398 if not os.path.exists(logs_dir):
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	399 os.makedirs(logs_dir)
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	400
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	401 message_table = Table('bitten_log_message', key=('log', 'line'))[
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	402 Column('log', type='int'), Column('line', type='int'),
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	403 Column('level', size=1), Column('message')
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	404 ]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	405
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	406 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	407 message_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	408 update_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	409 cursor.execute("SELECT id FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	410 for log_id, in cursor.fetchall():
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	411 filename = "%s.log" % (log_id,)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	412 message_cursor.execute("SELECT message, level FROM bitten_log_message WHERE log=%s ORDER BY line", (log_id,))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	413 full_filename = os.path.join(logs_dir, filename)
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	414 message_file = codecs.open(full_filename, "wb", "UTF-8")
727 508636ffbf11 Added a constant so we don't have any more confusion about what files are used for levels - see #517 dfraser parents: 708 diff changeset	415 # Note: the original version of this code erroneously wrote to filename + ".level" instead of ".levels", producing unused level files
735 90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	416 level_file = codecs.open(full_filename + '.levels', "wb", "UTF-8")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	417 for message, level in message_cursor.fetchall() or []:
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	418 message_file.write(to_unicode(message) + "\n")
b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	419 level_file.write(to_unicode(level) + "\n")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	420 message_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	421 level_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	422 update_cursor.execute("UPDATE bitten_log SET filename=%s WHERE id=%s", (filename, log_id))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	423 env.log.info("Migrated log %s", log_id)
518 18485105d1c3 Fix typo - see #329 dfraser parents: 516 diff changeset	424 env.log.warning("Logs have been migrated from the database to files in %s. "
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	425 "Ensure permissions are set correctly on this file. "
619 3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	426 "Since we presume that the migration worked correctly, "
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	427 "we are now dropping the bitten_log_message table in the database (aren't you glad you backed up)", logs_dir)
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	428 cursor.close()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	429 cursor = db.cursor()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	430 cursor.execute("DROP TABLE bitten_log_message")
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	431 cursor.close()
3dd1b2d4b2a7 After lengthy waiting and no complaints of data loss, add the final step of dropping the old data with witty commentary. See #329 dfraser parents: 565 diff changeset	432 env.log.warning("We have dropped the bitten_log_message table - you may want to vaccuum/compress your database to save space")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	433
735 90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	434 def fix_log_levels_misnaming(env, db):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	435 """Renames or removes *.log.level files created by older versions of migrate_logs_to_files."""
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	436 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	437 if not os.path.isabs(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	438 logs_dir = os.path.join(env.path, logs_dir)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	439 if not os.path.isdir(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	440 return
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	441
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	442 rename_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	443 rename_error_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	444 delete_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	445 delete_error_count = 0
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	446
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	447 for wrong_filename in os.listdir(logs_dir):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	448 if not wrong_filename.endswith('.log.level'):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	449 continue
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	450
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	451 log_filename = os.path.splitext(wrong_filename)[0]
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	452 right_filename = log_filename + '.levels'
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	453 full_log_filename = os.path.join(logs_dir, log_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	454 full_wrong_filename = os.path.join(logs_dir, wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	455 full_right_filename = os.path.join(logs_dir, right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	456
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	457 if not os.path.exists(full_log_filename):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	458 try:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	459 os.remove(full_wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	460 delete_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	461 env.log.info("Deleted stray log level file %s", wrong_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	462 except Exception, e:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	463 delete_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	464 env.log.warning("Error removing stray log level file %s: %s", wrong_filename, e)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	465 else:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	466 if os.path.exists(full_right_filename):
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	467 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: new filename already exists",
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	468 full_wrong_filename, full_right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	469 rename_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	470 continue
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	471 try:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	472 os.rename(full_wrong_filename, full_right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	473 rename_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	474 env.log.info("Renamed incorrectly named log level file %s to %s", wrong_filename, right_filename)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	475 except Exception, e:
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	476 env.log.warning("Error renaming %s to %s in fix_log_levels_misnaming: %s", full_wrong_filename, full_right_filename, e)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	477 rename_error_count += 1
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	478
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	479 env.log.info("Renamed %d incorrectly named log level files from previous migrate (%d errors)", rename_count, rename_error_count)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	480 env.log.info("Deleted %d stray log level (%d errors)", delete_count, delete_error_count)
90903d6cc932 Add upgrade script to fix badly named .log.level files. Don't use BuildLog.LEVELS_SUFFIX in upgrade scripts. See #517. hodgestar parents: 727 diff changeset	481
737 5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	482 def remove_stray_log_levels_files(env, db):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	483 """Remove .log.levels files without a matching .log file (old Bitten versions did not delete .log.levels files when builds were deleted)"""
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	484 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	485 if not os.path.isabs(logs_dir):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	486 logs_dir = os.path.join(env.path, logs_dir)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	487 if not os.path.isdir(logs_dir):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	488 return
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	489
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	490 delete_count = 0
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	491 delete_error_count = 0
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	492
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	493 for filename in os.listdir(logs_dir):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	494 if not filename.endswith('.log.levels'):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	495 continue
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	496
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	497 log_filename = os.path.splitext(filename)[0]
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	498 full_log_filename = os.path.join(logs_dir, log_filename)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	499 full_filename = os.path.join(logs_dir, filename)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	500
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	501 if not os.path.exists(full_log_filename):
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	502 try:
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	503 os.remove(full_filename)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	504 delete_count += 1
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	505 env.log.info("Deleted stray log levels file %s", filename)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	506 except Exception, e:
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	507 delete_error_count += 1
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	508 env.log.warning("Error removing stray log levels file %s: %s", filename, e)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	509
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	510 env.log.info("Deleted %d stray log levels (%d errors)", delete_count, delete_error_count)
5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	511
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	512 def recreate_rule_with_int_id(env, db):
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	513 """Recreates the bitten_rule table with an integer id column rather than a text one."""
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	514 from bitten.model import TargetPlatform
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	515 cursor = db.cursor()
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	516 connector, _ = DatabaseManager(env)._get_connector()
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	517
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	518 for table in TargetPlatform._schema:
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	519 if table.name is 'bitten_rule':
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	520 env.log.info("Migrating bitten_rule table to integer ids")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	521 cursor.execute("CREATE TEMPORARY TABLE old_rule AS SELECT * FROM bitten_rule")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	522 cursor.execute("DROP TABLE bitten_rule")
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	523 for stmt in connector.to_sql(table):
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	524 cursor.execute(stmt)
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	525 cursor.execute("INSERT INTO bitten_rule (id,propname,pattern,orderno) SELECT %s,propname,pattern,orderno FROM old_rule" % db.cast('id', 'int'))
78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	526
650 b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	527 def add_config_platform_rev_index_to_build(env, db):
b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	528 """Adds a unique index on (config, platform, rev) to the bitten_build table.
b1a50f2d92eb 0.6dev: Database upgrade to ensure no duplicate builds are created due to thread race condition when populating builds. Threaded test included. osimons parents: 619 diff changeset	529 Also drops the old index on bitten_build that serves no real purpose anymore."""
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	530 # check for existing duplicates
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	531 duplicates_cursor = db.cursor()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	532 build_cursor = db.cursor()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	533
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	534 duplicates_cursor.execute("SELECT config, rev, platform FROM bitten_build GROUP BY config, rev, platform HAVING COUNT(config) > 1")
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	535 duplicates_exist = False
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	536 for config, rev, platform in duplicates_cursor.fetchall():
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	537 if not duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	538 duplicates_exist = True
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	539 print "\nConfig Name, Revision, Platform :: [<list of build ids>]"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	540 print "--------------------------------------------------------"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	541
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	542 build_cursor.execute("SELECT id FROM bitten_build WHERE config='%s' AND rev='%s' AND platform='%s'" % (config, rev, platform))
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	543 build_ids = [row[0] for row in build_cursor.fetchall()]
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	544 print "%s, %s, %s :: %s" % (config, rev, platform, build_ids)
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	545
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	546 if duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	547 print "--------------------------------------------------------\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	548 print "Duplicate builds found. You can remove the builds you don't want to"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	549 print "keep by using this one-line command:\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	550 print "$ python -c \"from bitten.model import Build; from trac.env import Environment; " \
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	551 "Build.fetch(Environment('%s'), <buildid>).delete()\"" % env.path
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	552 print "\n...where <buildid> is the id of the build to remove.\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	553 print "Upgrades cannot be performed until conflicts are resolved."
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	554 print "The upgrade script will now exit with an error:\n"
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	555
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	556 duplicates_cursor.close()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	557 build_cursor.close()
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	558
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	559 if not duplicates_exist:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	560 cursor = db.cursor()
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	561 scheme = _parse_scheme(env)
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	562 if scheme == "mysql":
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	563 # 111 = 333 / len(columns in index) -- this is the Trac default
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	564 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config(111), rev(111), platform)")
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	565 else:
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	566 cursor.execute("CREATE UNIQUE INDEX bitten_build_config_rev_platform_idx ON bitten_build (config,rev,platform)")
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	567 drop_index(env, db, 'bitten_build', 'bitten_build_config_rev_slave_idx')
656 5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	568 else:
5467873fc0dc 0.6dev: Fix for #214 whereby creating the new index in the upgrade would fail if duplicates aleady exist (`UNIQUE`). In case of duplicates, the duplicates are now printed with information on how to resolve manually. osimons parents: 650 diff changeset	569 raise TracError('')
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	570
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	571 def fix_sequences(env, db):
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	572 """Fixes any auto increment sequences that might have been left in an inconsistent state.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	573
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	574 Upgrade scripts for schema versions > 10 should handle sequence updates correctly themselves.
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	575 """
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	576 update_sequence(env, db, 'bitten_build', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	577 update_sequence(env, db, 'bitten_log', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	578 update_sequence(env, db, 'bitten_platform', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	579 update_sequence(env, db, 'bitten_report', 'id')
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	580
0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	581
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	582 map = {
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	583 2: [add_log_table],
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	584 3: [add_recipe_to_config],
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	585 4: [add_config_to_reports],
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	586 5: [add_order_to_log, add_report_tables, xmldb_to_db],
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	587 6: [normalize_file_paths, fixup_generators],
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	588 7: [add_error_table],
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	589 8: [add_filename_to_logs,migrate_logs_to_files],
565 78109beea395 Type fixes for Postgres 8.x (we noticed issues on 8.3. Newer versions of Postgres don't auto massage integers from Python to string values in the database. In Bitten, this was causing issues with the 'id' field of bitten_rule, which was actually an integer (coming from the id field of the bitten_platform table.) Without explicitly converting it to a string, you'd see the error reported in #390. Added upgrade steps, tested on Postgres8.3. Closes #390. We may want to rename this field to 'platform' later. wbell parents: 524 diff changeset	590 9: [recreate_rule_with_int_id],
708 0d7d6552477e Database upgrade cleanup: hodgestar parents: 683 diff changeset	591 10: [add_config_platform_rev_index_to_build, fix_sequences],
737 5f95a6f38490 Add upgrade method that cleans up .log.levels files which at one point where not properly deleted when builds were removed (with test). Fix fix_log_levels_misnaming test to not rely on the order of logging messages (log message order reflects the directory listing order which can vary). hodgestar parents: 735 diff changeset	592 11: [fix_log_levels_misnaming, remove_stray_log_levels_files],
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 737 diff changeset	593 12: [add_last_activity_to_build],
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	594 }

Mercurial > bitten > bitten-test

annotate bitten/upgrades.py @ 762:5f0cfee44540