bitten/bitten-test: bitten/upgrades.py annotate

annotate bitten/upgrades.py @ 521:b661ea254972

Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): * Use the `codecs` module to write as well as read from files * Explicitly change the mode to binary to avoid confusion * Adjust tests to read in binary mode

author	dfraser
date	Mon, 16 Mar 2009 12:03:19 +0000
parents	384e59137bf8
children	c022478bc111

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 323 diff changeset	1 # -- coding: utf-8 --
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	2 #
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	3 # Copyright (C) 2007 Edgewall Software
933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	5 # All rights reserved.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	6 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 147 diff changeset	8 # you should have received as part of this distribution. The terms
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 379 diff changeset	9 # are also available at http://bitten.edgewall.org/wiki/License.
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	10
316 87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	11 """Automated upgrades for the Bitten database tables, and other data stored
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	12 in the Trac environment."""
87c9b1e8f086 More docstring improvements. cmlenz parents: 245 diff changeset	13
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	14 import os
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	15 import sys
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	16
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	17 from trac.db import DatabaseManager
519 384e59137bf8 Support unicode by converting everything to UTF-8 on write and back to unicode on read - should fix #369 dfraser parents: 518 diff changeset	18 from trac.util.text import to_unicode
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	19 import codecs
320 a8b713254286 Fixes for compatibility with Trac trunk and 0.9.3. cmlenz parents: 316 diff changeset	20
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	21 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	22
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	23 def add_log_table(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	24 """Add a table for storing the builds logs."""
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	25 from bitten.model import BuildLog, BuildStep
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	26 cursor = db.cursor()
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	27
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	28 connector, _ = DatabaseManager(env)._get_connector()
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	29 for table in BuildLog._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	30 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	31 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	32
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	33 cursor.execute("SELECT build,name,log FROM bitten_step "
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	34 "WHERE log IS NOT NULL")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	35 for build, step, log in cursor:
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	36 build_log = BuildLog(env, build, step)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	37 build_log.messages = [(BuildLog.INFO, msg) for msg in log.splitlines()]
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	38 build_log.insert(db)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	39
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	40 cursor.execute("CREATE TEMP TABLE old_step AS SELECT * FROM bitten_step")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	41 cursor.execute("DROP TABLE bitten_step")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	42 for table in BuildStep._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	43 for stmt in connector.to_sql(table):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	44 cursor.execute(stmt)
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	45 cursor.execute("INSERT INTO bitten_step (build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	46 "started,stopped) SELECT build,name,description,status,"
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	47 "started,stopped FROM old_step")
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	48
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	49 def add_recipe_to_config(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	50 """Add a column for storing the build recipe to the build configuration
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	51 table."""
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	52 from bitten.model import BuildConfig
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	53 cursor = db.cursor()
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	54
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	55 cursor.execute("CREATE TEMP TABLE old_config AS "
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	56 "SELECT * FROM bitten_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	57 cursor.execute("DROP TABLE bitten_config")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	58
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	59 connector, _ = DatabaseManager(env)._get_connector()
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	60 for table in BuildConfig._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	61 for stmt in connector.to_sql(table):
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	62 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	63
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	64 cursor.execute("INSERT INTO bitten_config (name,path,active,recipe,min_rev,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	65 "max_rev,label,description) SELECT name,path,0,'',NULL,"
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	66 "NULL,label,description FROM old_config")
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	67
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	68 def add_config_to_reports(env, db):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	69 """Add the name of the build configuration as metadata to report documents
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	70 stored in the BDB XML database."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	71
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	72 from bitten.model import Build
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	73 try:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	74 from bsddb3 import db as bdb
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	75 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	76 except ImportError:
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	77 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	78
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	79 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	80 if not os.path.isfile(dbfile):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	81 return
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	82
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	83 dbenv = bdb.DBEnv()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	84 dbenv.open(os.path.dirname(dbfile),
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	85 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	86 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	87
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	88 mgr = dbxml.XmlManager(dbenv, 0)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	89 xtn = mgr.createTransaction()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	90 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	91 uc = mgr.createUpdateContext()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	92
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	93 container.addIndex(xtn, '', 'config', 'node-metadata-equality-string', uc)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	94
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	95 qc = mgr.createQueryContext()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	96 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc):
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	97 doc = value.asDocument()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	98 metaval = dbxml.XmlValue()
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	99 if doc.getMetaData('', 'build', metaval):
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	100 build_id = int(metaval.asNumber())
79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	101 build = Build.fetch(env, id=build_id, db=db)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	102 if build:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	103 doc.setMetaData('', 'config', dbxml.XmlValue(build.config))
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	104 container.updateDocument(xtn, doc, uc)
175 f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	105 else:
f7c2f112afe6 Fix the BDB XML upgrade procedure: If a document is found that is associated with a non-existing build, just remove it. cmlenz parents: 174 diff changeset	106 # an orphaned report, for whatever reason... just remove it
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	107 container.deleteDocument(xtn, doc, uc)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	108
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	109 xtn.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	110 container.close()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 196 diff changeset	111 dbenv.close(0)
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	112
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	113 def add_order_to_log(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	114 """Add order column to log table to make sure that build logs are displayed
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	115 in the order they were generated."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	116 from bitten.model import BuildLog
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	117 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	118
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	119 cursor.execute("CREATE TEMP TABLE old_log AS "
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	120 "SELECT * FROM bitten_log")
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	121 cursor.execute("DROP TABLE bitten_log")
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	122
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	123 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	124 for stmt in connector.to_sql(BuildLog._schema[0]):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	125 cursor.execute(stmt)
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	126
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	127 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno) "
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	128 "SELECT id,build,step,type,0 FROM old_log")
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	129
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	130 def add_report_tables(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	131 """Add database tables for report storage."""
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	132 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	133 cursor = db.cursor()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	134
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	135 connector, _ = DatabaseManager(env)._get_connector()
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	136 for table in Report._schema:
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	137 for stmt in connector.to_sql(table):
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	138 cursor.execute(stmt)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	139
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	140 def xmldb_to_db(env, db):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	141 """Migrate report data from Berkeley DB XML to SQL database.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	142
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	143 Depending on the number of reports stored, this might take rather long.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	144 After the upgrade is done, the bitten.dbxml file (and any BDB XML log files)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	145 may be deleted. BDB XML is no longer used by Bitten.
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	146 """
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	147 from bitten.model import Report
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	148 from bitten.util import xmlio
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	149 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	150 from bsddb3 import db as bdb
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	151 import dbxml
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	152 except ImportError:
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	153 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	154
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	155 dbfile = os.path.join(env.path, 'db', 'bitten.dbxml')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	156 if not os.path.isfile(dbfile):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	157 return
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	158
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	159 dbenv = bdb.DBEnv()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	160 dbenv.open(os.path.dirname(dbfile),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	161 bdb.DB_CREATE \| bdb.DB_INIT_LOCK \| bdb.DB_INIT_LOG \|
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	162 bdb.DB_INIT_MPOOL \| bdb.DB_INIT_TXN, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	163
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	164 mgr = dbxml.XmlManager(dbenv, 0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	165 xtn = mgr.createTransaction()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	166 container = mgr.openContainer(dbfile, dbxml.DBXML_TRANSACTIONAL)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	167
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	168 def get_pylint_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	169 for problems_elem in xml.children('problems'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	170 for problem_elem in problems_elem.children('problem'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	171 item = {'type': 'problem'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	172 item.update(problem_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	173 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	174
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	175 def get_trace_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	176 for cov_elem in xml.children('coverage'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	177 item = {'type': 'coverage', 'name': cov_elem.attr['module'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	178 'file': cov_elem.attr['file'],
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	179 'percentage': cov_elem.attr['percentage']}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	180 lines = 0
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	181 line_hits = []
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	182 for line_elem in cov_elem.children('line'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	183 lines += 1
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	184 line_hits.append(line_elem.attr['hits'])
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	185 item['lines'] = lines
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	186 item['line_hits'] = ' '.join(line_hits)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	187 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	188
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	189 def get_unittest_items(xml):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	190 for test_elem in xml.children('test'):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	191 item = {'type': 'test'}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	192 item.update(test_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	193 for child_elem in test_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	194 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	195 yield item
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	196
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	197 qc = mgr.createQueryContext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	198 for value in mgr.query(xtn, 'collection("%s")/report' % dbfile, qc, 0):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	199 doc = value.asDocument()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	200 metaval = dbxml.XmlValue()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	201 build, step = None, None
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	202 if doc.getMetaData('', 'build', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	203 build = metaval.asNumber()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	204 if doc.getMetaData('', 'step', metaval):
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	205 step = metaval.asString()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	206
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	207 report_types = {'pylint': ('lint', get_pylint_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	208 'trace': ('coverage', get_trace_items),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	209 'unittest': ('test', get_unittest_items)}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	210 xml = xmlio.parse(value.asString())
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	211 report_type = xml.attr['type']
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	212 category, get_items = report_types[report_type]
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	213 sys.stderr.write('.')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	214 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	215 report = Report(env, build, step, category=category,
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	216 generator=report_type)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	217 report.items = list(get_items(xml))
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	218 try:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	219 report.insert(db=db)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	220 except AssertionError:
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	221 # Duplicate report, skip
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	222 pass
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	223 sys.stderr.write('\n')
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	224 sys.stderr.flush()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	225
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	226 xtn.abort()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	227 container.close()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	228 dbenv.close(0)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	229
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	230 def normalize_file_paths(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	231 """Normalize the file separator in file names in reports."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	232 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	233 cursor.execute("SELECT report,item,value FROM bitten_report_item "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	234 "WHERE name='file'")
223 067bde207c23 Fix upgrade for installs with no stored reports. Thanks to Matt Good for catching this. cmlenz parents: 219 diff changeset	235 rows = cursor.fetchall() or []
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	236 for report, item, value in rows:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	237 if '\\' in value:
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	238 cursor.execute("UPDATE bitten_report_item SET value=%s "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	239 "WHERE report=%s AND item=%s AND name='file'",
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	240 (value.replace('\\', '/'), report, item))
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	241
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	242 def fixup_generators(env, db):
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	243 """Upgrade the identifiers for the recipe commands that generated log
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	244 messages and report data."""
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	245
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	246 mapping = {
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	247 'pipe': 'http://bitten.cmlenz.net/tools/sh#pipe',
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	248 'make': 'http://bitten.cmlenz.net/tools/c#make',
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	249 'distutils': 'http://bitten.cmlenz.net/tools/python#distutils',
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	250 'exec_': 'http://bitten.cmlenz.net/tools/python#exec' # Ambigious
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	251 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	252 cursor = db.cursor()
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	253 cursor.execute("SELECT id,generator FROM bitten_log "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	254 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	255 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	256 for log_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	257 cursor.execute("UPDATE bitten_log SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	258 "WHERE id=%s", (mapping[generator], log_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	259
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	260 mapping = {
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	261 'unittest': 'http://bitten.cmlenz.net/tools/python#unittest',
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	262 'trace': 'http://bitten.cmlenz.net/tools/python#trace',
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	263 'pylint': 'http://bitten.cmlenz.net/tools/python#pylint'
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	264 }
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	265 cursor.execute("SELECT id,generator FROM bitten_report "
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	266 "WHERE generator IN (%s)"
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	267 % ','.join([repr(key) for key in mapping.keys()]))
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	268 for report_id, generator in cursor:
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	269 cursor.execute("UPDATE bitten_report SET generator=%s "
219 aef09843d367 Some pylint-inspired cleanup. cmlenz parents: 213 diff changeset	270 "WHERE id=%s", (mapping[generator], report_id))
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	271
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	272 def add_error_table(env, db):
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	273 """Add the bitten_error table for recording step failure reasons."""
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	274 from trac.db import Table, Column
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	275
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	276 table = Table('bitten_error', key=('build', 'step', 'orderno'))[
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	277 Column('build', type='int'), Column('step'), Column('message'),
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	278 Column('orderno', type='int')
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	279 ]
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	280 cursor = db.cursor()
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	281
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	282 connector, _ = DatabaseManager(env)._get_connector()
7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	283 for stmt in connector.to_sql(table):
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	284 cursor.execute(stmt)
a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	285
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	286 def add_filename_to_logs(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	287 """Add filename column to log table to save where log files are stored."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	288 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	289 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	290
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	291 cursor.execute("CREATE TEMP TABLE old_log AS "
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	292 "SELECT * FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	293 cursor.execute("DROP TABLE bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	294
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	295 connector, _ = DatabaseManager(env)._get_connector()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	296 for stmt in connector.to_sql(BuildLog._schema[0]):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	297 cursor.execute(stmt)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	298
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	299 cursor.execute("INSERT INTO bitten_log (id,build,step,generator,orderno,filename) "
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	300 "SELECT id,build,step,generator,orderno,'' FROM old_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	301
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	302 def migrate_logs_to_files(env, db):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	303 """Migrates logs that are stored in the bitten_log_messages table into files."""
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	304 from trac.db import Table, Column
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	305 from bitten.model import BuildLog
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	306 log_table = BuildLog._schema[0]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	307 logs_dir = env.config.get("bitten", "logs_dir", "log/bitten")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	308 if not os.path.isabs(logs_dir):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	309 logs_dir = os.path.join(env.path, logs_dir)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	310 if not os.path.exists(logs_dir):
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	311 os.mkdir(logs_dir)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	312
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	313 message_table = Table('bitten_log_message', key=('log', 'line'))[
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	314 Column('log', type='int'), Column('line', type='int'),
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	315 Column('level', size=1), Column('message')
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	316 ]
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	317
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	318 cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	319 message_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	320 update_cursor = db.cursor()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	321 cursor.execute("SELECT id FROM bitten_log")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	322 for log_id, in cursor.fetchall():
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	323 filename = "%s.log" % (log_id,)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	324 message_cursor.execute("SELECT message, level FROM bitten_log_message WHERE log=%s ORDER BY line", (log_id,))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	325 full_filename = os.path.join(logs_dir, filename)
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	326 message_file = codecs.open(full_filename, "wb", "UTF-8")
b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	327 level_file = codecs.open(full_filename+".level", "wb", "UTF-8")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	328 for message, level in message_cursor.fetchall() or []:
521 b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	329 message_file.write(to_unicode(message) + "\n")
b661ea254972 Ensure log files are stored and read in binary, not text format (otherwise Unicode gets confused): dfraser parents: 519 diff changeset	330 level_file.write(to_unicode(level) + "\n")
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	331 message_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	332 level_file.close()
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	333 update_cursor.execute("UPDATE bitten_log SET filename=%s WHERE id=%s", (filename, log_id))
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	334 env.log.info("Migrated log %s", log_id)
518 18485105d1c3 Fix typo - see #329 dfraser parents: 516 diff changeset	335 env.log.warning("Logs have been migrated from the database to files in %s. "
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	336 "Ensure permissions are set correctly on this file. "
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	337 "When you have confirmed that the migration worked correctly, "
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	338 "you can drop the bitten_log_message table in the database (it remains as a backup)", logs_dir)
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	339
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	340 map = {
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 112 diff changeset	341 2: [add_log_table],
174 79c61c26a4e1 * Changed the `IReportStore` interface to allow querying with [http://www.w3.org/XML/Query/ XQuery]. This should make it possible to efficiently query the report store for any existing metrics. cmlenz parents: 163 diff changeset	342 3: [add_recipe_to_config],
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 200 diff changeset	343 4: [add_config_to_reports],
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 203 diff changeset	344 5: [add_order_to_log, add_report_tables, xmldb_to_db],
245 a22ec8fce6c9 Store the reason(s) for build step failure in the database. cmlenz parents: 223 diff changeset	345 6: [normalize_file_paths, fixup_generators],
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	346 7: [add_error_table],
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 411 diff changeset	347 8: [add_filename_to_logs,migrate_logs_to_files],
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: diff changeset	348 }

Mercurial > bitten > bitten-test

annotate bitten/upgrades.py @ 521:b661ea254972