bitten/bitten-test: bitten/master.py annotate

annotate bitten/master.py @ 135:d2b833187429 release-0.3

Bump up version number.

author	cmlenz
date	Sat, 13 Aug 2005 10:59:42 +0000
parents	091ead7d2876
children	395b67aa072e

rev	line source
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	1 # -- coding: iso8859-1 --
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	4 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	5 # Bitten is free software; you can redistribute it and/or
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	6 # modify it under the terms of the GNU General Public License as
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	7 # published by the Free Software Foundation; either version 2 of the
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	8 # License, or (at your option) any later version.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	9 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	10 # Trac is distributed in the hope that it will be useful,
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	13 # General Public License for more details.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	14 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	15 # You should have received a copy of the GNU General Public License
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	16 # along with this program; if not, write to the Free Software
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	17 # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	18 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	19 # Author: Christopher Lenz <cmlenz@gmx.de>
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	20
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	21 from datetime import datetime, timedelta
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	22 from itertools import ifilter
15 06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4. cmlenz parents: 14 diff changeset	23 import logging
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	24 import os.path
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	25 import re
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	26 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	27 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	28 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	29 from sets import Set as set
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	30 import time
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	31
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	32 from trac.env import Environment
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	33 from bitten.model import BuildConfig, TargetPlatform, Build, BuildStep, BuildLog
116 86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	34 from bitten.store import ReportStore
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	35 from bitten.util import archive, beep, xmlio
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	36
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	37 log = logging.getLogger('bitten.master')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	38
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	39 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	40
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	41
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	42 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	43
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	44 def __init__(self, env_path, ip, port, adjust_timestamps=False,
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	45 check_interval=DEFAULT_CHECK_INTERVAL):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	46 beep.Listener.__init__(self, ip, port)
24 cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	47 self.profiles[OrchestrationProfileHandler.URI] = OrchestrationProfileHandler
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	48 self.env = Environment(env_path)
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	49 self.adjust_timestamps = adjust_timestamps
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	50 self.check_interval = check_interval
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	51
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	52 self.slaves = {}
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	53
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	54 # path to generated snapshot archives, key is (config name, revision)
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	55 self.snapshots = {}
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	56 for config in BuildConfig.select(self.env):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	57 snapshots = archive.index(self.env, prefix=config.name)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	58 for rev, format, path in snapshots:
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	59 self.snapshots[(config.name, rev, format)] = path
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	60
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	61 self._cleanup_orphaned_builds()
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	62 self.schedule(self.check_interval, self._check_build_triggers)
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	63
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	64 def close(self):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	65 self._cleanup_orphaned_builds()
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	66 beep.Listener.close(self)
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	67
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	68 def _check_build_triggers(self, when):
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	69 self.schedule(self.check_interval, self._check_build_triggers)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	70
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	71 repos = self.env.get_repository()
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	72 try:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	73 repos.sync()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	74
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	75 db = self.env.get_db_cnx()
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	76 for config in BuildConfig.select(self.env, db=db):
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	77 log.debug('Checking for changes to "%s" at %s', config.label,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	78 config.path)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	79 node = repos.get_node(config.path)
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	80 for path, rev, chg in node.get_history():
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	81 enqueued = False
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	82 for platform in TargetPlatform.select(self.env,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	83 config.name, db=db):
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	84 # Check whether the latest revision of the configuration
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	85 # has already been built on this platform
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	86 builds = Build.select(self.env, config.name, rev,
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	87 platform.id, db=db)
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	88 if not list(builds):
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	89 log.info('Enqueuing build of configuration "%s" at '
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	90 'revision [%s] on %s', config.name, rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	91 platform.name)
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	92 build = Build(self.env)
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	93 build.config = config.name
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	94 build.rev = str(rev)
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	95 build.rev_time = repos.get_changeset(rev).date
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	96 build.platform = platform.id
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	97 build.insert(db)
76 ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	98 enqueued = True
ffa1ffd8c7db * Implement basic slave selection based on configured target platforms. Closes #15. cmlenz parents: 73 diff changeset	99 if enqueued:
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	100 db.commit()
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	101 break
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	102 finally:
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	103 repos.close()
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	104
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	105 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	106 self.schedule(self.check_interval * 1.8, self._cleanup_snapshots)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	107
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	108 def _check_build_queue(self, when):
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	109 if not self.slaves:
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	110 return
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	111 log.debug('Checking for pending builds...')
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	112 for build in Build.select(self.env, status=Build.PENDING):
86 110bfa3cbc32 Slaves were getting associated with the wrong target platform. cmlenz parents: 85 diff changeset	113 for slave in self.slaves.get(build.platform, []):
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	114 active_builds = Build.select(self.env, slave=slave.name,
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	115 status=Build.IN_PROGRESS)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	116 if not list(active_builds):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	117 slave.send_initiation(build)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	118 return
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	119
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	120 def _cleanup_orphaned_builds(self):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	121 # Remove all pending or in-progress builds
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	122 db = self.env.get_db_cnx()
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	123 for build in Build.select(self.env, status=Build.IN_PROGRESS, db=db):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	124 build.status = Build.PENDING
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	125 build.update(db=db)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	126 for build in Build.select(self.env, status=Build.PENDING, db=db):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	127 build.delete(db=db)
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	128 db.commit()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	129
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	130 def _cleanup_snapshots(self, when):
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	131 log.debug('Checking for unused snapshot archives...')
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	132 for (config, rev, format), path in self.snapshots.items():
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	133 keep = False
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	134 for build in Build.select(self.env, config=config, rev=rev):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	135 if build.status not in (Build.SUCCESS, Build.FAILURE):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	136 keep = True
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	137 break
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	138 if not keep:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	139 log.info('Removing unused snapshot %s', path)
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	140 os.unlink(path)
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	141 del self.snapshots[(config, rev, format)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	142
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	143 def get_snapshot(self, build, format):
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	144 snapshot = self.snapshots.get((build.config, build.rev, format))
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	145 if not snapshot:
96 c8c36f34ff5a Change model class pattern: cmlenz parents: 95 diff changeset	146 config = BuildConfig.fetch(self.env, build.config)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	147 snapshot = archive.pack(self.env, path=config.path, rev=build.rev,
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	148 prefix=config.name, format=format)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	149 log.info('Prepared snapshot archive at %s' % snapshot)
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	150 self.snapshots[(build.config, build.rev, format)] = snapshot
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	151 return snapshot
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	152
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	153 def register(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	154 any_match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	155 for config in BuildConfig.select(self.env):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	156 for platform in TargetPlatform.select(self.env, config=config.name):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	157 if not platform.id in self.slaves:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	158 self.slaves[platform.id] = set()
85 acb7b67b8152 Fix matching of slave properties against target platform rules. cmlenz parents: 84 diff changeset	159 match = True
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	160 for property, pattern in ifilter(None, platform.rules):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	161 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	162 if not re.match(pattern, handler.info.get(property)):
86 110bfa3cbc32 Slaves were getting associated with the wrong target platform. cmlenz parents: 85 diff changeset	163 match = False
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	164 break
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	165 except re.error, e:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	166 log.error('Invalid platform matching pattern "%s"',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	167 pattern, exc_info=True)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	168 match = False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	169 break
84 30d09249de80 Fix var name in slave registration. cmlenz parents: 83 diff changeset	170 if match:
96 c8c36f34ff5a Change model class pattern: cmlenz parents: 95 diff changeset	171 log.debug('Slave %s matched target platform %s',
c8c36f34ff5a Change model class pattern: cmlenz parents: 95 diff changeset	172 handler.name, platform.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	173 self.slaves[platform.id].add(handler)
86 110bfa3cbc32 Slaves were getting associated with the wrong target platform. cmlenz parents: 85 diff changeset	174 any_match = True
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	175
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	176 if not any_match:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	177 log.warning('Slave %s does not match any of the configured target '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	178 'platforms', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	179 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	180
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	181 self.schedule(self.check_interval * 0.2, self._check_build_queue)
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	182
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	183 log.info('Registered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	184 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	185
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	186 def unregister(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	187 for slaves in self.slaves.values():
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	188 slaves.discard(handler)
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	189
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	190 db = self.env.get_db_cnx()
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	191 for build in Build.select(self.env, slave=handler.name,
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	192 status=Build.IN_PROGRESS, db=db):
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	193 log.info('Build [%s] of "%s" by %s cancelled', build.rev,
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	194 build.config, handler.name)
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	195 for step in BuildStep.select(self.env, build=build.id):
2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	196 step.delete(db=db)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	197 build.slave = None
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	198 build.slave_info = {}
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	199 build.status = Build.PENDING
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	200 build.started = 0
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	201 build.update(db=db)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	202 break
117 2f0f2f006526 Delete build steps when cleaning up pending builds. cmlenz parents: 116 diff changeset	203 db.commit()
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	204 log.info('Unregistered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	205
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	206
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	207 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	208 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	209 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	210 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	211 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	212
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	213 def handle_connect(self):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	214 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	215 assert self.master
69 b92d7c7d70fd Record build slave properties in database. cmlenz parents: 66 diff changeset	216 self.env = self.master.env
b92d7c7d70fd Record build slave properties in database. cmlenz parents: 66 diff changeset	217 assert self.env
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	218 self.name = None
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	219 self.info = {}
24 cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	220
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	221 def handle_disconnect(self):
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	222 self.master.unregister(self)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	223
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	224 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	225 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	226 elem = xmlio.parse(payload.body)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	227
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	228 if elem.name == 'register':
69 b92d7c7d70fd Record build slave properties in database. cmlenz parents: 66 diff changeset	229 self.name = elem.attr['name']
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	230 self.info[Build.IP_ADDRESS] = self.session.addr[0]
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	231 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	232 if child.name == 'platform':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	233 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	234 self.info[Build.PROCESSOR] = child.attr.get('processor')
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	235 elif child.name == 'os':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	236 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	237 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	238 self.info[Build.OS_VERSION] = child.attr.get('version')
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	239 elif child.name == 'package':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	240 for name, value in child.attr.items():
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	241 if name == 'name':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	242 continue
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	243 self.info[child.attr['name'] + '.' + name] = value
14 1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}. cmlenz parents: 13 diff changeset	244
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	245 if not self.master.register(self):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	246 xml = xmlio.Element('error', code=550)[
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	247 'Nothing for you to build here, please move along'
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	248 ]
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	249 self.channel.send_err(msgno, beep.Payload(xml))
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	250 return
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	251
29 2adc3480e4aa Some cleanup and additional docstrings. cmlenz parents: 24 diff changeset	252 xml = xmlio.Element('ok')
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	253 self.channel.send_rpy(msgno, beep.Payload(xml))
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	254
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	255 def send_initiation(self, build):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	256 log.info('Initiating build of "%s" on slave %s', build.config,
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	257 self.name)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	258
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	259 def handle_reply(cmd, msgno, ansno, payload):
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	260 if cmd == 'ERR':
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	261 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	262 elem = xmlio.parse(payload.body)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	263 if elem.name == 'error':
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	264 log.warning('Slave %s refused build request: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	265 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	266 int(elem.attr['code']))
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	267 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	268
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	269 elem = xmlio.parse(payload.body)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	270 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	271 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	272 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	273 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	274 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	275 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	276 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	277 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	278 break
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	279 type = None
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	280 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	281 xml = xmlio.Element('error', code=550)[
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	282 'None of the accepted archive formats supported'
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	283 ]
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	284 self.channel.send_err(beep.Payload(xml))
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	285 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	286 self.send_snapshot(build, type, encoding)
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	287
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	288 xml = xmlio.Element('build', recipe='recipe.xml')
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	289 self.channel.send_msg(beep.Payload(xml), handle_reply=handle_reply)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	290
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	291 def send_snapshot(self, build, type, encoding):
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	292
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	293 timestamp_delta = 0
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	294 if self.master.adjust_timestamps:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	295 d = datetime.now() - timedelta(seconds=self.master.check_interval) \
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	296 - datetime.fromtimestamp(build.rev_time)
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	297 log.info('Warping timestamps by %s' % d)
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	298 timestamp_delta = d.days * 86400 + d.seconds
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	299
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	300 def handle_reply(cmd, msgno, ansno, payload):
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	301 if cmd == 'ERR':
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	302 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	303 elem = xmlio.parse(payload.body)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	304 if elem.name == 'error':
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	305 log.warning('Slave %s did not accept archive: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	306 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	307 int(elem.attr['code']))
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	308
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	309 elif cmd == 'ANS':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	310 assert payload.content_type == beep.BEEP_XML
94 e5d1a792aa45 Cleanup and records when a build is aborted. cmlenz parents: 93 diff changeset	311 db = self.env.get_db_cnx()
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	312 elem = xmlio.parse(payload.body)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	313 if elem.name == 'started':
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	314 self._build_started(db, build, elem, timestamp_delta)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	315 elif elem.name == 'step':
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	316 self._build_step_completed(db, build, elem, timestamp_delta)
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	317 elif elem.name == 'completed':
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	318 self._build_completed(db, build, elem, timestamp_delta)
70 ccd03b6f04ef Updated DTD for orchestration profile. cmlenz parents: 69 diff changeset	319 elif elem.name == 'aborted':
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	320 self._build_aborted(db, build, elem, timestamp_delta)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	321 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	322 build.status = Build.FAILURE
94 e5d1a792aa45 Cleanup and records when a build is aborted. cmlenz parents: 93 diff changeset	323 build.update(db=db)
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	324 db.commit()
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	325
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	326 snapshot_format = {
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	327 ('application/tar', 'bzip2'): 'bzip2',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	328 ('application/tar', 'gzip'): 'gzip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	329 ('application/tar', None): 'tar',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	330 ('application/zip', None): 'zip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	331 }[(type, encoding)]
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	332 snapshot_path = self.master.get_snapshot(build, snapshot_format)
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	333 snapshot_name = os.path.basename(snapshot_path)
90 2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17. cmlenz parents: 88 diff changeset	334 message = beep.Payload(file(snapshot_path), content_type=type,
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	335 content_disposition=snapshot_name,
90 2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17. cmlenz parents: 88 diff changeset	336 content_encoding=encoding)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	337 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	338
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	339 def _build_started(self, db, build, elem, timestamp_delta=None):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	340 build.slave = self.name
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	341 build.slave_info.update(self.info)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	342 build.started = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	343 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	344 build.started -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	345 build.status = Build.IN_PROGRESS
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	346 log.info('Slave %s started build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	347 self.name, build.id, build.config, build.rev)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	348
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	349 def _build_step_completed(self, db, build, elem, timestamp_delta=None):
119 430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	350 log.debug('Slave completed step "%s" with status %s', elem.attr['id'],
430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	351 elem.attr['result'])
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	352 step = BuildStep(self.env, build=build.id, name=elem.attr['id'],
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	353 description=elem.attr.get('description'))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	354 step.started = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	355 step.stopped = step.started + int(elem.attr['duration'])
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	356 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	357 step.started -= timestamp_delta
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	358 step.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	359 if elem.attr['result'] == 'failure':
135 d2b833187429 Bump up version number. cmlenz parents: 130 diff changeset	360 log.warning('Step failed')
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	361 step.status = BuildStep.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	362 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	363 step.status = BuildStep.SUCCESS
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	364 step.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	365
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	366 level_map = {'debug': BuildLog.DEBUG, 'info': BuildLog.INFO,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	367 'warning': BuildLog.WARNING, 'error': BuildLog.ERROR}
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	368 for log_elem in elem.children('log'):
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	369 build_log = BuildLog(self.env, build=build.id, step=step.name,
a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	370 type=log_elem.attr.get('type'))
115 16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	371 for message_elem in log_elem.children('message'):
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	372 build_log.messages.append((message_elem.attr['level'],
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	373 message_elem.gettext()))
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	374 build_log.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	375
116 86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	376 store = ReportStore(self.env)
86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	377 for report in elem.children('report'):
86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	378 store.store_report(build, step, report)
86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	379
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	380 def _build_completed(self, db, build, elem, timestamp_delta=None):
119 430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	381 log.info('Slave %s completed build %d ("%s" as of [%s]) with status %s',
430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	382 self.name, build.id, build.config, build.rev,
430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	383 elem.attr['result'])
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	384 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	385 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	386 build.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	387 if elem.attr['result'] == 'failure':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	388 build.status = Build.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	389 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	390 build.status = Build.SUCCESS
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	391
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	392 def _build_aborted(self, db, build, elem, timestamp_delta=None):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	393 log.info('Slave "%s" aborted build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	394 self.name, build.id, build.config, build.rev)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	395 build.slave = None
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	396 build.started = 0
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	397 build.status = Build.PENDING
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	398 build.slave_info = {}
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	399 for step in BuildStep.select(self.env, build=build.id, db=db):
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	400 step.delete(db=db)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	401
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	402
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	403 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	404 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	405
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	406 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	407 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	408 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	409 string = string.split('.', 1)[0] # strip out microseconds
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	410 secs = time.mktime(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	411 tzoffset = time.timezone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	412 if time.daylight:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	413 tzoffset = time.altzone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	414 return secs - tzoffset
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	415 except ValueError, e:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	416 raise ValueError, 'Invalid ISO date/time %s (%s)' % (string, e)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	417
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	418 def main():
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	419 from bitten import __version__ as VERSION
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	420 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	421
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	422 # Parse command-line arguments
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	423 parser = OptionParser(usage='usage: %prog [options] env-path',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	424 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	425 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	426 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	427 parser.add_option('-H', '--host', action='store', dest='host',
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	428 metavar='HOSTNAME',
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	429 help='the host name or IP address to bind to')
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	430 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	431 help='write log messages to FILENAME')
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	432 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	433 default=DEFAULT_CHECK_INTERVAL, type='int',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	434 help='poll interval for changeset detection')
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	435 parser.add_option('--timewarp', action='store_const', dest='timewarp',
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	436 const=True,
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	437 help='adjust timestamps of builds to be neat the '
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	438 'timestamps of the corresponding changesets')
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	439 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	440 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	441 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	442 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	443 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	444 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	445 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	446 options, args = parser.parse_args()
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	447
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	448 if len(args) < 1:
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	449 parser.error('incorrect number of arguments')
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	450 env_path = args[0]
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	451
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	452 # Configure logging
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	453 log = logging.getLogger('bitten')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	454 log.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	455 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	456 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	457 handler.setLevel(logging.WARNING)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	458 else:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	459 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	460 formatter = logging.Formatter('%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	461 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	462 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	463 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	464 handler = logging.FileHandler(options.logfile)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	465 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	466 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	467 '%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	468 handler.setFormatter(formatter)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	469 log.addHandler(handler)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	470
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	471 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	472 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	473 parser.error('port must be an integer in the range 1-65535')
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	474
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	475 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	476 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	477 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	478 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	479 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	480 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	481 except socket.error, e:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	482 log.warning('Reverse host name lookup failed (%s)', e)
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	483 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	484
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	485 master = Master(env_path, host, port, adjust_timestamps=options.timewarp,
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	486 check_interval=options.interval)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	487 try:
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	488 master.run(timeout=5.0)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	489 except KeyboardInterrupt:
34 6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7. cmlenz parents: 33 diff changeset	490 master.quit()
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	491
0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	492 if __name__ == '__main__':
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	493 main()

Mercurial > bitten > bitten-test

annotate bitten/master.py @ 135:d2b833187429 release-0.3