bitten/bitten-test: bitten/master.py annotate

annotate bitten/master.py @ 379:0df178e07fdb

Use UTF-8 as encoding of source files.

author	cmlenz
date	Tue, 24 Jul 2007 17:32:02 +0000
parents	a465b5dbfecf
children	026d9aa41b85

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 377 diff changeset	1 # -- coding: utf-8 --
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	4 # All rights reserved.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	5 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	6 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	7 # you should have received as part of this distribution. The terms
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	8 # are also available at http://bitten.cmlenz.net/wiki/License.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	9
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	10 """Build master implementation.
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	11
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	12 This module is runnable as a script to launch the build master. The build
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	13 master starts a single process that handles connections to any number of build
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	14 slaves.
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	15 """
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	16
312 1016c3d12cbc Fix UTC timestamp handling in build master. cmlenz parents: 310 diff changeset	17 import calendar
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	18 from datetime import datetime, timedelta
15 06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4. cmlenz parents: 14 diff changeset	19 import logging
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	20 import os
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	21 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	22 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	23 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	24 from sets import Set as set
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	25 import sys
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	26 import time
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	27
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	28 from trac.env import Environment
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	29 from bitten.model import BuildConfig, Build, BuildStep, BuildLog, Report
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	30 from bitten.queue import BuildQueue
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	31 from bitten.trac_ext.main import BuildSystem
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	32 from bitten.util import beep, xmlio
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	33
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	34 log = logging.getLogger('bitten.master')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	35
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	36 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	37
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	38
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	39 class Master(beep.Listener):
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	40 """BEEP listener implementation for the build master."""
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	41
348 ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	42 def __init__(self, envs, ip, port, build_all=False,
ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	43 adjust_timestamps=False,
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	44 check_interval=DEFAULT_CHECK_INTERVAL):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	45 beep.Listener.__init__(self, ip, port)
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	46 self.profiles[OrchestrationProfileHandler.URI] = \
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	47 OrchestrationProfileHandler
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	48 self.adjust_timestamps = adjust_timestamps
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	49 self.check_interval = check_interval
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	50 self.handlers = {} # Map of connected slaves keyed by name
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	51
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	52 self.queues = []
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	53 for env in envs:
348 ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	54 self.queues.append(BuildQueue(env, build_all=build_all))
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	55
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	56 self.schedule(self.check_interval, self._enqueue_builds)
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	57
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	58 def close(self):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	59 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	60 queue.reset_orphaned_builds()
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	61 beep.Listener.close(self)
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	62
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	63 def _enqueue_builds(self):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	64 self.schedule(self.check_interval, self._enqueue_builds)
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	65
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	66 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	67 queue.populate()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	68
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	69 self.schedule(self.check_interval * 0.2, self._initiate_builds)
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	70
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	71 def _initiate_builds(self):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	72 available_slaves = set([name for name in self.handlers
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	73 if not self.handlers[name].building])
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	74 for idx, queue in enumerate(self.queues[:]):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	75 build, slave = queue.get_next_pending_build(available_slaves)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	76 if build:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	77 self.handlers[slave].send_initiation(queue, build)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	78 available_slaves.discard(slave)
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	79 self.queues.append(self.queues.pop(idx)) # Round robin
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	80
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	81 def register(self, handler):
337 5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	82 if handler.name in self.handlers:
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	83 # The slave is for some reason still registered... this shouldn't
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	84 # happen in theory, but apparently it does in the real world (see
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	85 # #106). We simply unregister it before trying to register it
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	86 # again.
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	87 self.unregister(handler)
5c0625957185 Fix for #106. Thanks to Walter Bell for the report and patch. cmlenz parents: 313 diff changeset	88
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	89 any_match = False
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	90 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	91 if queue.register_slave(handler.name, handler.info):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	92 any_match = True
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	93
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	94 if not any_match:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	95 log.warning('Slave %s does not match any of the configured target '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	96 'platforms', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	97 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	98
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	99 self.handlers[handler.name] = handler
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	100 self.schedule(self.check_interval * 0.2, self._initiate_builds)
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	101
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	102 log.info('Registered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	103 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	104
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	105 def unregister(self, handler):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	106 if handler.name not in self.handlers:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	107 return
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	108
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	109 for queue in self.queues:
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	110 if queue.unregister_slave(handler.name):
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	111 for build in list(Build.select(queue.env, slave=handler.name,
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	112 status=Build.IN_PROGRESS)):
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	113 handler._build_aborted(queue, build)
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	114
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	115 del self.handlers[handler.name]
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	116
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	117 log.info('Unregistered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	118
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	119
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	120 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	121 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	122 the perspective of the build master.
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	123
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	124 An instance of this class is associated with exactly one remote build slave.
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	125 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	126 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	127
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	128 def handle_connect(self):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	129 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	130 assert self.master
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	131 self.name = None
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	132 self.building = False
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	133 self.info = {}
24 cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	134
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	135 def handle_disconnect(self):
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	136 self.master.unregister(self)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	137
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	138 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	139 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	140 elem = xmlio.parse(payload.body)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	141
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	142 if elem.name == 'register':
69 b92d7c7d70fd Record build slave properties in database. cmlenz parents: 66 diff changeset	143 self.name = elem.attr['name']
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	144 self.info[Build.IP_ADDRESS] = self.session.addr[0]
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	145 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	146 if child.name == 'platform':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	147 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	148 self.info[Build.PROCESSOR] = child.attr.get('processor')
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	149 elif child.name == 'os':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	150 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	151 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	152 self.info[Build.OS_VERSION] = child.attr.get('version')
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	153 elif child.name == 'package':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	154 for name, value in child.attr.items():
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	155 if name == 'name':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	156 continue
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	157 self.info[child.attr['name'] + '.' + name] = value
14 1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}. cmlenz parents: 13 diff changeset	158
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	159 if not self.master.register(self):
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	160 raise beep.ProtocolError(550, 'Nothing for you to build here, '
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	161 'please move along')
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	162
29 2adc3480e4aa Some cleanup and additional docstrings. cmlenz parents: 24 diff changeset	163 xml = xmlio.Element('ok')
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	164 self.channel.send_rpy(msgno, beep.Payload(xml))
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	165
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	166 def send_initiation(self, queue, build):
377 a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	167 log.info('Initiating build of %d ("%s" as of [%s]) on slave %s',
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	168 build.id, build.config, build.rev, self.name)
284 ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	169
ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	170 build.slave = self.name
ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	171 build.slave_info.update(self.info)
ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	172 build.status = Build.IN_PROGRESS
ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	173 build.update()
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	174 self.building = True
284 ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	175
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	176 config = BuildConfig.fetch(queue.env, build.config)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	177
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	178 def handle_reply(cmd, msgno, ansno, payload):
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	179 if cmd == 'ERR':
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	180 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	181 elem = xmlio.parse(payload.body)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	182 if elem.name == 'error':
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	183 log.warning('Slave %s refused build request: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	184 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	185 int(elem.attr['code']))
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	186 self.building = False
284 ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	187 self._build_aborted(queue, build)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	188 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	189
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	190 elem = xmlio.parse(payload.body)
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	191 if elem.name != 'proceed':
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	192 raise beep.ProtocolError(500)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	193
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	194 snapshots = queue.snapshots[config.name]
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	195 snapshot = snapshots.get(build.rev)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	196 if not snapshot:
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	197 # Request a snapshot for this build, and schedule a poll
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	198 # function that kicks off the snapshot transmission once the
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	199 # archive has been completely built
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	200 worker = snapshots.create(build.rev)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	201 def _check_snapshot():
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	202 worker.join(.5)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	203 if worker.isAlive():
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	204 self.master.schedule(2, _check_snapshot)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	205 else:
281 33625fa61d6c * If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place. cmlenz parents: 277 diff changeset	206 if self.name not in self.master.handlers:
33625fa61d6c * If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place. cmlenz parents: 277 diff changeset	207 # The slave disconnected while we were building
33625fa61d6c * If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place. cmlenz parents: 277 diff changeset	208 # the archive
33625fa61d6c * If a slave disconnects after the master has started to create a snapshot archive for it, just remain calm and keep the archive in place. cmlenz parents: 277 diff changeset	209 return
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	210 snapshot = snapshots.get(build.rev)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	211 if snapshot is None:
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	212 log.error('Failed to create snapshot archive for '
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	213 '%s@%s', config.path, build.rev)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	214 return
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	215 self.send_snapshot(queue, build, snapshot)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	216 _check_snapshot()
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	217 else:
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	218 self.send_snapshot(queue, build, snapshot)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	219
287 6abd43d0cd8a The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation. cmlenz parents: 284 diff changeset	220 xml = xmlio.parse(config.recipe)
6abd43d0cd8a The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation. cmlenz parents: 284 diff changeset	221 xml.attr['project'] = os.path.basename(queue.env.path)
373 b81c67778df7 Flush pipes to the master before after each step so it shows up in the UI immediately, not just at the end of the build. Closes #18, #96 wbell parents: 368 diff changeset	222 self.channel.send_msg(beep.Payload(xml),
b81c67778df7 Flush pipes to the master before after each step so it shows up in the UI immediately, not just at the end of the build. Closes #18, #96 wbell parents: 368 diff changeset	223 handle_reply=handle_reply, force_flush=True)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	224
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	225 def send_snapshot(self, queue, build, snapshot):
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	226 timestamp_delta = 0
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	227 if self.master.adjust_timestamps:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	228 d = datetime.now() - timedelta(seconds=self.master.check_interval) \
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	229 - datetime.fromtimestamp(build.rev_time)
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	230 log.info('Warping timestamps by %s', d)
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	231 timestamp_delta = d.days * 86400 + d.seconds
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	232
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	233 def handle_reply(cmd, msgno, ansno, payload):
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	234 if cmd == 'ERR':
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	235 if payload.content_type != beep.BEEP_XML:
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	236 raise beep.ProtocolError(500)
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	237 elem = xmlio.parse(payload.body)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	238 if elem.name == 'error':
185 2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds. cmlenz parents: 184 diff changeset	239 log.warning('Slave %s refused to start build: %s (%d)',
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	240 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	241 int(elem.attr['code']))
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	242 self.building = False
284 ddc93fe34cad Fix a bug in the build master (introduced in [289]) where multiple slaves matching the same target platform would potentially start the same build. This is done by setting the build to `IN_PROGRESS` as soon as the build initiation is sent, as opposed to when the slave accepts the build request. cmlenz parents: 281 diff changeset	243 self._build_aborted(queue, build)
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	244
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	245 elif cmd == 'ANS':
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	246 if payload.content_type != beep.BEEP_XML:
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	247 raise beep.ProtocolError(500)
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	248 elem = xmlio.parse(payload.body)
374 446092a2d2fe Don't accept build results from a slave if the build has been invalidated or it's being built by another slave. Closes #100. wbell parents: 373 diff changeset	249
377 a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	250 # verify that the build hasn't been modified out from
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	251 # under us-- we don't want to accept any build information
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	252 # from builds that have been invalidated or claimed by
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	253 # other slaves.
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	254 current_build = Build.fetch(queue.env, build.id)
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	255 if current_build.status != Build.IN_PROGRESS or \
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	256 current_build.slave != self.name:
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	257 raise beep.ProtocolError(550, 'Build %s has been '
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	258 'invalidated, will not accept completed steps from '
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	259 'slave %s' % (build.id, self.name))
a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	260
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	261 if elem.name == 'started':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	262 self._build_started(queue, build, elem, timestamp_delta)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	263 elif elem.name == 'step':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	264 self._build_step_completed(queue, build, elem,
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	265 timestamp_delta)
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	266 elif elem.name == 'completed':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	267 self._build_completed(queue, build, elem, timestamp_delta)
70 ccd03b6f04ef Updated DTD for orchestration profile. cmlenz parents: 69 diff changeset	268 elif elem.name == 'aborted':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	269 self._build_aborted(queue, build)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	270 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	271 build.status = Build.FAILURE
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	272
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	273 elif cmd == 'NUL':
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	274 self.building = False
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	275
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	276 snapshot_name = os.path.basename(snapshot)
1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	277 message = beep.Payload(file(snapshot, 'rb'),
310 0ffdab3a7032 Transmit source archives as `tar.bz2` instead of as zip. See #76. cmlenz parents: 301 diff changeset	278 content_type='application/tar',
0ffdab3a7032 Transmit source archives as `tar.bz2` instead of as zip. See #76. cmlenz parents: 301 diff changeset	279 content_encoding='bzip2',
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	280 content_disposition=snapshot_name)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	281 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	282
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	283 def _build_started(self, queue, build, elem, timestamp_delta=None):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	284 build.started = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	285 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	286 build.started -= timestamp_delta
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	287 build.update()
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	288
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	289 log.info('Slave %s started build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	290 self.name, build.id, build.config, build.rev)
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	291 for listener in BuildSystem(queue.env).listeners:
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	292 listener.build_started(build)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	293
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	294 def _build_step_completed(self, queue, build, elem, timestamp_delta=None):
204 e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	295 log.debug('Slave %s completed step "%s" with status %s', self.name,
e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	296 elem.attr['id'], elem.attr['result'])
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	297
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	298 db = queue.env.get_db_cnx()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	299
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	300 step = BuildStep(queue.env, build=build.id, name=elem.attr['id'],
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	301 description=elem.attr.get('description'))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	302 step.started = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	303 step.stopped = step.started + int(elem.attr['duration'])
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	304 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	305 step.started -= timestamp_delta
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	306 step.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	307 if elem.attr['result'] == 'failure':
377 a465b5dbfecf Tabs and line length. cmlenz parents: 374 diff changeset	308 log.warning('Build %s Step %s failed', build.id, elem.attr['id'])
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	309 step.status = BuildStep.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	310 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	311 step.status = BuildStep.SUCCESS
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	312 step.errors += [error.gettext() for error in elem.children('error')]
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	313 step.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	314
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	315 for idx, log_elem in enumerate(elem.children('log')):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	316 build_log = BuildLog(queue.env, build=build.id, step=step.name,
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	317 generator=log_elem.attr.get('generator'),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	318 orderno=idx)
115 16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	319 for message_elem in log_elem.children('message'):
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	320 build_log.messages.append((message_elem.attr['level'],
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	321 message_elem.gettext()))
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	322 build_log.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	323
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	324 for report_elem in elem.children('report'):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	325 report = Report(queue.env, build=build.id, step=step.name,
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	326 category=report_elem.attr.get('category'),
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	327 generator=report_elem.attr.get('generator'))
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	328 for item_elem in report_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	329 item = {'type': item_elem.name}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	330 item.update(item_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	331 for child_elem in item_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	332 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	333 report.items.append(item)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	334 report.insert(db=db)
116 86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	335
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	336 db.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	337
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	338 def _build_completed(self, queue, build, elem, timestamp_delta=None):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	339 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	340 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	341 build.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	342 if elem.attr['result'] == 'failure':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	343 build.status = Build.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	344 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	345 build.status = Build.SUCCESS
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	346 build.update()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	347
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	348 log.info('Slave %s completed build %d ("%s" as of [%s]) with status %s',
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	349 self.name, build.id, build.config, build.rev,
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	350 build.status == Build.FAILURE and 'FAILURE' or 'SUCCESS')
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	351 for listener in BuildSystem(queue.env).listeners:
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	352 listener.build_completed(build)
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	353
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	354 def _build_aborted(self, queue, build):
204 e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	355 log.info('Slave %s aborted build %d ("%s" as of [%s])',
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	356 self.name, build.id, build.config, build.rev)
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	357 for listener in BuildSystem(queue.env).listeners:
cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	358 listener.build_aborted(build)
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	359
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	360 db = queue.env.get_db_cnx()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	361
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	362 for step in list(BuildStep.select(queue.env, build=build.id, db=db)):
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	363 step.delete(db=db)
fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	364
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	365 build.slave = None
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	366 build.slave_info = {}
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	367 build.started = 0
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	368 build.status = Build.PENDING
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	369 build.update(db=db)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	370
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	371 db.commit()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	372
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	373
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	374 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	375 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	376
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	377 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	378 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	379 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	380 string = string.split('.', 1)[0] # strip out microseconds
312 1016c3d12cbc Fix UTC timestamp handling in build master. cmlenz parents: 310 diff changeset	381 return calendar.timegm(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	382 except ValueError, e:
368 2d58319eea84 Use modern syntax for raising exceptions. cmlenz parents: 348 diff changeset	383 raise ValueError('Invalid ISO date/time %s (%s)' % (string, e))
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	384
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	385 def main():
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	386 """Main entry point for running the build master."""
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	387 from bitten import __version__ as VERSION
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	388 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	389
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	390 # Parse command-line arguments
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	391 parser = OptionParser(usage='usage: %prog [options] ENV_PATHS',
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	392 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	393 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	394 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	395 parser.add_option('-H', '--host', action='store', dest='host',
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	396 metavar='HOSTNAME',
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	397 help='the host name or IP address to bind to')
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	398 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	399 help='write log messages to FILENAME')
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	400 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	401 default=DEFAULT_CHECK_INTERVAL, type='int',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	402 help='poll interval for changeset detection')
348 ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	403 parser.add_option('--build-all', action='store_true', dest='buildall',
ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	404 help='build older revisions even when a build for a '
ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	405 'newer revision has already been performed')
242 372d1de2e3ec * Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`. cmlenz parents: 228 diff changeset	406 parser.add_option('--timewarp', action='store_true', dest='timewarp',
267 7429be6b5720 * Fix some typos. cmlenz parents: 253 diff changeset	407 help='adjust timestamps of builds to be near the '
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	408 'timestamps of the corresponding changesets')
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	409 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	410 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	411 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	412 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	413 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	414 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	415 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	416 options, args = parser.parse_args()
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	417
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	418 if len(args) < 1:
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	419 parser.error('incorrect number of arguments')
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	420
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	421 # Configure logging
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	422 logger = logging.getLogger('bitten')
2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	423 logger.setLevel(options.loglevel)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	424 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	425 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	426 handler.setLevel(logging.WARNING)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	427 else:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	428 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	429 formatter = logging.Formatter('%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	430 handler.setFormatter(formatter)
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	431 logger.addHandler(handler)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	432 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	433 handler = logging.FileHandler(options.logfile)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	434 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	435 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	436 '%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	437 handler.setFormatter(formatter)
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	438 logger.addHandler(handler)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	439
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	440 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	441 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	442 parser.error('port must be an integer in the range 1-65535')
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	443
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	444 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	445 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	446 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	447 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	448 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	449 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	450 except socket.error, e:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	451 log.warning('Reverse host name lookup failed (%s)', e)
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	452 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	453
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	454 envs = []
301 d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	455 env_names = set()
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	456 for env_path in [os.path.normpath(arg) for arg in args]:
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	457 if not os.path.isdir(env_path):
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	458 log.warning('Ignoring %s: not a directory', env_path)
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	459 continue
301 d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	460 env_name = os.path.basename(env_path)
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	461 if env_name in env_names:
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	462 log.warning('Ignoring %s: duplicate project name "%s"', env_path,
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	463 env_name)
287 6abd43d0cd8a The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation. cmlenz parents: 284 diff changeset	464 continue
301 d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	465 env_names.add(env_name)
d486e34084af Normalize the path names of environments passed to the `bitten-master` script. Previously, a path with a trailing slash would cause the master to assume an empty string as the base name of the project. cmlenz parents: 287 diff changeset	466 env = Environment(env_path)
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	467 if BuildSystem(env):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	468 if env.needs_upgrade():
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	469 log.warning('Environment at %s needs to be upgraded', env.path)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	470 continue
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	471 envs.append(env)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	472 if not envs:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	473 log.error('None of the specified environments has support for Bitten')
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	474 sys.exit(2)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	475
348 ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	476 master = Master(envs, host, port, build_all=options.buildall,
ef795ebeac00 Updated change log for [milestone:0.5.3]. cmlenz parents: 337 diff changeset	477 adjust_timestamps=options.timewarp,
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	478 check_interval=options.interval)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	479 try:
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	480 master.run(timeout=5.0)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	481 except KeyboardInterrupt:
34 6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7. cmlenz parents: 33 diff changeset	482 master.quit()
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	483
0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	484 if __name__ == '__main__':
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	485 main()

Mercurial > bitten > bitten-test

annotate bitten/master.py @ 379:0df178e07fdb