bitten/bitten-test: bitten/master.py annotate

annotate bitten/master.py @ 242:372d1de2e3ec

* Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`. * Clean-up of the `optparse` options: use `store_true` instead o `store_const` with `const=True`.

author	cmlenz
date	Sun, 02 Oct 2005 17:59:15 +0000
parents	a8c9dd7e3f71
children	a22ec8fce6c9

rev	line source
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	1 # -- coding: iso8859-1 --
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	2 #
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	3 # Copyright (C) 2005 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	4 # All rights reserved.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	5 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	6 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	7 # you should have received as part of this distribution. The terms
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	8 # are also available at http://bitten.cmlenz.net/wiki/License.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	9
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	10 from datetime import datetime, timedelta
15 06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4. cmlenz parents: 14 diff changeset	11 import logging
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	12 import os
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	13 try:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	14 set
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	15 except NameError:
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	16 from sets import Set as set
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	17 import sys
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	18 import time
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	19
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	20 from trac.env import Environment
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	21 from bitten.model import BuildConfig, Build, BuildStep, BuildLog, Report
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	22 from bitten.queue import BuildQueue
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	23 from bitten.trac_ext.main import BuildSystem
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	24 from bitten.util import archive, beep, xmlio
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	25
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	26 log = logging.getLogger('bitten.master')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	27
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	28 DEFAULT_CHECK_INTERVAL = 120 # 2 minutes
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	29
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	30
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	31 class Master(beep.Listener):
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	32
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	33 def __init__(self, envs, ip, port, adjust_timestamps=False,
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	34 check_interval=DEFAULT_CHECK_INTERVAL):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	35 beep.Listener.__init__(self, ip, port)
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	36 self.profiles[OrchestrationProfileHandler.URI] = \
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	37 OrchestrationProfileHandler
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	38 self.adjust_timestamps = adjust_timestamps
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	39 self.check_interval = check_interval
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	40 self.handlers = {} # Map of connected slaves keyed by name
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	41
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	42 self.queues = []
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	43 for env in envs:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	44 self.queues.append(BuildQueue(env))
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	45
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	46 self.schedule(self.check_interval, self._enqueue_builds)
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	47
083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	48 def close(self):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	49 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	50 queue.reset_orphaned_builds()
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	51 beep.Listener.close(self)
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	52
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	53 def _cleanup(self, when):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	54 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	55 queue.remove_unused_snapshots()
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	56
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	57 def _enqueue_builds(self, when):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	58 self.schedule(self.check_interval, self._enqueue_builds)
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	59
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	60 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	61 queue.populate()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	62
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	63 self.schedule(self.check_interval * 0.2, self._initiate_builds)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	64 self.schedule(self.check_interval * 1.8, self._cleanup)
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	65
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	66 def _initiate_builds(self, when):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	67 available_slaves = set([name for name in self.handlers
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	68 if not self.handlers[name].building])
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	69
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	70 for idx, queue in enumerate(self.queues[:]):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	71 build, slave = queue.get_next_pending_build(available_slaves)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	72 if build:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	73 self.handlers[slave].send_initiation(queue, build)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	74 available_slaves.discard(slave)
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	75 self.queues.append(self.queues.pop(idx)) # Round robin
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	76
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	77 def register(self, handler):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	78 any_match = False
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	79 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	80 if queue.register_slave(handler.name, handler.info):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	81 any_match = True
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	82
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	83 if not any_match:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	84 log.warning('Slave %s does not match any of the configured target '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	85 'platforms', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	86 return False
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	87
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	88 self.handlers[handler.name] = handler
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	89 self.schedule(self.check_interval * 0.2, self._initiate_builds)
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	90
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	91 log.info('Registered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	92 return True
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	93
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	94 def unregister(self, handler):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	95 if handler.name not in self.handlers:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	96 return
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	97
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	98 for queue in self.queues:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	99 queue.unregister_slave(handler.name)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	100 del self.handlers[handler.name]
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	101
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	102 log.info('Unregistered slave "%s"', handler.name)
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	103
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	104
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	105 class OrchestrationProfileHandler(beep.ProfileHandler):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	106 """Handler for communication on the Bitten build orchestration profile from
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	107 the perspective of the build master.
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	108 """
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	109 URI = 'http://bitten.cmlenz.net/beep/orchestration'
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	110
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	111 def handle_connect(self):
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	112 self.master = self.session.listener
591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	113 assert self.master
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	114 self.name = None
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	115 self.building = False
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	116 self.info = {}
24 cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	117
cad6d28b8975 * Proper separation between {{{beep.ProfileHandler}}} instances between different channels. cmlenz parents: 19 diff changeset	118 def handle_disconnect(self):
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	119 self.master.unregister(self)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	120
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	121 def handle_msg(self, msgno, payload):
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	122 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	123 elem = xmlio.parse(payload.body)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	124
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	125 if elem.name == 'register':
69 b92d7c7d70fd Record build slave properties in database. cmlenz parents: 66 diff changeset	126 self.name = elem.attr['name']
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	127 self.info[Build.IP_ADDRESS] = self.session.addr[0]
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	128 for child in elem.children():
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	129 if child.name == 'platform':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	130 self.info[Build.MACHINE] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	131 self.info[Build.PROCESSOR] = child.attr.get('processor')
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	132 elif child.name == 'os':
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	133 self.info[Build.OS_NAME] = child.gettext()
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	134 self.info[Build.OS_FAMILY] = child.attr.get('family')
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	135 self.info[Build.OS_VERSION] = child.attr.get('version')
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	136 elif child.name == 'package':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	137 for name, value in child.attr.items():
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	138 if name == 'name':
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	139 continue
a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	140 self.info[child.attr['name'] + '.' + name] = value
14 1733c601d2f8 Refactored the asyncore loop and shutdown procedure into {{{beep.Initiator}}}. cmlenz parents: 13 diff changeset	141
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	142 if not self.master.register(self):
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	143 xml = xmlio.Element('error', code=550)[
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	144 'Nothing for you to build here, please move along'
42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	145 ]
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	146 self.channel.send_err(msgno, beep.Payload(xml))
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	147 return
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	148
29 2adc3480e4aa Some cleanup and additional docstrings. cmlenz parents: 24 diff changeset	149 xml = xmlio.Element('ok')
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	150 self.channel.send_rpy(msgno, beep.Payload(xml))
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	151
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	152 def send_initiation(self, queue, build):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	153 log.info('Initiating build of "%s" on slave %s', build.config,
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	154 self.name)
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	155 self.building = True
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	156
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	157 def handle_reply(cmd, msgno, ansno, payload):
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	158 if cmd == 'ERR':
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	159 if payload.content_type == beep.BEEP_XML:
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	160 elem = xmlio.parse(payload.body)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	161 if elem.name == 'error':
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	162 log.warning('Slave %s refused build request: %s (%d)',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	163 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	164 int(elem.attr['code']))
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	165 self.building = False
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	166 return
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	167
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	168 elem = xmlio.parse(payload.body)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	169 assert elem.name == 'proceed'
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	170 type = encoding = None
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	171 for child in elem.children('accept'):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	172 type, encoding = child.attr['type'], child.attr.get('encoding')
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	173 if (type, encoding) in (('application/tar', 'gzip'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	174 ('application/tar', 'bzip2'),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	175 ('application/tar', None),
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	176 ('application/zip', None)):
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	177 break
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	178 type = None
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	179 if not type:
5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	180 xml = xmlio.Element('error', code=550)[
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	181 'None of the accepted archive formats supported'
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	182 ]
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	183 self.channel.send_err(beep.Payload(xml))
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	184 self.building = False
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	185 return
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	186 self.send_snapshot(queue, build, type, encoding)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	187
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	188 config = BuildConfig.fetch(queue.env, build.config)
147 395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 135 diff changeset	189 self.channel.send_msg(beep.Payload(config.recipe),
395b67aa072e Build recipes are now stored in the database with the build configuration. This means that it is no longer necessary to store the recipe in the repository. Closes #41. cmlenz parents: 135 diff changeset	190 handle_reply=handle_reply)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	191
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	192 def send_snapshot(self, queue, build, type, encoding):
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	193 timestamp_delta = 0
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	194 if self.master.adjust_timestamps:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	195 d = datetime.now() - timedelta(seconds=self.master.check_interval) \
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	196 - datetime.fromtimestamp(build.rev_time)
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	197 log.info('Warping timestamps by %s' % d)
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	198 timestamp_delta = d.days * 86400 + d.seconds
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	199
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	200 def handle_reply(cmd, msgno, ansno, payload):
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	201 if cmd == 'ERR':
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	202 assert payload.content_type == beep.BEEP_XML
8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	203 elem = xmlio.parse(payload.body)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	204 if elem.name == 'error':
185 2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds. cmlenz parents: 184 diff changeset	205 log.warning('Slave %s refused to start build: %s (%d)',
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	206 self.name, elem.gettext(),
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	207 int(elem.attr['code']))
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	208 self.building = False
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	209
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	210 elif cmd == 'ANS':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	211 assert payload.content_type == beep.BEEP_XML
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	212 elem = xmlio.parse(payload.body)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	213 if elem.name == 'started':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	214 self._build_started(queue, build, elem, timestamp_delta)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	215 elif elem.name == 'step':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	216 self._build_step_completed(queue, build, elem,
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	217 timestamp_delta)
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	218 elif elem.name == 'completed':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	219 self._build_completed(queue, build, elem, timestamp_delta)
70 ccd03b6f04ef Updated DTD for orchestration profile. cmlenz parents: 69 diff changeset	220 elif elem.name == 'aborted':
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	221 self._build_aborted(queue, build)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	222 elif elem.name == 'error':
2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	223 build.status = Build.FAILURE
80 dc1c7fc9b915 Record the output of build steps in the database. See #12. Still need to get better granularity in transmitting the log output from slave to master before #12 can be closed. cmlenz parents: 76 diff changeset	224
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	225 elif cmd == 'NUL':
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	226 self.building = False
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	227
92 2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	228 snapshot_format = {
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	229 ('application/tar', 'bzip2'): 'bzip2',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	230 ('application/tar', 'gzip'): 'gzip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	231 ('application/tar', None): 'tar',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	232 ('application/zip', None): 'zip',
2e090827c63a Delete snapshots for builds that have been completed by all configured target platforms, and are thus no longer needed. Closes #20. cmlenz parents: 90 diff changeset	233 }[(type, encoding)]
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	234 snapshot_path = queue.get_snapshot(build, snapshot_format, create=True)
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	235 snapshot_name = os.path.basename(snapshot_path)
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	236 message = beep.Payload(file(snapshot_path, 'rb'), content_type=type,
88 8ae753a22494 Some refactoring of the BEEP implementation: cmlenz parents: 86 diff changeset	237 content_disposition=snapshot_name,
90 2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17. cmlenz parents: 88 diff changeset	238 content_encoding=encoding)
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	239 self.channel.send_msg(message, handle_reply=handle_reply)
efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	240
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	241 def _build_started(self, queue, build, elem, timestamp_delta=None):
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	242 build.slave = self.name
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	243 build.slave_info.update(self.info)
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	244 build.started = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	245 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	246 build.started -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	247 build.status = Build.IN_PROGRESS
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	248 log.info('Slave %s started build %d ("%s" as of [%s])',
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	249 self.name, build.id, build.config, build.rev)
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	250 build.update()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	251
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	252 def _build_step_completed(self, queue, build, elem, timestamp_delta=None):
204 e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	253 log.debug('Slave %s completed step "%s" with status %s', self.name,
e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	254 elem.attr['id'], elem.attr['result'])
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	255
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	256 db = queue.env.get_db_cnx()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	257
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	258 step = BuildStep(queue.env, build=build.id, name=elem.attr['id'],
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	259 description=elem.attr.get('description'))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	260 step.started = int(_parse_iso_datetime(elem.attr['time']))
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	261 step.stopped = step.started + int(elem.attr['duration'])
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	262 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	263 step.started -= timestamp_delta
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	264 step.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	265 if elem.attr['result'] == 'failure':
135 d2b833187429 Bump up version number. cmlenz parents: 130 diff changeset	266 log.warning('Step failed')
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	267 step.status = BuildStep.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	268 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	269 step.status = BuildStep.SUCCESS
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	270 step.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	271
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	272 for idx, log_elem in enumerate(elem.children('log')):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	273 build_log = BuildLog(queue.env, build=build.id, step=step.name,
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	274 generator=log_elem.attr.get('generator'),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	275 orderno=idx)
115 16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	276 for message_elem in log_elem.children('message'):
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	277 build_log.messages.append((message_elem.attr['level'],
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	278 message_elem.gettext()))
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	279 build_log.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	280
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	281 for report_elem in elem.children('report'):
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	282 report = Report(queue.env, build=build.id, step=step.name,
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	283 category=report_elem.attr.get('category'),
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	284 generator=report_elem.attr.get('generator'))
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	285 for item_elem in report_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	286 item = {'type': item_elem.name}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	287 item.update(item_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	288 for child_elem in item_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	289 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	290 report.items.append(item)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	291 report.insert(db=db)
116 86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	292
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	293 db.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	294
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	295 def _build_completed(self, queue, build, elem, timestamp_delta=None):
119 430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	296 log.info('Slave %s completed build %d ("%s" as of [%s]) with status %s',
430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	297 self.name, build.id, build.config, build.rev,
430c518ebb14 * More logging in master and slave about the build status. cmlenz parents: 117 diff changeset	298 elem.attr['result'])
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	299
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	300 build.stopped = int(_parse_iso_datetime(elem.attr['time']))
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	301 if timestamp_delta:
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	302 build.stopped -= timestamp_delta
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	303 if elem.attr['result'] == 'failure':
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	304 build.status = Build.FAILURE
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	305 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	306 build.status = Build.SUCCESS
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	307 build.update()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	308
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	309 def _build_aborted(self, queue, build):
204 e1a53e70b43f * Make deletion of build steps compatible with PySQLite2: records cannot be deleted inside a `select()` loop. cmlenz parents: 203 diff changeset	310 log.info('Slave %s aborted build %d ("%s" as of [%s])',
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	311 self.name, build.id, build.config, build.rev)
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	312
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	313 db = queue.env.get_db_cnx()
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	314
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	315 for step in list(BuildStep.select(queue.env, build=build.id, db=db)):
184 fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	316 step.delete(db=db)
fbf949f4c706 Allow invalidation of builds from the web interface. This results in the build being reset to ''PENDING'' status, and all build logs, slave information and reports deleted. Basically initiates a rebuild for a specific revision and target platform. cmlenz parents: 163 diff changeset	317
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	318 build.slave = None
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	319 build.started = 0
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	320 build.status = Build.PENDING
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	321 build.slave_info = {}
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	322 build.update(db=db)
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	323
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	324 db.commit()
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	325
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	326
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	327 def _parse_iso_datetime(string):
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	328 """Minimal parser for ISO date-time strings.
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	329
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	330 Return the time as floating point number. Only handles UTC timestamps
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	331 without time zone information."""
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	332 try:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	333 string = string.split('.', 1)[0] # strip out microseconds
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	334 secs = time.mktime(time.strptime(string, '%Y-%m-%dT%H:%M:%S'))
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	335 tzoffset = time.timezone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	336 if time.daylight:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	337 tzoffset = time.altzone
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	338 return secs - tzoffset
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	339 except ValueError, e:
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	340 raise ValueError, 'Invalid ISO date/time %s (%s)' % (string, e)
01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	341
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	342 def main():
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	343 from bitten import __version__ as VERSION
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	344 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	345
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	346 # Parse command-line arguments
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	347 parser = OptionParser(usage='usage: %prog [options] ENV_PATHS',
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	348 version='%%prog %s' % VERSION)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	349 parser.add_option('-p', '--port', action='store', type='int', dest='port',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	350 help='port number to use')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	351 parser.add_option('-H', '--host', action='store', dest='host',
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	352 metavar='HOSTNAME',
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	353 help='the host name or IP address to bind to')
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	354 parser.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	355 help='write log messages to FILENAME')
95 1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	356 parser.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	357 default=DEFAULT_CHECK_INTERVAL, type='int',
1984b2e01998 Make the repository poll interval configurable. It now defaults to 2 minutes instead of 10 seconds. cmlenz parents: 94 diff changeset	358 help='poll interval for changeset detection')
242 372d1de2e3ec * Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`. cmlenz parents: 228 diff changeset	359 parser.add_option('--timewarp', action='store_true', dest='timewarp',
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	360 help='adjust timestamps of builds to be neat the '
091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	361 'timestamps of the corresponding changesets')
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	362 parser.add_option('--debug', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	363 const=logging.DEBUG, help='enable debugging output')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	364 parser.add_option('-v', '--verbose', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	365 const=logging.INFO, help='print as much as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	366 parser.add_option('-q', '--quiet', action='store_const', dest='loglevel',
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	367 const=logging.ERROR, help='print as little as possible')
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	368 parser.set_defaults(port=7633, loglevel=logging.WARNING)
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	369 options, args = parser.parse_args()
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	370
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	371 if len(args) < 1:
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	372 parser.error('incorrect number of arguments')
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	373
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	374 # Configure logging
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	375 logger = logging.getLogger('bitten')
2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	376 logger.setLevel(options.loglevel)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	377 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	378 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	379 handler.setLevel(logging.WARNING)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	380 else:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	381 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	382 formatter = logging.Formatter('%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	383 handler.setFormatter(formatter)
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	384 logger.addHandler(handler)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	385 if options.logfile:
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	386 handler = logging.FileHandler(options.logfile)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	387 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	388 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	389 '%(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	390 handler.setFormatter(formatter)
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 152 diff changeset	391 logger.addHandler(handler)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	392
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	393 port = options.port
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	394 if not (1 <= port <= 65535):
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	395 parser.error('port must be an integer in the range 1-65535')
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	396
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	397 host = options.host
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	398 if not host:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	399 import socket
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	400 ip = socket.gethostbyname(socket.gethostname())
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	401 try:
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	402 host = socket.gethostbyaddr(ip)[0]
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	403 except socket.error, e:
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 92 diff changeset	404 log.warning('Reverse host name lookup failed (%s)', e)
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	405 host = ip
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	406
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	407 envs = []
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	408 for arg in args:
228 a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	409 if not os.path.isdir(arg):
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	410 log.warning('Ignoring %s: not a directory', arg)
a8c9dd7e3f71 * Cleanup and documentation for the `BuildQueue` class added in [236]. cmlenz parents: 227 diff changeset	411 continue
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	412 env = Environment(arg)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	413 if BuildSystem(env):
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	414 if env.needs_upgrade():
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	415 log.warning('Environment at %s needs to be upgraded', env.path)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	416 continue
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	417 envs.append(env)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	418 if not envs:
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	419 log.error('None of the specified environments has support for Bitten')
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	420 sys.exit(2)
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	421
014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	422 master = Master(envs, host, port, adjust_timestamps=options.timewarp,
130 091ead7d2876 Add a {{{--timewarp}}} option to the build master. If provided, the build master will adjust the timestamps of all builds to shortly after the time of the corresponding changeset. This is mostly for building the history of a project while keeping a chronological association between changeset and build. cmlenz parents: 127 diff changeset	423 check_interval=options.interval)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	424 try:
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	425 master.run(timeout=5.0)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	426 except KeyboardInterrupt:
34 6da9468a6879 The build master now gracefully exits by first terminating all active sessions. Fixes #7. cmlenz parents: 33 diff changeset	427 master.quit()
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	428
0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	429 if __name__ == '__main__':
33 d8d44216258a Exit the slave script when the master disconnects; and other minor fixes. cmlenz parents: 31 diff changeset	430 main()

Mercurial > bitten > bitten-test

annotate bitten/master.py @ 242:372d1de2e3ec