bitten/bitten-test: bitten/master.py annotate

annotate bitten/master.py @ 872:f320205ca1f9

Ensure that master only sends utf-8 bodies (especially errors are prone to contain strings of unknown origin, potentially unicode).

author	osimons
date	Tue, 19 Oct 2010 20:55:13 +0000
parents	59acaa8b52c0
children

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 377 diff changeset	1 # -- coding: utf-8 --
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	2 #
832 7c80375d4817 Updated copyright to 2010. osimons parents: 826 diff changeset	3 # Copyright (C) 2007-2010 Edgewall Software
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 402 diff changeset	4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	5 # All rights reserved.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	6 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 157 diff changeset	8 # you should have received as part of this distribution. The terms
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 402 diff changeset	9 # are also available at http://bitten.edgewall.org/wiki/License.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	10
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	11 """Build master implementation."""
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 312 diff changeset	12
312 1016c3d12cbc Fix UTC timestamp handling in build master. cmlenz parents: 310 diff changeset	13 import calendar
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	14 import re
47 083e848088ee * Improvements to the model classes, and a couple of unit tests. cmlenz parents: 45 diff changeset	15 import time
629 f3bb52da9e3c 0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build. osimons parents: 611 diff changeset	16 from StringIO import StringIO
18 591a5a836ecc * {{{beep.Listener}}} now has an event loop (based on code mostly from medusa) cmlenz parents: 15 diff changeset	17
629 f3bb52da9e3c 0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build. osimons parents: 611 diff changeset	18 from trac.attachment import Attachment
577 dcee8ff20e82 0.6dev: The `logs_dir` option is not a real `PathOption` as for that it would need to support config-relative path with possibly chained configs following trac:ticket:8358 (for 0.11.5). The fix just makes it a regular `Option` to keep current behaviour for all Trac 0.11+ versions. osimons parents: 568 diff changeset	19 from trac.config import BoolOption, IntOption, Option
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	20 from trac.core import *
629 f3bb52da9e3c 0.6dev: Adding support for attachments to configurations and build - full web implementation that mirrors what is available in Ticket and Wiki. Also added a new generic `<attach/>` command that enables attaching files to be part of a recipe and uploaded by slaves as part of build. osimons parents: 611 diff changeset	21 from trac.resource import ResourceNotFound
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	22 from trac.web import IRequestHandler, RequestDone
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	23
654 cc1fefd5d814 0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz. osimons parents: 649 diff changeset	24 from bitten import PROTOCOL_VERSION
554 2c27f3581100 Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376 dfraser parents: 542 diff changeset	25 from bitten.model import BuildConfig, Build, BuildStep, BuildLog, Report, \
2c27f3581100 Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376 dfraser parents: 542 diff changeset	26 TargetPlatform
2c27f3581100 Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376 dfraser parents: 542 diff changeset	27
410 7930cdd83d13 More restructuring: got rid of the `trac_ext` subpackage, which makes no sense now that the master is also coupled to Trac. cmlenz parents: 408 diff changeset	28 from bitten.main import BuildSystem
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	29 from bitten.queue import BuildQueue
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	30 from bitten.recipe import Recipe
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	31 from bitten.util import xmlio
83 42970c14524a Perform slave/platform matching at slave registration. Use builtin {{{set}}} type on Python >= 2.4. cmlenz parents: 82 diff changeset	32
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	33 __all__ = ['BuildMaster']
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	34 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 410 diff changeset	35
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	36
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	37 HTTP_BAD_REQUEST = 400
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	38 HTTP_FORBIDDEN = 403
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	39 HTTP_NOT_FOUND = 404
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	40 HTTP_METHOD_NOT_ALLOWED = 405
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	41 HTTP_CONFLICT = 409
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	42
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	43
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	44 class BuildMaster(Component):
542 cacf018eb190 Small documentation fixes. wbell parents: 516 diff changeset	45 """Trac request handler implementation for the build master."""
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	46
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	47 implements(IRequestHandler)
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	48
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	49 # Configuration options
51 5caccd7b247e Proper archive format negotiation; improved representation of parsed XML content in {{{bitten.util.xmlio}}}. cmlenz parents: 49 diff changeset	50
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	51 adjust_timestamps = BoolOption('bitten', 'adjust_timestamps', False, doc=
432 74c51f648466 Started some tests for the new admin interface. cmlenz parents: 426 diff changeset	52 """Whether the timestamps of builds should be adjusted to be close
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	53 to the timestamps of the corresponding changesets.""")
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	54
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	55 build_all = BoolOption('bitten', 'build_all', False, doc=
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	56 """Whether to request builds of older revisions even if a younger
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	57 revision has already been built.""")
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	58
468 44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	59 stabilize_wait = IntOption('bitten', 'stabilize_wait', 0, doc=
44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	60 """The time in seconds to wait for the repository to stabilize before
44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	61 queuing up a new build. This allows time for developers to check in
44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	62 a group of related changes back to back without spawning multiple
44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	63 builds.""")
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	64
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	65 slave_timeout = IntOption('bitten', 'slave_timeout', 3600, doc=
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	66 """The time in seconds after which a build is cancelled if the slave
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	67 does not report progress.""")
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	68
577 dcee8ff20e82 0.6dev: The `logs_dir` option is not a real `PathOption` as for that it would need to support config-relative path with possibly chained configs following trac:ticket:8358 (for 0.11.5). The fix just makes it a regular `Option` to keep current behaviour for all Trac 0.11+ versions. osimons parents: 568 diff changeset	69 logs_dir = Option('bitten', 'logs_dir', "log/bitten", doc=
516 2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 494 diff changeset	70 """The directory on the server in which client log files will be stored.""")
2f3b7c17d3c3 Switch to storing log messages in files rather than in database rows: dfraser parents: 494 diff changeset	71
557 b4d3d9cbf200 Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373 dfraser parents: 554 diff changeset	72 quick_status = BoolOption('bitten', 'quick_status', False, doc=
806 b62562828364 Improved documentation for the quick-status in navigation. See performance discussion at #536. osimons parents: 804 diff changeset	73 """Whether to show the current build status within the Trac main
b62562828364 Improved documentation for the quick-status in navigation. See performance discussion at #536. osimons parents: 804 diff changeset	74 navigation bar. '''Note:''' The feature requires expensive database and
b62562828364 Improved documentation for the quick-status in navigation. See performance discussion at #536. osimons parents: 804 diff changeset	75 repository checks for every page request, and should not be enabled
b62562828364 Improved documentation for the quick-status in navigation. See performance discussion at #536. osimons parents: 804 diff changeset	76 if the project has a large repository or uses a non-Subversion
b62562828364 Improved documentation for the quick-status in navigation. See performance discussion at #536. osimons parents: 804 diff changeset	77 repository such as Mercurial or Git.""")
557 b4d3d9cbf200 Alter the appearance of the ''Build Status'' button, to show the current build status [eblot] - fixes #373 dfraser parents: 554 diff changeset	78
568 84a45f7c9833 0.6dev: Adding Bitten version info to About Trac. osimons parents: 557 diff changeset	79 def __init__(self):
84a45f7c9833 0.6dev: Adding Bitten version info to About Trac. osimons parents: 557 diff changeset	80 self.env.systeminfo.append(('Bitten',
84a45f7c9833 0.6dev: Adding Bitten version info to About Trac. osimons parents: 557 diff changeset	81 __import__('bitten', ['__version__']).__version__))
84a45f7c9833 0.6dev: Adding Bitten version info to About Trac. osimons parents: 557 diff changeset	82
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	83 # IRequestHandler methods
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	84
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	85 def match_request(self, req):
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	86 match = re.match(r'/builds(?:/(\d+)(?:/(\w+)/([^/]+)?)?)?$',
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	87 req.path_info)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	88 if match:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	89 if match.group(1):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	90 req.args['id'] = match.group(1)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	91 req.args['collection'] = match.group(2)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	92 req.args['member'] = match.group(3)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	93 return True
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	94
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	95 def process_request(self, req):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	96 req.perm.assert_permission('BUILD_EXEC')
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	97
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	98 if 'trac_auth' in req.incookie:
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	99 slave_token = req.incookie['trac_auth'].value
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	100 else:
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	101 slave_token = req.session.sid
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	102
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	103 if 'id' not in req.args:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	104 if req.method != 'POST':
715 cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459. osimons parents: 701 diff changeset	105 self._send_response(req,
cd4405466a77 Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459. osimons parents: 701 diff changeset	106 body='Only POST allowed for build creation.')
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	107 return self._process_build_creation(req, slave_token)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	108
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	109 build = Build.fetch(self.env, req.args['id'])
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	110 if not build:
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	111 self._send_error(req, HTTP_NOT_FOUND,
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	112 'No such build (%s)' % req.args['id'])
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	113
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	114 build_token = build.slave_info.get('token', '')
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	115 if build_token != slave_token:
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	116 self._send_error(req, HTTP_CONFLICT,
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	117 'Token mismatch (wrong slave): slave=%s, build=%s' \
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	118 % (slave_token, build_token))
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	119
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	120 config = BuildConfig.fetch(self.env, build.config)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	121
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	122 if not req.args['collection']:
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	123 if req.method == 'DELETE':
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	124 return self._process_build_cancellation(req, config, build)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	125 else:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	126 return self._process_build_initiation(req, config, build)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	127
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	128 if req.method != 'POST':
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	129 self._send_error(req, HTTP_METHOD_NOT_ALLOWED,
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	130 'Method %s not allowed' % req.method)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	131
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	132 if req.args['collection'] == 'steps':
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	133 return self._process_build_step(req, config, build)
835 59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	134 elif req.args['collection'] == 'attach':
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	135 return self._process_attachment(req, config, build)
785 8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	136 elif req.args['collection'] == 'keepalive':
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	137 return self._process_keepalive(req, config, build)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	138 else:
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	139 self._send_error(req, HTTP_NOT_FOUND,
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	140 "No such collection '%s'" % req.args['collection'])
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	141
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	142 # Internal methods
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	143
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	144 def _send_response(self, req, code=200, body='', headers=None):
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	145 """ Formats and sends the response, raising ``RequestDone``. """
872 f320205ca1f9 Ensure that master only sends utf-8 bodies (especially errors are prone to contain strings of unknown origin, potentially unicode). osimons parents: 835 diff changeset	146 if isinstance(body, unicode):
f320205ca1f9 Ensure that master only sends utf-8 bodies (especially errors are prone to contain strings of unknown origin, potentially unicode). osimons parents: 835 diff changeset	147 body = body.encode('utf-8')
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	148 req.send_response(code)
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	149 headers = headers or {}
701 492c18db8f45 Set the `'Content-Length'` header on master-slave communication. Needed to be http/1.1 compatible. Fixes #475. osimons parents: 654 diff changeset	150 headers.setdefault('Content-Length', len(body))
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	151 for header in headers:
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	152 req.send_header(header, headers[header])
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	153 req.write(body)
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	154 raise RequestDone
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	155
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	156 def _send_error(self, req, code=500, message=''):
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	157 """ Formats and sends the error, raising ``RequestDone``. """
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	158 headers = {'Content-Type': 'text/plain',
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	159 'Content-Length': str(len(message))}
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	160 self._send_response(req, code, body=message, headers=headers)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	161
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	162 def _process_build_creation(self, req, slave_token):
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	163 queue = BuildQueue(self.env, build_all=self.build_all,
468 44c2b4ac6157 Add stabilization time parameter to build master. Closes #189. Many thanks to Allen Bierbaum for the patch. cmlenz parents: 466 diff changeset	164 stabilize_wait=self.stabilize_wait,
419 b72802dc0632 Fix resetting of builds when multiple slaves are building simultaneously, and implement the `slave_timeout` trac.ini option. cmlenz parents: 411 diff changeset	165 timeout=self.slave_timeout)
804 4c73a3cea9f5 Basic Trac 0.12 support, supporting just a `(default)` repository - essentially Trac 0.11 behaviour. Thanks to those that have contributed to #480 to get this working and tested. osimons parents: 785 diff changeset	166 try:
4c73a3cea9f5 Basic Trac 0.12 support, supporting just a `(default)` repository - essentially Trac 0.11 behaviour. Thanks to those that have contributed to #480 to get this working and tested. osimons parents: 785 diff changeset	167 queue.populate()
4c73a3cea9f5 Basic Trac 0.12 support, supporting just a `(default)` repository - essentially Trac 0.11 behaviour. Thanks to those that have contributed to #480 to get this working and tested. osimons parents: 785 diff changeset	168 except AssertionError, e:
4c73a3cea9f5 Basic Trac 0.12 support, supporting just a `(default)` repository - essentially Trac 0.11 behaviour. Thanks to those that have contributed to #480 to get this working and tested. osimons parents: 785 diff changeset	169 self.log.error(e.message, exc_info=True)
4c73a3cea9f5 Basic Trac 0.12 support, supporting just a `(default)` repository - essentially Trac 0.11 behaviour. Thanks to those that have contributed to #480 to get this working and tested. osimons parents: 785 diff changeset	170 self._send_error(req, HTTP_BAD_REQUEST, e.message)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	171
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	172 try:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	173 elem = xmlio.parse(req.read())
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	174 except xmlio.ParseError, e:
492 56f0ad35c60a Log errors when build master encounters an XML parse error. cmlenz parents: 479 diff changeset	175 self.log.error('Error parsing build initialization request: %s', e,
56f0ad35c60a Log errors when build master encounters an XML parse error. cmlenz parents: 479 diff changeset	176 exc_info=True)
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	177 self._send_error(req, HTTP_BAD_REQUEST, 'XML parser error')
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	178
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	179 slave_version = int(elem.attr.get('version', 1))
785 8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	180
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	181 # FIXME: Remove version compatibility code.
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	182 # The initial difference between protocol version 3 and 4 is that
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	183 # the master allows keepalive requests-- the master must be
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	184 # at least 4 before slaves supporting version 4 are allowed. When
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	185 # the first force master/slave upgrade requirement comes in
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	186 # (or we bump the) version number again, remove this code.
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	187 if slave_version == 3 and PROTOCOL_VERSION == 4:
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	188 self.log.info('Allowing slave version %d to process build for '
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	189 'compatibility. Upgrade slave to support build '
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	190 'keepalives.', slave_version)
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	191 elif slave_version != PROTOCOL_VERSION:
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	192 self._send_error(req, HTTP_BAD_REQUEST,
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	193 "Master-Slave version mismatch: master=%d, slave=%d" % \
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	194 (PROTOCOL_VERSION, slave_version))
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	195
426 a08fa5ce9aff Applied patch by Sam Steele for #167. cmlenz parents: 420 diff changeset	196 slavename = elem.attr['name']
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	197 properties = {'name': slavename, Build.IP_ADDRESS: req.remote_addr,
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	198 Build.TOKEN: slave_token}
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	199 self.log.info('Build slave %r connected from %s with token %s',
eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 645 diff changeset	200 slavename, req.remote_addr, slave_token)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	201
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	202 for child in elem.children():
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	203 if child.name == 'platform':
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	204 properties[Build.MACHINE] = child.gettext()
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	205 properties[Build.PROCESSOR] = child.attr.get('processor')
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	206 elif child.name == 'os':
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	207 properties[Build.OS_NAME] = child.gettext()
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	208 properties[Build.OS_FAMILY] = child.attr.get('family')
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	209 properties[Build.OS_VERSION] = child.attr.get('version')
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	210 elif child.name == 'package':
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	211 for name, value in child.attr.items():
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	212 if name == 'name':
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	213 continue
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	214 properties[child.attr['name'] + '.' + name] = value
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	215
444 22d6a7da8777 Another minor improvement to logging in the build master. cmlenz parents: 436 diff changeset	216 self.log.debug('Build slave configuration: %r', properties)
22d6a7da8777 Another minor improvement to logging in the build master. cmlenz parents: 436 diff changeset	217
426 a08fa5ce9aff Applied patch by Sam Steele for #167. cmlenz parents: 420 diff changeset	218 build = queue.get_build_for_slave(slavename, properties)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	219 if not build:
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	220 self._send_response(req, 204, '', {})
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	221
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	222 self._send_response(req, 201, 'Build pending', headers={
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	223 'Content-Type': 'text/plain',
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	224 'Location': req.abs_href.builds(build.id)})
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	225
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	226 def _process_build_cancellation(self, req, config, build):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	227 self.log.info('Build slave %r cancelled build %d', build.slave,
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	228 build.id)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	229 build.status = Build.PENDING
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	230 build.slave = None
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	231 build.slave_info = {}
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	232 build.started = 0
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	233 db = self.env.get_db_cnx()
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	234 for step in list(BuildStep.select(self.env, build=build.id, db=db)):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	235 step.delete(db=db)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	236 build.update(db=db)
745 91aabd647610 Delete attachments when builds are cancelled/invalidated. wbell parents: 715 diff changeset	237
91aabd647610 Delete attachments when builds are cancelled/invalidated. wbell parents: 715 diff changeset	238 Attachment.delete_all(self.env, 'build', build.resource.id, db)
91aabd647610 Delete attachments when builds are cancelled/invalidated. wbell parents: 715 diff changeset	239
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	240 db.commit()
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	241
458 c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	242 for listener in BuildSystem(self.env).listeners:
c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	243 listener.build_aborted(build)
c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	244
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	245 self._send_response(req, 204, '', {})
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	246
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	247 def _process_build_initiation(self, req, config, build):
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	248 self.log.info('Build slave %r initiated build %d', build.slave,
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 419 diff changeset	249 build.id)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	250 build.started = int(time.time())
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	251 build.last_activity = build.started
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	252 build.update()
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	253
458 c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	254 for listener in BuildSystem(self.env).listeners:
c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	255 listener.build_started(build)
c9ac97df8a5e Fix build listener invocation. cmlenz parents: 444 diff changeset	256
287 6abd43d0cd8a The build slave now stores snapshot archives and the corresponding work directories in project folders of the main work folder, to keep build configurations from different projects that share the same name separate. This also requires transmitting the project name (simply the name of the environment directory) with the build initiation. cmlenz parents: 284 diff changeset	257 xml = xmlio.parse(config.recipe)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	258 xml.attr['path'] = config.path
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	259 xml.attr['revision'] = build.rev
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 463 diff changeset	260 xml.attr['config'] = config.name
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 463 diff changeset	261 xml.attr['build'] = str(build.id)
554 2c27f3581100 Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376 dfraser parents: 542 diff changeset	262 target_platform = TargetPlatform.fetch(self.env, build.platform)
2c27f3581100 Supply the target platform as a variable `platform` for build recipes, and adjust test accordingly - fixes #376 dfraser parents: 542 diff changeset	263 xml.attr['platform'] = target_platform.name
611 294641e84e89 0.6dev: Adding `${name}` and `${basedir}` (#325) for recipe substitution. Updated docs + new test. osimons parents: 577 diff changeset	264 xml.attr['name'] = build.slave
835 59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	265 xml.attr['form_token'] = req.form_token # For posting attachments
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	266 body = str(xml)
374 446092a2d2fe Don't accept build results from a slave if the build has been invalidated or it's being built by another slave. Closes #100. wbell parents: 373 diff changeset	267
473 b1d346c4e539 Only accept build results from the slave that's supposed to be processing this build. Additional logging. wbell parents: 468 diff changeset	268 self.log.info('Build slave %r initiated build %d', build.slave,
b1d346c4e539 Only accept build results from the slave that's supposed to be processing this build. Additional logging. wbell parents: 468 diff changeset	269 build.id)
b1d346c4e539 Only accept build results from the slave that's supposed to be processing this build. Additional logging. wbell parents: 468 diff changeset	270
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	271 # create the first step, mark it as in-progress.
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	272
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	273 recipe = Recipe(xmlio.parse(config.recipe))
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	274 stepname = recipe.__iter__().next().id
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	275
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	276 step = self._start_new_step(build, stepname)
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	277 step.insert()
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	278
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	279 self._send_response(req, 200, body, headers={
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	280 'Content-Type': 'application/x-bitten+xml',
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	281 'Content-Length': str(len(body)),
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	282 'Content-Disposition':
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	283 'attachment; filename=recipe_%s_r%s.xml' %
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	284 (config.name, build.rev)})
227 014bc6c29dff * Factor build queue logic into a class separate from the build master. cmlenz parents: 213 diff changeset	285
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	286 def _process_build_step(self, req, config, build):
a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	287 try:
a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	288 elem = xmlio.parse(req.read())
a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	289 except xmlio.ParseError, e:
492 56f0ad35c60a Log errors when build master encounters an XML parse error. cmlenz parents: 479 diff changeset	290 self.log.error('Error parsing build step result: %s', e,
56f0ad35c60a Log errors when build master encounters an XML parse error. cmlenz parents: 479 diff changeset	291 exc_info=True)
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	292 self._send_error(req, HTTP_BAD_REQUEST, 'XML parser error')
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	293 stepname = elem.attr['step']
632 01c9848950d5 0.6dev: Cleaning all usage of tabs (replaced with spaces). osimons parents: 629 diff changeset	294
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	295 # we should have created this step previously; if it hasn't,
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	296 # the master and slave are processing steps out of order.
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	297 step = BuildStep.fetch(self.env, build=build.id, name=stepname)
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	298 if not step:
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	299 self._send_error(req, HTTP_CONFLICT, 'Build step has not been created.')
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	300
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	301 recipe = Recipe(xmlio.parse(config.recipe))
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	302 index = None
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	303 current_step = None
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	304 for num, recipe_step in enumerate(recipe):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	305 if recipe_step.id == stepname:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	306 index = num
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	307 current_step = recipe_step
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	308 if index is None:
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	309 self._send_error(req, HTTP_FORBIDDEN,
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	310 'No such build step' % stepname)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	311 last_step = index == num
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	312
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 492 diff changeset	313 self.log.debug('Slave %s (build %d) completed step %d (%s) with '
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 492 diff changeset	314 'status %s', build.slave, build.id, index, stepname,
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 492 diff changeset	315 elem.attr['status'])
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	316
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	317 db = self.env.get_db_cnx()
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	318
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	319 step.stopped = int(time.time())
758 d11ef8024d7c Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master. wbell parents: 754 diff changeset	320
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	321 if elem.attr['status'] == 'failure':
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	322 self.log.warning('Build %s step %s failed', build.id, stepname)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	323 step.status = BuildStep.FAILURE
459 6537f01b4730 Clarify `last_step` logic in master. cmlenz parents: 458 diff changeset	324 if current_step.onerror == 'fail':
6537f01b4730 Clarify `last_step` logic in master. cmlenz parents: 458 diff changeset	325 last_step = True
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	326 else:
5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	327 step.status = BuildStep.SUCCESS
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 267 diff changeset	328 step.errors += [error.gettext() for error in elem.children('error')]
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	329
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	330 # TODO: step.update(db=db)
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	331 step.delete(db=db)
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	332 step.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	333
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	334 # Collect log messages from the request body
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	335 for idx, log_elem in enumerate(elem.children('log')):
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	336 build_log = BuildLog(self.env, build=build.id, step=stepname,
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	337 generator=log_elem.attr.get('generator'),
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	338 orderno=idx)
115 16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	339 for message_elem in log_elem.children('message'):
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	340 build_log.messages.append((message_elem.attr['level'],
16d69eb6e047 Add support for XML fragments to the {{{xmlio}}} module, so that build output and reports don't need to be nested in a meaningless element (such as {{{<log type="distutils"><messages><message ...>}}}). cmlenz parents: 112 diff changeset	341 message_elem.gettext()))
112 a38eabd4b6e1 * Store build logs in a structured way, for example to highlight messages on the error stream. cmlenz parents: 110 diff changeset	342 build_log.insert(db=db)
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	343
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	344 # Collect report data from the request body
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	345 for report_elem in elem.children('report'):
401 a10942252ebc Use POST for submitting step results, instead of PUT. cmlenz parents: 392 diff changeset	346 report = Report(self.env, build=build.id, step=stepname,
213 25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	347 category=report_elem.attr.get('category'),
25f84dd9f159 * Refactoring of build recipes, the file format has changed slightly: cmlenz parents: 204 diff changeset	348 generator=report_elem.attr.get('generator'))
203 e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	349 for item_elem in report_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	350 item = {'type': item_elem.name}
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	351 item.update(item_elem.attr)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	352 for child_elem in item_elem.children():
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	353 item[child_elem.name] = child_elem.gettext()
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	354 report.items.append(item)
e6ddca1e5712 Huge refactoring to remove dependency on BDB XML. Report data is now stored in the Trac database (SQLite/PostgreSQL). cmlenz parents: 202 diff changeset	355 report.insert(db=db)
116 86439c2aa6d6 Store report data in BDB XML database. Closes #31. cmlenz parents: 115 diff changeset	356
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	357 # If this was the last step in the recipe we mark the build as
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	358 # completed otherwise just update last_activity
459 6537f01b4730 Clarify `last_step` logic in master. cmlenz parents: 458 diff changeset	359 if last_step:
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	360 self.log.info('Slave %s completed build %d ("%s" as of [%s])',
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	361 build.slave, build.id, build.config, build.rev)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	362 build.stopped = step.stopped
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	363 build.last_activity = build.stopped
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	364
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	365 # Determine overall outcome of the build by checking the outcome
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	366 # of the individual steps against the "onerror" specification of
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	367 # each step in the recipe
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	368 for num, recipe_step in enumerate(recipe):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	369 step = BuildStep.fetch(self.env, build.id, recipe_step.id)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	370 if step.status == BuildStep.FAILURE:
754 545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping. wbell parents: 748 diff changeset	371 if recipe_step.onerror == 'fail' or \
545be0c8f405 Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping. wbell parents: 748 diff changeset	372 recipe_step.onerror == 'continue':
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	373 build.status = Build.FAILURE
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	374 break
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	375 else:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	376 build.status = Build.SUCCESS
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	377
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	378 build.update(db=db)
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	379 else:
762 5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	380 build.last_activity = step.stopped
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	381 build.update(db=db)
5f0cfee44540 Add new last_activity field to build. I considered reusing stopped, but this seemed cleaner and more obvious, which seems like the right way to go. wbell parents: 761 diff changeset	382
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	383 # start the next step.
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	384 for num, recipe_step in enumerate(recipe):
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	385 if num == index + 1:
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	386 next_step = recipe_step
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	387 if next_step is None:
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	388 self._send_error(req, HTTP_FORBIDDEN,
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	389 'Unable to find step after ' % stepname)
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	390
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	391 step = self._start_new_step(build, next_step.id)
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	392 step.insert(db=db)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	393
200 692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	394 db.commit()
692924ffed80 Changes to the BDB XML report store to support transactions. Closes #47. cmlenz parents: 197 diff changeset	395
459 6537f01b4730 Clarify `last_step` logic in master. cmlenz parents: 458 diff changeset	396 if last_step:
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	397 for listener in BuildSystem(self.env).listeners:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	398 listener.build_completed(build)
253 cda723f3ac31 Provide hooks for build notification. Closes #62. cmlenz parents: 245 diff changeset	399
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 379 diff changeset	400 body = 'Build step processed'
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	401 self._send_response(req, 201, body, {
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	402 'Content-Type': 'text/plain',
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	403 'Content-Length': str(len(body)),
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	404 'Location': req.abs_href.builds(
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 632 diff changeset	405 build.id, 'steps', stepname)})
109 5bf22bb87915 Transmit build log and generated data back to the build master in XML format. Closes #23. cmlenz parents: 96 diff changeset	406
835 59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	407 def _process_attachment(self, req, config, build):
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	408 resource_id = req.args['member'] == 'config' \
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	409 and build.config or build.resource.id
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	410 upload = req.args['file']
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	411 if not upload.file:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	412 send_error(req, message="Attachment not received.")
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	413 self.log.debug('Received attachment %s for attaching to build:%s',
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	414 upload.filename, resource_id)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	415
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	416 # Determine size of file
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	417 upload.file.seek(0, 2) # to the end
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	418 size = upload.file.tell()
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	419 upload.file.seek(0) # beginning again
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	420
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	421 # Delete attachment if it already exists
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	422 try:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	423 old_attach = Attachment(self.env, 'build',
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	424 parent_id=resource_id, filename=upload.filename)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	425 old_attach.delete()
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	426 except ResourceNotFound:
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	427 pass
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	428
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	429 # Save new attachment
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	430 attachment = Attachment(self.env, 'build', parent_id=resource_id)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	431 attachment.description = req.args.get('description', '')
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	432 attachment.author = req.authname
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	433 attachment.insert(upload.filename, upload.file, size)
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	434
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	435 self._send_response(req, 201, 'Attachment created', headers={
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	436 'Content-Type': 'text/plain',
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	437 'Content-Length': str(len('Attachment created'))})
59acaa8b52c0 Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack... osimons parents: 832 diff changeset	438
785 8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	439 def _process_keepalive(self, req, config, build):
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	440 build.last_activity = int(time.time())
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	441 build.update()
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	442
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	443 self.log.info('Slave %s build %d keepalive ("%s" as of [%s])',
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	444 build.slave, build.id, build.config, build.rev)
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	445
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	446 body = 'Keepalive processed'
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	447 self._send_response(req, 200, body, {
8e76b8f6310a Adding keepalives to the bitten client/server protocol. wbell parents: 762 diff changeset	448 'Content-Type': 'text/plain',
826 77c3de8f5c46 Stop sending the 'Location' header on keepalive responses, which is against the http spec and confuses some proxies. wbell parents: 806 diff changeset	449 'Content-Length': str(len(body))})
77c3de8f5c46 Stop sending the 'Location' header on keepalive responses, which is against the http spec and confuses some proxies. wbell parents: 806 diff changeset	450
761 b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	451 def _start_new_step(self, build, stepname):
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	452 """Creates the in-memory representation for a newly started
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	453 step, ready to be persisted to the database.
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	454 """
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	455 step = BuildStep(self.env, build=build.id, name=stepname)
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	456 step.status = BuildStep.IN_PROGRESS
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	457 step.started = int(time.time())
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	458 step.stopped = 0
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	459
b2272caf5ac4 Master now creates next step as in progress when the previous step is complete. While a build is running, the currently running build step now shows up in the ui as in-progress with an updating duration, which makes it much easier to figure out what's going on or if a build is hanging. wbell parents: 758 diff changeset	460 return step

Mercurial > bitten > bitten-test

annotate bitten/master.py @ 872:f320205ca1f9