bitten/bitten-test: bitten/slave.py annotate

annotate bitten/slave.py @ 871:0a59d3e71870 0.6.x

0.6dev: Merged [948] from trunk.

author	hodgestar
date	Mon, 18 Oct 2010 14:04:27 +0000
parents	8df9bc903edb
children	9ac91d38198c

rev	line source
379 0df178e07fdb Use UTF-8 as encoding of source files. cmlenz parents: 377 diff changeset	1 # -- coding: utf-8 --
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	2 #
833 f4d07544722b 0.6dev: Merged [910] from trunk. osimons parents: 797 diff changeset	3 # Copyright (C) 2007-2010 Edgewall Software
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 405 diff changeset	4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de>
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 159 diff changeset	5 # All rights reserved.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	6 #
163 634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 159 diff changeset	7 # This software is licensed as described in the file COPYING, which
634be6cbb808 Flip the switch: Bitten is now BSD-licensed. cmlenz parents: 159 diff changeset	8 # you should have received as part of this distribution. The terms
408 933105ab516b Update file headers and other stuff pointing to the old home. cmlenz parents: 405 diff changeset	9 # are also available at http://bitten.edgewall.org/wiki/License.
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	10
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	11 """Implementation of the build slave."""
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	12
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	13 from datetime import datetime
457 b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	14 import errno
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	15 import urllib
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	16 import urllib2
15 06207499c58c * Use logging in the BEEP core as well as in the master and slave scripts. Closes #4. cmlenz parents: 14 diff changeset	17 import logging
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	18 import os
66 df40a4eb7553 Use the standard {{{platform}}} module for reporting of system info by slaves. Closes #6. cmlenz parents: 63 diff changeset	19 import platform
90 2c4e104afef8 The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17. cmlenz parents: 88 diff changeset	20 import shutil
457 b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	21 import socket
654 cc1fefd5d814 0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz. osimons parents: 652 diff changeset	22 import sys
42 efa525876b1e Basic infrastructure for transmission of snapshot archives to build slaves. See #8. cmlenz parents: 34 diff changeset	23 import tempfile
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	24 import time
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	25 import re
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	26 import cookielib
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	27 import threading
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	28 import os
836 8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	29 import mimetools
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	30 from ConfigParser import MissingSectionHeaderError
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	31
654 cc1fefd5d814 0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz. osimons parents: 652 diff changeset	32 from bitten import PROTOCOL_VERSION
61 47ab019508dd Moved {{{BuildError}}} class into package {{{bitten.build}}}. cmlenz parents: 60 diff changeset	33 from bitten.build import BuildError
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	34 from bitten.build.config import Configuration, ConfigFileNotFound
436 cfbc9ee622d5 Finish the move of build configuration management into the admin interface. cmlenz parents: 424 diff changeset	35 from bitten.recipe import Recipe
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	36 from bitten.util import xmlio
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	37
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	38 EX_OK = getattr(os, "EX_OK", 0)
7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	39 EX_UNAVAILABLE = getattr(os, "EX_UNAVAILABLE", 69)
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	40 EX_IOERR = getattr(os, "EX_IOERR", 74)
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	41 EX_PROTOCOL = getattr(os, "EX_PROTOCOL", 76)
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	42 EX_NOPERM = getattr(os, "EX_NOPERM", 77)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	43
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	44 FORM_TOKEN_RE = re.compile('__FORM_TOKEN\" value=\"(.+)\"')
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	45
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	46 __all__ = ['BuildSlave', 'ExitSlave']
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	47 __docformat__ = 'restructuredtext en'
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	48
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	49 log = logging.getLogger('bitten.slave')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	50
457 b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	51 # List of network errors which are usually temporary and non critical.
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	52 temp_net_errors = [errno.ENETUNREACH, errno.ENETDOWN, errno.ETIMEDOUT,
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	53 errno.ECONNREFUSED]
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	54
474 8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	55 def _rmtree(root):
549 cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there) dfraser parents: 545 diff changeset	56 """Catch shutil.rmtree failures on Windows when files are read-only, and only remove if root exists."""
474 8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	57 def _handle_error(fn, path, excinfo):
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	58 os.chmod(path, 0666)
8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	59 fn(path)
549 cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there) dfraser parents: 545 diff changeset	60 if os.path.exists(root):
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there) dfraser parents: 545 diff changeset	61 return shutil.rmtree(root, onerror=_handle_error)
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there) dfraser parents: 545 diff changeset	62 else:
cb137f2121b9 Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there) dfraser parents: 545 diff changeset	63 return False
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	64
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	65
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	66 class SaneHTTPRequest(urllib2.Request):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	67
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	68 def __init__(self, method, url, data=None, headers={}):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	69 urllib2.Request.__init__(self, url, data, headers)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	70 self.method = method
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	71
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	72 def get_method(self):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	73 if self.method is None:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	74 self.method = self.has_data() and 'POST' or 'GET'
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	75 return self.method
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	76
836 8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	77
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	78 def encode_multipart_formdata(fields):
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	79 """
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	80 Given a dictionary field parameters, returns the HTTP request body and the
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	81 content_type (which includes the boundary string), to be used with an
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	82 httplib-like call.
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	83
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	84 Normal key/value items are treated as regular parameters, but key/tuple
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	85 items are treated as files, where a value tuple is a (filename, data) tuple.
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	86
871 0a59d3e71870 0.6dev: Merged [948] from trunk. hodgestar parents: 836 diff changeset	87 For example::
0a59d3e71870 0.6dev: Merged [948] from trunk. hodgestar parents: 836 diff changeset	88
836 8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	89 fields = {
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	90 'foo': 'bar',
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	91 'foofile': ('foofile.txt', 'contents of foofile'),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	92 }
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	93 body, content_type = encode_multipart_formdata(fields)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	94
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	95 Note: Adapted from http://code.google.com/p/urllib3/ (MIT license)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	96 """
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	97
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	98 BOUNDARY = mimetools.choose_boundary()
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	99 ENCODE_TEMPLATE= "--%(boundary)s\r\n" \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	100 "Content-Disposition: form-data; name=\"%(name)s\"\r\n" \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	101 "\r\n%(value)s\r\n"
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	102 ENCODE_TEMPLATE_FILE = "--%(boundary)s\r\n" \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	103 "Content-Disposition: form-data; name=\"%(name)s\"; " \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	104 "filename=\"%(filename)s\"\r\n" \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	105 "Content-Type: %(contenttype)s\r\n" \
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	106 "\r\n%(value)s\r\n"
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	107
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	108 body = ""
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	109 for key, value in fields.iteritems():
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	110 if isinstance(value, tuple):
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	111 filename, value = value
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	112 body += ENCODE_TEMPLATE_FILE % {
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	113 'boundary': BOUNDARY,
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	114 'name': str(key),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	115 'value': str(value),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	116 'filename': str(filename),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	117 'contenttype': 'application/octet-stream'
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	118 }
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	119 else:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	120 body += ENCODE_TEMPLATE % {
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	121 'boundary': BOUNDARY,
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	122 'name': str(key),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	123 'value': str(value)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	124 }
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	125 body += '--%s--\r\n' % BOUNDARY
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	126 content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	127 return body, content_type
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	128
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	129
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	130 class KeepAliveThread(threading.Thread):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	131 "A thread to periodically send keep-alive messages to the master"
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	132
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	133 def __init__(self, opener, build_url, single_build, keepalive_interval):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	134 threading.Thread.__init__(self, None, None, "KeepaliveThread")
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	135 self.build_url = build_url
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	136 self.keepalive_interval = keepalive_interval
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	137 self.single_build = single_build
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	138 self.last_keepalive = int(time.time())
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	139 self.kill = False
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	140 self.opener = opener
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	141
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	142 def keepalive(self):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	143 log.debug('Sending keepalive')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	144 method = 'POST'
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	145 url = self.build_url + '/keepalive/'
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	146 body = None
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	147 shutdown = False
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	148 headers = {
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	149 'Content-Type': 'application/x-bitten+xml'
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	150 }
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	151
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	152 log.debug('Sending %s request to %r', method, url)
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	153 req = SaneHTTPRequest(method, url, body, headers or {})
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	154 try:
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	155 return self.opener.open(req)
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	156 except urllib2.HTTPError, e:
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	157 # a conflict error lets us know that we've been
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	158 # invalidated. Ideally, we'd engineer something to stop any
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	159 # running steps in progress, but killing threads is tricky
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	160 # stuff. For now, we'll wait for whatever's going
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	161 # on to stop, and the main thread'll figure out that we've
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	162 # been invalidated.
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	163 log.warning('Server returned keepalive error %d: %s', e.code, e.msg)
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	164 except:
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	165 log.warning('Server returned unknown keepalive error')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	166
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	167 def run(self):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	168 log.debug('Keepalive thread starting.')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	169 while (not self.kill):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	170 now = int(time.time())
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	171 if (self.last_keepalive + self.keepalive_interval) < now:
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	172 self.keepalive()
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	173 self.last_keepalive = now
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	174
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	175 time.sleep(1)
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	176 log.debug('Keepalive thread exiting.')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	177
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	178 def stop(self):
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	179 log.debug('Stopping keepalive thread')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	180 self.kill = True
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	181 self.join(30)
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	182 log.debug('Keepalive thread stopped')
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	183
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	184
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	185 class BuildSlave(object):
542 cacf018eb190 Small documentation fixes. wbell parents: 525 diff changeset	186 """HTTP client implementation for the build slave."""
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	187
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	188 def __init__(self, urls, name=None, config=None, dry_run=False,
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	189 work_dir=None, build_dir="build_${build}",
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	190 keep_files=False, single_build=False,
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	191 poll_interval=300, keepalive_interval = 60,
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	192 username=None, password=None,
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	193 dump_reports=False, no_loop=False, form_auth=False):
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	194 """Create the build slave instance.
90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	195
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	196 :param urls: a list of URLs of the build masters to connect to, or a
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	197 single-element list containing the path to a build recipe
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	198 file
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	199 :param name: the name with which this slave should identify itself
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	200 :param config: the path to the slave configuration file
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	201 :param dry_run: wether the build outcome should not be reported back
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	202 to the master
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	203 :param work_dir: the working directory to use for build execution
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	204 :param build_dir: the pattern to use for naming the build subdir
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	205 :param keep_files: whether files and directories created for build
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	206 execution should be kept when done
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	207 :param single_build: whether this slave should exit after completing a
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	208 single build, or continue processing builds forever
449 01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	209 :param poll_interval: the time in seconds to wait between requesting
451 57822fefd441 Fix default for poll interval of slave. cmlenz parents: 449 diff changeset	210 builds from the build master (default is five
57822fefd441 Fix default for poll interval of slave. cmlenz parents: 449 diff changeset	211 minutes)
871 0a59d3e71870 0.6dev: Merged [948] from trunk. hodgestar parents: 836 diff changeset	212 :param keepalive_interval: the time in seconds to wait between sending
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	213 keepalive heartbeats (default is 30 seconds)
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	214 :param username: the username to use when authentication against the
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	215 build master is requested
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	216 :param password: the password to use when authentication is needed
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	217 :param dump_reports: whether report data should be written to the
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	218 standard output, in addition to being transmitted
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	219 to the build master
525 bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	220 :param no_loop: for this slave to just perform a single check, regardless
bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	221 of whether a build is done or not
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	222 :param form_auth: login using AccountManager HTML form instead of
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	223 HTTP authentication for all urls
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	224 """
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	225 self.urls = urls
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	226 self.local = len(urls) == 1 and not urls[0].startswith('http://') \
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	227 and not urls[0].startswith('https://')
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	228 if name is None:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	229 name = platform.node().split('.', 1)[0].lower()
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	230 self.name = name
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	231 self.config = Configuration(config)
185 2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds. cmlenz parents: 163 diff changeset	232 self.dry_run = dry_run
208 f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	233 if not work_dir:
f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	234 work_dir = tempfile.mkdtemp(prefix='bitten')
f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	235 elif not os.path.exists(work_dir):
f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	236 os.makedirs(work_dir)
f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	237 self.work_dir = work_dir
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	238 self.build_dir = build_dir
241 d3c8a74922cd * Add an option to `bitten-slave` that tells it to keep any files in the working directory. cmlenz parents: 233 diff changeset	239 self.keep_files = keep_files
377 a465b5dbfecf Tabs and line length. cmlenz parents: 373 diff changeset	240 self.single_build = single_build
525 bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	241 self.no_loop = no_loop
449 01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	242 self.poll_interval = poll_interval
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	243 self.keepalive_interval = keepalive_interval
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	244 self.dump_reports = dump_reports
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	245 self.cookiejar = cookielib.CookieJar()
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	246 self.username = username \
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	247 or self.config['authentication.username'] or ''
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	248
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	249 if not self.local:
569 74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	250 self.password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	251 if self.username:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	252 log.debug('Enabling authentication with username %r',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	253 self.username)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	254 self.form_auth = form_auth
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	255 password = password \
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	256 or self.config['authentication.password'] or ''
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	257 self.config.packages.pop('authentication', None)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	258 urls = [url[:-7] for url in urls]
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	259 self.password_mgr.add_password(
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	260 None, urls, self.username, password)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	261 self.auth_map = dict(map(lambda x: (x, False), urls))
569 74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	262
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	263 def _get_opener(self):
652 de04ce69da53 0.6dev: Removing code and updated docs related to Trac < 0.11 and Python < 2.4 (base requirements for Bitten 0.6). osimons parents: 649 diff changeset	264 opener = urllib2.build_opener(urllib2.HTTPErrorProcessor())
569 74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	265 opener.add_handler(urllib2.HTTPBasicAuthHandler(self.password_mgr))
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	266 opener.add_handler(urllib2.HTTPDigestAuthHandler(self.password_mgr))
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	267 opener.add_handler(urllib2.HTTPCookieProcessor(self.cookiejar))
569 74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	268 return opener
74bc339ce7d5 0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request. osimons parents: 560 diff changeset	269 opener = property(_get_opener)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	270
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	271 def request(self, method, url, body=None, headers=None):
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	272 log.debug('Sending %s request to %r', method, url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	273 req = SaneHTTPRequest(method, url, body, headers or {})
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	274 try:
559 bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163 dfraser parents: 549 diff changeset	275 resp = self.opener.open(req)
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163 dfraser parents: 549 diff changeset	276 if not hasattr(resp, 'code'):
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163 dfraser parents: 549 diff changeset	277 resp.code = 200
bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163 dfraser parents: 549 diff changeset	278 return resp
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	279 except urllib2.HTTPError, e:
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	280 if e.code >= 300:
647 6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't). osimons parents: 645 diff changeset	281 if hasattr(e, 'headers') and \
6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't). osimons parents: 645 diff changeset	282 e.headers.getheader('Content-Type', ''
6fe6fc8ee36c 0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't). osimons parents: 645 diff changeset	283 ).startswith('text/plain'):
645 8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 628 diff changeset	284 content = e.read()
8c824b14e1c5 0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging). osimons parents: 628 diff changeset	285 else:
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	286 content = 'no message available'
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	287 log.debug('Server returned error %d: %s (%s)',
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	288 e.code, e.msg, content)
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	289 raise
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	290 return e
29 2adc3480e4aa Some cleanup and additional docstrings. cmlenz parents: 19 diff changeset	291
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	292 def run(self):
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	293 if self.local:
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	294 fileobj = open(self.urls[0])
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	295 try:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	296 self._execute_build(None, fileobj)
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	297 finally:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	298 fileobj.close()
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	299 return EX_OK
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	300
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	301 urls = []
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	302 while True:
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	303 if not urls:
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	304 urls[:] = self.urls
9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	305 url = urls.pop(0)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	306 try:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	307 try:
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	308 if self.username and not self.auth_map.get(url):
716 cc620fb0ee60 0.6dev: Merging [792:793] from trunk. osimons parents: 675 diff changeset	309 login_url = '%s/login?referer=%s' % (url[:-7],
cc620fb0ee60 0.6dev: Merging [792:793] from trunk. osimons parents: 675 diff changeset	310 urllib.quote_plus(url))
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	311 # First request to url, authentication needed
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	312 if self.form_auth:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	313 log.debug('Performing http form authentication')
716 cc620fb0ee60 0.6dev: Merging [792:793] from trunk. osimons parents: 675 diff changeset	314 resp = self.request('POST', login_url)
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	315 match = FORM_TOKEN_RE.search(resp.read())
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	316 if not match:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	317 log.error("Project %s does not support form "
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	318 "authentication" % url[:-7])
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	319 raise ExitSlave(EX_NOPERM)
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	320 values = {'user': self.username,
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	321 'password':
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	322 self.password_mgr.find_user_password(
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	323 None, url)[1],
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	324 'referer': '',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	325 '__FORM_TOKEN': match.group(1)}
716 cc620fb0ee60 0.6dev: Merging [792:793] from trunk. osimons parents: 675 diff changeset	326 self.request('POST', login_url,
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	327 body=urllib.urlencode(values))
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	328 else:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	329 log.debug('Performing basic/digest authentication')
716 cc620fb0ee60 0.6dev: Merging [792:793] from trunk. osimons parents: 675 diff changeset	330 self.request('HEAD', login_url)
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	331 self.auth_map[url] = True
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	332 elif self.username:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	333 log.debug('Reusing authentication information.')
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	334 else:
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	335 log.debug('Authentication not provided. Attempting to '
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	336 'execute build anonymously.')
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	337 job_done = self._create_build(url)
480 5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181 wbell parents: 474 diff changeset	338 if job_done:
5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181 wbell parents: 474 diff changeset	339 continue
465 04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError mgood parents: 461 diff changeset	340 except urllib2.HTTPError, e:
04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError mgood parents: 461 diff changeset	341 # HTTPError doesn't have the "reason" attribute of URLError
04205ba5dc98 handle HTTPErrors separately to work around them not having a "reason" attribute like URLError mgood parents: 461 diff changeset	342 log.error(e)
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	343 raise ExitSlave(EX_UNAVAILABLE)
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	344 except urllib2.URLError, e:
457 b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	345 # Is this a temporary network glitch or something a bit
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	346 # more severe?
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	347 if isinstance(e.reason, socket.error) and \
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	348 e.reason.args[0] in temp_net_errors:
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	349 log.warning(e)
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	350 else:
b267ad4b93e9 Make the slave differentiate between benign and fatal network errors and jonas parents: 456 diff changeset	351 log.error(e)
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	352 raise ExitSlave(EX_UNAVAILABLE)
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	353 except ExitSlave, e:
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	354 return e.exit_code
525 bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	355 if self.no_loop:
bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	356 break
449 01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	357 time.sleep(self.poll_interval)
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	358
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	359 def quit(self):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	360 log.info('Shutting down')
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	361 raise ExitSlave(EX_OK)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	362
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	363 def _create_build(self, url):
649 eed0149c302a 0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421. osimons parents: 648 diff changeset	364 xml = xmlio.Element('slave', name=self.name, version=PROTOCOL_VERSION)[
233 8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	365 xmlio.Element('platform', processor=self.config['processor'])[
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	366 self.config['machine']
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	367 ],
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	368 xmlio.Element('os', family=self.config['family'],
244 1aa624af9ebb * Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49. cmlenz parents: 242 diff changeset	369 version=self.config['version'])[
233 8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	370 self.config['os']
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	371 ],
29 2adc3480e4aa Some cleanup and additional docstrings. cmlenz parents: 19 diff changeset	372 ]
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	373
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	374 log.debug('Configured packages: %s', self.config.packages)
233 8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	375 for package, properties in self.config.packages.items():
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	376 xml.append(xmlio.Element('package', name=package, **properties))
8f816147620f * Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]). cmlenz parents: 221 diff changeset	377
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	378 body = str(xml)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	379 log.debug('Sending slave configuration: %s', body)
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	380 resp = self.request('POST', url, body, {
559 bf3e0c2f6b80 Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163 dfraser parents: 549 diff changeset	381 'Content-Length': str(len(body)),
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	382 'Content-Type': 'application/x-bitten+xml'
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	383 })
277 1141027071b3 Changes to snapshot archive creation/transmission: cmlenz parents: 244 diff changeset	384
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	385 if resp.code == 201:
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	386 self._initiate_build(resp.info().get('location'))
480 5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181 wbell parents: 474 diff changeset	387 return True
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	388 elif resp.code == 204:
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	389 log.info('No pending builds')
480 5181353ccb41 Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181 wbell parents: 474 diff changeset	390 return False
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	391 else:
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	392 log.error('Unexpected response (%d %s)', resp.code, resp.msg)
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	393 raise ExitSlave(EX_PROTOCOL)
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	394
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	395 def _initiate_build(self, build_url):
418 0cfc877405d1 Fix build slave error introduced in [461]. cmlenz parents: 417 diff changeset	396 log.info('Build pending at %s', build_url)
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	397 try:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	398 resp = self.request('GET', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	399 if resp.code == 200:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	400 self._execute_build(build_url, resp)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	401 else:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	402 log.error('Unexpected response (%d): %s', resp.code, resp.msg)
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	403 self._cancel_build(build_url, exit_code=EX_PROTOCOL)
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	404 except KeyboardInterrupt:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	405 log.warning('Build interrupted')
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	406 self._cancel_build(build_url)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	407
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	408 def _execute_build(self, build_url, fileobj):
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	409 build_id = build_url and int(build_url.split('/')[-1]) or 0
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	410 xml = xmlio.parse(fileobj)
580 c66190747a9c 0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail. osimons parents: 576 diff changeset	411 basedir = ''
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	412 try:
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	413 if not self.local:
797 7eaece84fe85 Merged [874] from trunk. osimons parents: 786 diff changeset	414 keepalive_thread = KeepAliveThread(self.opener, build_url,
7eaece84fe85 Merged [874] from trunk. osimons parents: 786 diff changeset	415 self.single_build, self.keepalive_interval)
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	416 keepalive_thread.start()
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	417 recipe = Recipe(xml, os.path.join(self.work_dir, self.build_dir),
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	418 self.config)
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	419 basedir = recipe.ctxt.basedir
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	420 log.debug('Running build in directory %s' % basedir)
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	421 if not os.path.exists(basedir):
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	422 os.mkdir(basedir)
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	423
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	424 for step in recipe:
755 ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	425 try:
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	426 log.info('Executing build step %r, onerror = %s', step.id, step.onerror)
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	427 if not self._execute_step(build_url, recipe, step):
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	428 log.warning('Stopping build due to failure')
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	429 break
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	430 except Exception, e:
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	431 log.error('Exception raised processing step %s. Reraising %s', step.id, e)
ab91df14f670 Merge of [832] from trunk. wbell parents: 722 diff changeset	432 raise
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	433 else:
417 ab2557ff0d97 Change the default log verbosity of the build slave, and remove the `--debug` option. cmlenz parents: 414 diff changeset	434 log.info('Build completed')
461 f487963dd886 When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187. cmlenz parents: 457 diff changeset	435 if self.dry_run:
f487963dd886 When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187. cmlenz parents: 457 diff changeset	436 self._cancel_build(build_url)
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	437 finally:
797 7eaece84fe85 Merged [874] from trunk. osimons parents: 786 diff changeset	438 if not self.local:
7eaece84fe85 Merged [874] from trunk. osimons parents: 786 diff changeset	439 keepalive_thread.stop()
580 c66190747a9c 0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail. osimons parents: 576 diff changeset	440 if not self.keep_files and os.path.isdir(basedir):
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	441 log.debug('Removing build directory %s' % basedir)
474 8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	442 _rmtree(basedir)
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	443 if self.single_build:
aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	444 log.info('Exiting after single build completed.')
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	445 raise ExitSlave(EX_OK)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	446
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	447 def _execute_step(self, build_url, recipe, step):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	448 failed = False
759 bfd5fd75d1d6 Port of [836] to 0.6.x wbell parents: 755 diff changeset	449 started = int(time.time())
bfd5fd75d1d6 Port of [836] to 0.6.x wbell parents: 755 diff changeset	450 xml = xmlio.Element('result', step=step.id)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	451 try:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	452 for type, category, generator, output in \
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	453 step.execute(recipe.ctxt):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	454 if type == Recipe.ERROR:
82 01200c88ddb0 Include timing information in the build messages transmitted from slave to master for better accuracy. cmlenz parents: 80 diff changeset	455 failed = True
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	456 if type == Recipe.REPORT and self.dump_reports:
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	457 print output
836 8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	458 if type == Recipe.ATTACH:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	459 # Attachments are added out-of-band due to major
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	460 # performance issues with inlined base64 xml content
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	461 self._attach_file(build_url, recipe, output)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	462 xml.append(xmlio.Element(type, category=category,
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	463 generator=generator)[
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	464 output
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	465 ])
414 aa34d82b2c9a The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script. cmlenz parents: 411 diff changeset	466 except KeyboardInterrupt:
417 ab2557ff0d97 Change the default log verbosity of the build slave, and remove the `--debug` option. cmlenz parents: 414 diff changeset	467 log.warning('Build interrupted')
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	468 self._cancel_build(build_url)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	469 except BuildError, e:
628 05686657e989 0.6dev: Implement a `subprocess` implementation of `execute()` that is now the default for all slaves where this is available - essentially all slaves with Python 2.4 and higher, or where module is installed separately. This fixes: osimons parents: 610 diff changeset	470 log.error('Build step %r failed', step.id)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	471 failed = True
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	472 except Exception, e:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	473 log.error('Internal error in build step %r', step.id, exc_info=True)
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	474 failed = True
759 bfd5fd75d1d6 Port of [836] to 0.6.x wbell parents: 755 diff changeset	475 xml.attr['duration'] = (time.time() - started)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	476 if failed:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	477 xml.attr['status'] = 'failure'
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	478 else:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	479 xml.attr['status'] = 'success'
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	480 log.info('Build step %s completed successfully', step.id)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	481
424 bc81d7b069ff Rudimentary implementation of ``dry-run`` option in build slave. Builds are still put in in progress mode, but no results are transmitted back, and the build is automatically invalidated after the slave is done. cmlenz parents: 420 diff changeset	482 if not self.local and not self.dry_run:
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	483 try:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	484 resp = self.request('POST', build_url + '/steps/', str(xml), {
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	485 'Content-Type': 'application/x-bitten+xml'
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	486 })
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	487 if resp.code != 201:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	488 log.error('Unexpected response (%d): %s', resp.code,
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	489 resp.msg)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	490 except KeyboardInterrupt:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	491 log.warning('Build interrupted')
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	492 self._cancel_build(build_url)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	493 return not failed or step.onerror != 'fail'
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	494
495 7e9a60417cb3 `os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully dfraser parents: 494 diff changeset	495 def _cancel_build(self, build_url, exit_code=EX_OK):
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	496 log.info('Cancelling build at %s', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	497 if not self.local:
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	498 resp = self.request('DELETE', build_url)
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	499 if resp.code not in (200, 204):
23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	500 log.error('Unexpected response (%d): %s', resp.code, resp.msg)
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	501 raise ExitSlave(exit_code)
420 23de253435b8 Slaves now attempt to explicitly cancel builds when they are interrupted. cmlenz parents: 418 diff changeset	502
836 8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	503 def _attach_file(self, build_url, recipe, attachment):
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	504 form_token = recipe._root.attr.get('form_token', '')
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	505 if self.local or self.dry_run or not form_token:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	506 log.info('Attachment %s not sent due to current slave options',
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	507 attachment.attr['file'])
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	508 return
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	509 resource_type = attachment.attr['resource']
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	510 url = str(build_url + '/attach/' + resource_type)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	511 path = recipe.ctxt.resolve(attachment.attr['filename'])
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	512 filename = os.path.basename(path)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	513 log.debug('Attaching file %s to %s...', attachment.attr['filename'],
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	514 resource_type)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	515 f = open(path)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	516 try:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	517 data, content_type = encode_multipart_formdata({
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	518 'file': (filename, f.read()),
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	519 'description': attachment.attr['description'],
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	520 '__FORM_TOKEN': form_token})
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	521 finally:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	522 f.close()
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	523 resp = self.request('POST', url , data, {
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	524 'Content-Type': content_type})
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	525 if not resp.code == 201:
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	526 msg = 'Error attaching %s to %s'
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	527 log.error(msg, filename, resource_type)
8df9bc903edb 0.6dev: Merged [912:913] from trunk. osimons parents: 833 diff changeset	528 raise BuildError(msg, filename, resource_type)
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	529
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	530 class ExitSlave(Exception):
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	531 """Exception used internally by the slave to signal that the slave process
411 a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	532 should be stopped.
a169d2e96463 Use reStructuredText as the API documentation syntax. cmlenz parents: 408 diff changeset	533 """
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	534 def __init__(self, exit_code):
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	535 self.exit_code = exit_code
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	536 Exception.__init__(self)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	537
21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	538
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	539 def main():
313 90422699a594 More and improved docstrings (using epydoc format). cmlenz parents: 310 diff changeset	540 """Main entry point for running the build slave."""
56 033366d81def Build slave now executes the build. Closes #10. cmlenz parents: 51 diff changeset	541 from bitten import __version__ as VERSION
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	542 from optparse import OptionParser
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	543
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	544 parser = OptionParser(usage='usage: %prog [options] url1 [url2] ...',
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	545 version='%%prog %s' % VERSION)
185 2c24d9a950ed Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds. cmlenz parents: 163 diff changeset	546 parser.add_option('--name', action='store', dest='name',
63 2332aedba328 * Allow specifying a different name for a build slave (default is the host name). cmlenz parents: 61 diff changeset	547 help='name of this slave (defaults to host name)')
208 f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	548 parser.add_option('-f', '--config', action='store', dest='config',
127 a9443f673344 Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29. cmlenz parents: 119 diff changeset	549 metavar='FILE', help='path to configuration file')
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	550 parser.add_option('-u', '--user', dest='username',
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	551 help='the username to use for authentication')
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	552 parser.add_option('-p', '--password', dest='password',
08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	553 help='the password to use when authenticating')
587 0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402. osimons parents: 580 diff changeset	554 def _ask_password(option, opt_str, value, parser):
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402. osimons parents: 580 diff changeset	555 from getpass import getpass
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402. osimons parents: 580 diff changeset	556 parser.values.password = getpass('Passsword: ')
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402. osimons parents: 580 diff changeset	557 parser.add_option('-P', '--ask-password', action='callback',
0ac21e3343a4 0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402. osimons parents: 580 diff changeset	558 callback=_ask_password, help='Prompt for password')
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	559 parser.add_option('--form-auth', action='store_true',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	560 dest='form_auth',
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	561 help='login using AccountManager HTML form instead of '
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	562 'HTTP authentication for all urls')
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	563
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	564 group = parser.add_option_group('building')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	565 group.add_option('-d', '--work-dir', action='store', dest='work_dir',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	566 metavar='DIR', help='working directory for builds')
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	567 group.add_option('--build-dir', action='store', dest='build_dir',
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	568 default = 'build_${config}_${build}',
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	569 help='name pattern for the build dir to use inside the '
79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	570 'working dir ["%default"]')
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	571 group.add_option('-k', '--keep-files', action='store_true',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	572 dest='keep_files',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	573 help='don\'t delete files after builds')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	574 group.add_option('-s', '--single', action='store_true',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	575 dest='single_build',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	576 help='exit after completing a single build')
525 bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	577 group.add_option('', '--no-loop', action='store_true',
bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	578 dest='no_loop',
bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	579 help='exit after completing a single check and running '
bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	580 'the required builds')
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	581 group.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	582 help='don\'t report results back to master')
449 01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	583 group.add_option('-i', '--interval', dest='interval', metavar='SECONDS',
01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	584 type='int', help='time to wait between requesting builds')
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	585 group.add_option('-b', '--keepalive_interval', dest='keepalive_interval', metavar='SECONDS', type='int', help='time to wait between keepalive heartbeats')
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	586 group = parser.add_option_group('logging')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	587 group.add_option('-l', '--log', dest='logfile', metavar='FILENAME',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	588 help='write log messages to FILENAME')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	589 group.add_option('-v', '--verbose', action='store_const', dest='loglevel',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	590 const=logging.DEBUG, help='print as much as possible')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	591 group.add_option('-q', '--quiet', action='store_const', dest='loglevel',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	592 const=logging.WARN, help='print as little as possible')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	593 group.add_option('--dump-reports', action='store_true', dest='dump_reports',
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	594 help='whether report data should be printed')
a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	595
242 372d1de2e3ec * Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`. cmlenz parents: 241 diff changeset	596 parser.set_defaults(dry_run=False, keep_files=False,
525 bb448ad9bf35 Added option for not looping after check. Fixes #371 dfraser parents: 515 diff changeset	597 loglevel=logging.INFO, single_build=False, no_loop=False,
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	598 dump_reports=False, interval=300, keepalive_interval=60,
38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	599 form_auth=False)
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	600 options, args = parser.parse_args()
9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	601
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	602 if len(args) < 1:
19 9db5f8eddb0d Proper {{{optparse}}}-based command-line interface for master and slave. cmlenz parents: 18 diff changeset	603 parser.error('incorrect number of arguments')
494 9c9bf7a69bf0 Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271. cmlenz parents: 490 diff changeset	604 urls = args
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	605
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 149 diff changeset	606 logger = logging.getLogger('bitten')
2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 149 diff changeset	607 logger.setLevel(options.loglevel)
93 b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	608 handler = logging.StreamHandler()
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	609 handler.setLevel(options.loglevel)
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	610 formatter = logging.Formatter('[%(levelname)-8s] %(message)s')
b289e572bc7e Improved logging; the build master can now optionally log to a file. Closes #13. cmlenz parents: 90 diff changeset	611 handler.setFormatter(formatter)
157 2efdc69e63c3 Some style/documentation improvements to make Pylint happier. cmlenz parents: 149 diff changeset	612 logger.addHandler(handler)
339 6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	613 if options.logfile:
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	614 handler = logging.FileHandler(options.logfile)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	615 handler.setLevel(options.loglevel)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	616 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: '
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	617 '%(message)s')
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	618 handler.setFormatter(formatter)
6469c692c603 Add a logfile option to the slave. Thanks to Walter Bell for the patch. cmlenz parents: 318 diff changeset	619 logger.addHandler(handler)
13 21aa17f97522 Initial code for build master and slave... these don't do a lot yet. cmlenz parents: diff changeset	620
610 610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	621 log.info("Slave launched at %s" % \
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	622 datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	623
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	624 slave = None
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	625 try:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	626 slave = BuildSlave(urls, name=options.name, config=options.config,
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	627 dry_run=options.dry_run, work_dir=options.work_dir,
466 79be3c00ae69 Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch. cmlenz parents: 465 diff changeset	628 build_dir=options.build_dir,
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	629 keep_files=options.keep_files,
402 08801667f00f Switch to urllib2 in slave so that we can support basic and digest authentication. cmlenz parents: 401 diff changeset	630 single_build=options.single_build,
543 ce16a5dcf8b3 Pass `--no-loop` option to the `BuildSlave` (fixes #375, from the patch there - thanks!) dfraser parents: 542 diff changeset	631 no_loop=options.no_loop,
449 01503ed994e7 Make the interval the slave sleeps between requesting builds configurable from the commandline. cmlenz parents: 442 diff changeset	632 poll_interval=options.interval,
786 38e11e15d95e Port of [864] to 0.6.x wbell parents: 759 diff changeset	633 keepalive_interval=options.keepalive_interval,
442 a8787de4fbc3 Improve the still experimental support for using `coverage.py`. cmlenz parents: 436 diff changeset	634 username=options.username, password=options.password,
648 a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	635 dump_reports=options.dump_reports,
a04e46a0bce3 0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219). osimons parents: 647 diff changeset	636 form_auth=options.form_auth)
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	637 try:
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	638 exit_code = slave.run()
392 026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	639 except KeyboardInterrupt:
026d9aa41b85 Merged HTTP branch into trunk. cmlenz parents: 387 diff changeset	640 slave.quit()
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	641 except ConfigFileNotFound, e:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	642 log.error(e)
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	643 exit_code = EX_IOERR
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	644 except MissingSectionHeaderError:
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	645 log.error("Error parsing configuration file %r. Wrong format?" \
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	646 % options.config)
251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	647 exit_code = EX_IOERR
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	648 except ExitSlave, e:
5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	649 exit_code = e.exit_code
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	650
675 251be647314c 0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format). osimons parents: 654 diff changeset	651 if slave and not (options.work_dir or options.keep_files):
576 4b75e3f6f27c 0.6dev: Fixing #229 again - keeping `slave.work_dir` if either `--keep_files` or `--work_dir=` are part of options. osimons parents: 570 diff changeset	652 log.debug('Removing working directory %s' % slave.work_dir)
474 8c3dfe0efe78 Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236. wbell parents: 466 diff changeset	653 _rmtree(slave.work_dir)
610 610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	654
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	655 log.info("Slave exited at %s" % \
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	656 datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
610a97f7e1ba 0.6dev: Logging start + stop time for the slave. Closes #268. osimons parents: 587 diff changeset	657
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	658 return exit_code
208 f39136272683 * Slave now removes build working directories when done. cmlenz parents: 203 diff changeset	659
31 0b82d012e42c Add thin script shells around master and slave. cmlenz parents: 29 diff changeset	660 if __name__ == '__main__':
490 5c6c56fbd9d7 make `bitten-slave` return a meaningful exit code mgood parents: 480 diff changeset	661 sys.exit(main())

Mercurial > bitten > bitten-test

annotate bitten/slave.py @ 871:0a59d3e71870 0.6.x