Mercurial > bitten > bitten-test
annotate bitten/slave.py @ 908:56c269e75737 0.6.x
Ported r989 back to 0.6.x branch.
author | cmlenz |
---|---|
date | Wed, 13 Apr 2011 13:25:27 +0000 |
parents | 0ab6b21b8010 |
children | 5bfdd7348d6f |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
2 # |
833 | 3 # Copyright (C) 2007-2010 Edgewall Software |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
163 | 5 # All rights reserved. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
6 # |
163 | 7 # This software is licensed as described in the file COPYING, which |
8 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
10 |
313 | 11 """Implementation of the build slave.""" |
12 | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
13 from datetime import datetime |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
14 import errno |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
15 import urllib |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
16 import urllib2 |
15
06207499c58c
* Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents:
14
diff
changeset
|
17 import logging |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
18 import os |
66
df40a4eb7553
Use the standard {{{platform}}} module for reporting of system info by slaves. Closes #6.
cmlenz
parents:
63
diff
changeset
|
19 import platform |
90
2c4e104afef8
The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents:
88
diff
changeset
|
20 import shutil |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
21 import socket |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
22 import sys |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
23 import tempfile |
392 | 24 import time |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
25 import re |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
26 import cookielib |
786 | 27 import threading |
836 | 28 import mimetools |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
29 from ConfigParser import MissingSectionHeaderError |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
30 |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
31 from bitten import PROTOCOL_VERSION |
61
47ab019508dd
Moved {{{BuildError}}} class into package {{{bitten.build}}}.
cmlenz
parents:
60
diff
changeset
|
32 from bitten.build import BuildError |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
33 from bitten.build.config import Configuration, ConfigFileNotFound |
436
cfbc9ee622d5
Finish the move of build configuration management into the admin interface.
cmlenz
parents:
424
diff
changeset
|
34 from bitten.recipe import Recipe |
392 | 35 from bitten.util import xmlio |
896 | 36 from bitten.util.compat import HTTPBasicAuthHandler |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
37 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
38 EX_OK = getattr(os, "EX_OK", 0) |
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
39 EX_UNAVAILABLE = getattr(os, "EX_UNAVAILABLE", 69) |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
40 EX_IOERR = getattr(os, "EX_IOERR", 74) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
41 EX_PROTOCOL = getattr(os, "EX_PROTOCOL", 76) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
42 EX_NOPERM = getattr(os, "EX_NOPERM", 77) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
43 |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
44 FORM_TOKEN_RE = re.compile('__FORM_TOKEN\" value=\"(.+)\"') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
45 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
46 __all__ = ['BuildSlave', 'ExitSlave'] |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
47 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
48 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
49 log = logging.getLogger('bitten.slave') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
50 |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
51 # List of network errors which are usually temporary and non critical. |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
52 temp_net_errors = [errno.ENETUNREACH, errno.ENETDOWN, errno.ETIMEDOUT, |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
53 errno.ECONNREFUSED] |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
54 |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
55 def _rmtree(root): |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
56 """Catch shutil.rmtree failures on Windows when files are read-only, and only remove if root exists.""" |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
57 def _handle_error(fn, path, excinfo): |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
58 os.chmod(path, 0666) |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
59 fn(path) |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
60 if os.path.exists(root): |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
61 return shutil.rmtree(root, onerror=_handle_error) |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
62 else: |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
63 return False |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
64 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
65 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
66 class SaneHTTPRequest(urllib2.Request): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
67 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
68 def __init__(self, method, url, data=None, headers={}): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
69 urllib2.Request.__init__(self, url, data, headers) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
70 self.method = method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
71 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
72 def get_method(self): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
73 if self.method is None: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
74 self.method = self.has_data() and 'POST' or 'GET' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
75 return self.method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
76 |
836 | 77 |
78 def encode_multipart_formdata(fields): | |
79 """ | |
80 Given a dictionary field parameters, returns the HTTP request body and the | |
81 content_type (which includes the boundary string), to be used with an | |
82 httplib-like call. | |
83 | |
84 Normal key/value items are treated as regular parameters, but key/tuple | |
85 items are treated as files, where a value tuple is a (filename, data) tuple. | |
86 | |
871 | 87 For example:: |
88 | |
836 | 89 fields = { |
90 'foo': 'bar', | |
91 'foofile': ('foofile.txt', 'contents of foofile'), | |
92 } | |
93 body, content_type = encode_multipart_formdata(fields) | |
94 | |
95 Note: Adapted from http://code.google.com/p/urllib3/ (MIT license) | |
96 """ | |
97 | |
98 BOUNDARY = mimetools.choose_boundary() | |
99 ENCODE_TEMPLATE= "--%(boundary)s\r\n" \ | |
100 "Content-Disposition: form-data; name=\"%(name)s\"\r\n" \ | |
101 "\r\n%(value)s\r\n" | |
102 ENCODE_TEMPLATE_FILE = "--%(boundary)s\r\n" \ | |
103 "Content-Disposition: form-data; name=\"%(name)s\"; " \ | |
104 "filename=\"%(filename)s\"\r\n" \ | |
105 "Content-Type: %(contenttype)s\r\n" \ | |
106 "\r\n%(value)s\r\n" | |
107 | |
108 body = "" | |
109 for key, value in fields.iteritems(): | |
110 if isinstance(value, tuple): | |
111 filename, value = value | |
112 body += ENCODE_TEMPLATE_FILE % { | |
113 'boundary': BOUNDARY, | |
114 'name': str(key), | |
115 'value': str(value), | |
116 'filename': str(filename), | |
117 'contenttype': 'application/octet-stream' | |
118 } | |
119 else: | |
120 body += ENCODE_TEMPLATE % { | |
121 'boundary': BOUNDARY, | |
122 'name': str(key), | |
123 'value': str(value) | |
124 } | |
125 body += '--%s--\r\n' % BOUNDARY | |
126 content_type = 'multipart/form-data; boundary=%s' % BOUNDARY | |
127 return body, content_type | |
128 | |
129 | |
786 | 130 class KeepAliveThread(threading.Thread): |
131 "A thread to periodically send keep-alive messages to the master" | |
132 | |
133 def __init__(self, opener, build_url, single_build, keepalive_interval): | |
134 threading.Thread.__init__(self, None, None, "KeepaliveThread") | |
135 self.build_url = build_url | |
136 self.keepalive_interval = keepalive_interval | |
137 self.single_build = single_build | |
138 self.last_keepalive = int(time.time()) | |
139 self.kill = False | |
140 self.opener = opener | |
141 | |
142 def keepalive(self): | |
143 log.debug('Sending keepalive') | |
144 method = 'POST' | |
145 url = self.build_url + '/keepalive/' | |
146 body = None | |
147 shutdown = False | |
148 headers = { | |
892 | 149 'Content-Type': 'application/x-bitten+xml', |
150 'Content-Length': '0' | |
786 | 151 } |
152 log.debug('Sending %s request to %r', method, url) | |
153 req = SaneHTTPRequest(method, url, body, headers or {}) | |
154 try: | |
155 return self.opener.open(req) | |
156 except urllib2.HTTPError, e: | |
157 # a conflict error lets us know that we've been | |
158 # invalidated. Ideally, we'd engineer something to stop any | |
159 # running steps in progress, but killing threads is tricky | |
160 # stuff. For now, we'll wait for whatever's going | |
161 # on to stop, and the main thread'll figure out that we've | |
162 # been invalidated. | |
163 log.warning('Server returned keepalive error %d: %s', e.code, e.msg) | |
164 except: | |
165 log.warning('Server returned unknown keepalive error') | |
166 | |
167 def run(self): | |
168 log.debug('Keepalive thread starting.') | |
169 while (not self.kill): | |
170 now = int(time.time()) | |
171 if (self.last_keepalive + self.keepalive_interval) < now: | |
172 self.keepalive() | |
173 self.last_keepalive = now | |
174 | |
175 time.sleep(1) | |
176 log.debug('Keepalive thread exiting.') | |
177 | |
178 def stop(self): | |
179 log.debug('Stopping keepalive thread') | |
180 self.kill = True | |
181 self.join(30) | |
182 log.debug('Keepalive thread stopped') | |
183 | |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
184 |
392 | 185 class BuildSlave(object): |
542 | 186 """HTTP client implementation for the build slave.""" |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
187 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
188 def __init__(self, urls, name=None, config=None, dry_run=False, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
189 work_dir=None, build_dir="build_${build}", |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
190 keep_files=False, single_build=False, |
786 | 191 poll_interval=300, keepalive_interval = 60, |
192 username=None, password=None, | |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
193 dump_reports=False, no_loop=False, form_auth=False): |
313 | 194 """Create the build slave instance. |
195 | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
196 :param urls: a list of URLs of the build masters to connect to, or a |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
197 single-element list containing the path to a build recipe |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
198 file |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
199 :param name: the name with which this slave should identify itself |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
200 :param config: the path to the slave configuration file |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
201 :param dry_run: wether the build outcome should not be reported back |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
202 to the master |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
203 :param work_dir: the working directory to use for build execution |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
204 :param build_dir: the pattern to use for naming the build subdir |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
205 :param keep_files: whether files and directories created for build |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
206 execution should be kept when done |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
207 :param single_build: whether this slave should exit after completing a |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
208 single build, or continue processing builds forever |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
209 :param poll_interval: the time in seconds to wait between requesting |
451 | 210 builds from the build master (default is five |
211 minutes) | |
871 | 212 :param keepalive_interval: the time in seconds to wait between sending |
786 | 213 keepalive heartbeats (default is 30 seconds) |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
214 :param username: the username to use when authentication against the |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
215 build master is requested |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
216 :param password: the password to use when authentication is needed |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
217 :param dump_reports: whether report data should be written to the |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
218 standard output, in addition to being transmitted |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
219 to the build master |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
220 :param no_loop: for this slave to just perform a single check, regardless |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
221 of whether a build is done or not |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
222 :param form_auth: login using AccountManager HTML form instead of |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
223 HTTP authentication for all urls |
313 | 224 """ |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
225 self.urls = urls |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
226 self.local = len(urls) == 1 and not urls[0].startswith('http://') \ |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
227 and not urls[0].startswith('https://') |
392 | 228 if name is None: |
229 name = platform.node().split('.', 1)[0].lower() | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
230 self.name = name |
392 | 231 self.config = Configuration(config) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
232 self.dry_run = dry_run |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
233 if not work_dir: |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
234 work_dir = tempfile.mkdtemp(prefix='bitten') |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
235 elif not os.path.exists(work_dir): |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
236 os.makedirs(work_dir) |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
237 self.work_dir = work_dir |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
238 self.build_dir = build_dir |
241
d3c8a74922cd
* Add an option to `bitten-slave` that tells it to keep any files in the working directory.
cmlenz
parents:
233
diff
changeset
|
239 self.keep_files = keep_files |
377 | 240 self.single_build = single_build |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
241 self.no_loop = no_loop |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
242 self.poll_interval = poll_interval |
786 | 243 self.keepalive_interval = keepalive_interval |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
244 self.dump_reports = dump_reports |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
245 self.cookiejar = cookielib.CookieJar() |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
246 self.username = username \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
247 or self.config['authentication.username'] or '' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
248 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
249 if not self.local: |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
250 self.password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
251 if self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
252 log.debug('Enabling authentication with username %r', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
253 self.username) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
254 self.form_auth = form_auth |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
255 password = password \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
256 or self.config['authentication.password'] or '' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
257 self.config.packages.pop('authentication', None) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
258 urls = [url[:-7] for url in urls] |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
259 self.password_mgr.add_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
260 None, urls, self.username, password) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
261 self.auth_map = dict(map(lambda x: (x, False), urls)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
262 |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
263 def _get_opener(self): |
652
de04ce69da53
0.6dev: Removing code and updated docs related to Trac < 0.11 and Python < 2.4 (base requirements for Bitten 0.6).
osimons
parents:
649
diff
changeset
|
264 opener = urllib2.build_opener(urllib2.HTTPErrorProcessor()) |
896 | 265 opener.add_handler(HTTPBasicAuthHandler(self.password_mgr)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
266 opener.add_handler(urllib2.HTTPDigestAuthHandler(self.password_mgr)) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
267 opener.add_handler(urllib2.HTTPCookieProcessor(self.cookiejar)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
268 return opener |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
269 opener = property(_get_opener) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
270 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
271 def request(self, method, url, body=None, headers=None): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
272 log.debug('Sending %s request to %r', method, url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
273 req = SaneHTTPRequest(method, url, body, headers or {}) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
274 try: |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
275 resp = self.opener.open(req) |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
276 if not hasattr(resp, 'code'): |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
277 resp.code = 200 |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
278 return resp |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
279 except urllib2.HTTPError, e: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
280 if e.code >= 300: |
647
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
281 if hasattr(e, 'headers') and \ |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
282 e.headers.getheader('Content-Type', '' |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
283 ).startswith('text/plain'): |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
284 content = e.read() |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
285 else: |
786 | 286 content = 'no message available' |
287 log.debug('Server returned error %d: %s (%s)', | |
288 e.code, e.msg, content) | |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
289 raise |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
290 return e |
29 | 291 |
392 | 292 def run(self): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
293 if self.local: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
294 fileobj = open(self.urls[0]) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
295 try: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
296 self._execute_build(None, fileobj) |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
297 finally: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
298 fileobj.close() |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
299 return EX_OK |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
300 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
301 urls = [] |
392 | 302 while True: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
303 if not urls: |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
304 urls[:] = self.urls |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
305 url = urls.pop(0) |
392 | 306 try: |
307 try: | |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
308 if self.username and not self.auth_map.get(url): |
716 | 309 login_url = '%s/login?referer=%s' % (url[:-7], |
310 urllib.quote_plus(url)) | |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
311 # First request to url, authentication needed |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
312 if self.form_auth: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
313 log.debug('Performing http form authentication') |
716 | 314 resp = self.request('POST', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
315 match = FORM_TOKEN_RE.search(resp.read()) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
316 if not match: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
317 log.error("Project %s does not support form " |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
318 "authentication" % url[:-7]) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
319 raise ExitSlave(EX_NOPERM) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
320 values = {'user': self.username, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
321 'password': |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
322 self.password_mgr.find_user_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
323 None, url)[1], |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
324 'referer': '', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
325 '__FORM_TOKEN': match.group(1)} |
716 | 326 self.request('POST', login_url, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
327 body=urllib.urlencode(values)) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
328 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
329 log.debug('Performing basic/digest authentication') |
716 | 330 self.request('HEAD', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
331 self.auth_map[url] = True |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
332 elif self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
333 log.debug('Reusing authentication information.') |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
334 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
335 log.debug('Authentication not provided. Attempting to ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
336 'execute build anonymously.') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
337 job_done = self._create_build(url) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
338 if job_done: |
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
339 continue |
465
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
340 except urllib2.HTTPError, e: |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
341 # HTTPError doesn't have the "reason" attribute of URLError |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
342 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
343 raise ExitSlave(EX_UNAVAILABLE) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
344 except urllib2.URLError, e: |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
345 # Is this a temporary network glitch or something a bit |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
346 # more severe? |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
347 if isinstance(e.reason, socket.error) and \ |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
348 e.reason.args[0] in temp_net_errors: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
349 log.warning(e) |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
350 else: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
351 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
352 raise ExitSlave(EX_UNAVAILABLE) |
490 | 353 except ExitSlave, e: |
354 return e.exit_code | |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
355 if self.no_loop: |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
356 break |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
357 time.sleep(self.poll_interval) |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
358 |
392 | 359 def quit(self): |
360 log.info('Shutting down') | |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
361 raise ExitSlave(EX_OK) |
392 | 362 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
363 def _create_build(self, url): |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
648
diff
changeset
|
364 xml = xmlio.Element('slave', name=self.name, version=PROTOCOL_VERSION)[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
365 xmlio.Element('platform', processor=self.config['processor'])[ |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
366 self.config['machine'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
367 ], |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
368 xmlio.Element('os', family=self.config['family'], |
244
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
242
diff
changeset
|
369 version=self.config['version'])[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
370 self.config['os'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
371 ], |
29 | 372 ] |
392 | 373 |
374 log.debug('Configured packages: %s', self.config.packages) | |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
375 for package, properties in self.config.packages.items(): |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
376 xml.append(xmlio.Element('package', name=package, **properties)) |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
377 |
392 | 378 body = str(xml) |
379 log.debug('Sending slave configuration: %s', body) | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
380 resp = self.request('POST', url, body, { |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
381 'Content-Length': str(len(body)), |
392 | 382 'Content-Type': 'application/x-bitten+xml' |
383 }) | |
277 | 384 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
385 if resp.code == 201: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
386 self._initiate_build(resp.info().get('location')) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
387 return True |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
388 elif resp.code == 204: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
389 log.info('No pending builds') |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
390 return False |
392 | 391 else: |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
392 log.error('Unexpected response (%d %s)', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
393 raise ExitSlave(EX_PROTOCOL) |
56 | 394 |
392 | 395 def _initiate_build(self, build_url): |
418 | 396 log.info('Build pending at %s', build_url) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
397 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
398 resp = self.request('GET', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
399 if resp.code == 200: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
400 self._execute_build(build_url, resp) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
401 else: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
402 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
403 self._cancel_build(build_url, exit_code=EX_PROTOCOL) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
404 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
405 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
406 self._cancel_build(build_url) |
392 | 407 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
408 def _execute_build(self, build_url, fileobj): |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
409 build_id = build_url and int(build_url.split('/')[-1]) or 0 |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
410 xml = xmlio.parse(fileobj) |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
411 basedir = '' |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
412 try: |
786 | 413 if not self.local: |
797 | 414 keepalive_thread = KeepAliveThread(self.opener, build_url, |
415 self.single_build, self.keepalive_interval) | |
786 | 416 keepalive_thread.start() |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
417 recipe = Recipe(xml, os.path.join(self.work_dir, self.build_dir), |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
418 self.config) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
419 basedir = recipe.ctxt.basedir |
490 | 420 log.debug('Running build in directory %s' % basedir) |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
421 if not os.path.exists(basedir): |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
422 os.mkdir(basedir) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
423 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
424 for step in recipe: |
755 | 425 try: |
426 log.info('Executing build step %r, onerror = %s', step.id, step.onerror) | |
427 if not self._execute_step(build_url, recipe, step): | |
428 log.warning('Stopping build due to failure') | |
429 break | |
430 except Exception, e: | |
431 log.error('Exception raised processing step %s. Reraising %s', step.id, e) | |
432 raise | |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
433 else: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
434 log.info('Build completed') |
461
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
435 if self.dry_run: |
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
436 self._cancel_build(build_url) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
437 finally: |
797 | 438 if not self.local: |
439 keepalive_thread.stop() | |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
440 if not self.keep_files and os.path.isdir(basedir): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
441 log.debug('Removing build directory %s' % basedir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
442 _rmtree(basedir) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
443 if self.single_build: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
444 log.info('Exiting after single build completed.') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
445 raise ExitSlave(EX_OK) |
392 | 446 |
447 def _execute_step(self, build_url, recipe, step): | |
448 failed = False | |
759 | 449 started = int(time.time()) |
450 xml = xmlio.Element('result', step=step.id) | |
392 | 451 try: |
452 for type, category, generator, output in \ | |
453 step.execute(recipe.ctxt): | |
454 if type == Recipe.ERROR: | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
455 failed = True |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
456 if type == Recipe.REPORT and self.dump_reports: |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
457 print output |
836 | 458 if type == Recipe.ATTACH: |
459 # Attachments are added out-of-band due to major | |
460 # performance issues with inlined base64 xml content | |
461 self._attach_file(build_url, recipe, output) | |
392 | 462 xml.append(xmlio.Element(type, category=category, |
463 generator=generator)[ | |
464 output | |
465 ]) | |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
466 except KeyboardInterrupt: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
467 log.warning('Build interrupted') |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
468 self._cancel_build(build_url) |
392 | 469 except BuildError, e: |
628
05686657e989
0.6dev: Implement a `subprocess` implementation of `execute()` that is now the default for all slaves where this is available - essentially all slaves with Python 2.4 and higher, or where module is installed separately. This fixes:
osimons
parents:
610
diff
changeset
|
470 log.error('Build step %r failed', step.id) |
392 | 471 failed = True |
472 except Exception, e: | |
473 log.error('Internal error in build step %r', step.id, exc_info=True) | |
474 failed = True | |
759 | 475 xml.attr['duration'] = (time.time() - started) |
392 | 476 if failed: |
477 xml.attr['status'] = 'failure' | |
478 else: | |
479 xml.attr['status'] = 'success' | |
480 log.info('Build step %s completed successfully', step.id) | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
481 |
424
bc81d7b069ff
Rudimentary implementation of ``dry-run`` option in build slave. Builds are still put in in progress mode, but no results are transmitted back, and the build is automatically invalidated after the slave is done.
cmlenz
parents:
420
diff
changeset
|
482 if not self.local and not self.dry_run: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
483 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
484 resp = self.request('POST', build_url + '/steps/', str(xml), { |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
485 'Content-Type': 'application/x-bitten+xml' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
486 }) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
487 if resp.code != 201: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
488 log.error('Unexpected response (%d): %s', resp.code, |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
489 resp.msg) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
490 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
491 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
492 self._cancel_build(build_url) |
392 | 493 return not failed or step.onerror != 'fail' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
494 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
495 def _cancel_build(self, build_url, exit_code=EX_OK): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
496 log.info('Cancelling build at %s', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
497 if not self.local: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
498 resp = self.request('DELETE', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
499 if resp.code not in (200, 204): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
500 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
490 | 501 raise ExitSlave(exit_code) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
502 |
836 | 503 def _attach_file(self, build_url, recipe, attachment): |
504 form_token = recipe._root.attr.get('form_token', '') | |
505 if self.local or self.dry_run or not form_token: | |
506 log.info('Attachment %s not sent due to current slave options', | |
906 | 507 attachment.attr['filename']) |
836 | 508 return |
509 resource_type = attachment.attr['resource'] | |
510 url = str(build_url + '/attach/' + resource_type) | |
511 path = recipe.ctxt.resolve(attachment.attr['filename']) | |
512 filename = os.path.basename(path) | |
513 log.debug('Attaching file %s to %s...', attachment.attr['filename'], | |
514 resource_type) | |
890 | 515 f = open(path, 'rb') |
836 | 516 try: |
517 data, content_type = encode_multipart_formdata({ | |
518 'file': (filename, f.read()), | |
519 'description': attachment.attr['description'], | |
520 '__FORM_TOKEN': form_token}) | |
521 finally: | |
522 f.close() | |
523 resp = self.request('POST', url , data, { | |
524 'Content-Type': content_type}) | |
525 if not resp.code == 201: | |
526 msg = 'Error attaching %s to %s' | |
527 log.error(msg, filename, resource_type) | |
528 raise BuildError(msg, filename, resource_type) | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
529 |
392 | 530 class ExitSlave(Exception): |
531 """Exception used internally by the slave to signal that the slave process | |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
532 should be stopped. |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
533 """ |
490 | 534 def __init__(self, exit_code): |
535 self.exit_code = exit_code | |
536 Exception.__init__(self) | |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
537 |
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
538 |
31 | 539 def main(): |
313 | 540 """Main entry point for running the build slave.""" |
56 | 541 from bitten import __version__ as VERSION |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
542 from optparse import OptionParser |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
543 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
544 parser = OptionParser(usage='usage: %prog [options] url1 [url2] ...', |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
545 version='%%prog %s' % VERSION) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
546 parser.add_option('--name', action='store', dest='name', |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
547 help='name of this slave (defaults to host name)') |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
548 parser.add_option('-f', '--config', action='store', dest='config', |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
549 metavar='FILE', help='path to configuration file') |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
550 parser.add_option('-u', '--user', dest='username', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
551 help='the username to use for authentication') |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
552 parser.add_option('-p', '--password', dest='password', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
553 help='the password to use when authenticating') |
587
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
554 def _ask_password(option, opt_str, value, parser): |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
555 from getpass import getpass |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
556 parser.values.password = getpass('Passsword: ') |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
557 parser.add_option('-P', '--ask-password', action='callback', |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
558 callback=_ask_password, help='Prompt for password') |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
559 parser.add_option('--form-auth', action='store_true', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
560 dest='form_auth', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
561 help='login using AccountManager HTML form instead of ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
562 'HTTP authentication for all urls') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
563 |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
564 group = parser.add_option_group('building') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
565 group.add_option('-d', '--work-dir', action='store', dest='work_dir', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
566 metavar='DIR', help='working directory for builds') |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
567 group.add_option('--build-dir', action='store', dest='build_dir', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
568 default = 'build_${config}_${build}', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
569 help='name pattern for the build dir to use inside the ' |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
570 'working dir ["%default"]') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
571 group.add_option('-k', '--keep-files', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
572 dest='keep_files', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
573 help='don\'t delete files after builds') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
574 group.add_option('-s', '--single', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
575 dest='single_build', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
576 help='exit after completing a single build') |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
577 group.add_option('', '--no-loop', action='store_true', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
578 dest='no_loop', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
579 help='exit after completing a single check and running ' |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
580 'the required builds') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
581 group.add_option('-n', '--dry-run', action='store_true', dest='dry_run', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
582 help='don\'t report results back to master') |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
583 group.add_option('-i', '--interval', dest='interval', metavar='SECONDS', |
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
584 type='int', help='time to wait between requesting builds') |
786 | 585 group.add_option('-b', '--keepalive_interval', dest='keepalive_interval', metavar='SECONDS', type='int', help='time to wait between keepalive heartbeats') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
586 group = parser.add_option_group('logging') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
587 group.add_option('-l', '--log', dest='logfile', metavar='FILENAME', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
588 help='write log messages to FILENAME') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
589 group.add_option('-v', '--verbose', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
590 const=logging.DEBUG, help='print as much as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
591 group.add_option('-q', '--quiet', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
592 const=logging.WARN, help='print as little as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
593 group.add_option('--dump-reports', action='store_true', dest='dump_reports', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
594 help='whether report data should be printed') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
595 |
242
372d1de2e3ec
* Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`.
cmlenz
parents:
241
diff
changeset
|
596 parser.set_defaults(dry_run=False, keep_files=False, |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
597 loglevel=logging.INFO, single_build=False, no_loop=False, |
786 | 598 dump_reports=False, interval=300, keepalive_interval=60, |
599 form_auth=False) | |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
600 options, args = parser.parse_args() |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
601 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
602 if len(args) < 1: |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
603 parser.error('incorrect number of arguments') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
604 urls = args |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
605 |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
606 logger = logging.getLogger('bitten') |
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
607 logger.setLevel(options.loglevel) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
608 handler = logging.StreamHandler() |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
609 handler.setLevel(options.loglevel) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
610 formatter = logging.Formatter('[%(levelname)-8s] %(message)s') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
611 handler.setFormatter(formatter) |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
612 logger.addHandler(handler) |
339
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
613 if options.logfile: |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
614 handler = logging.FileHandler(options.logfile) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
615 handler.setLevel(options.loglevel) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
616 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: ' |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
617 '%(message)s') |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
618 handler.setFormatter(formatter) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
619 logger.addHandler(handler) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
620 |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
621 log.info("Slave launched at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
622 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
623 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
624 slave = None |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
625 try: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
626 slave = BuildSlave(urls, name=options.name, config=options.config, |
392 | 627 dry_run=options.dry_run, work_dir=options.work_dir, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
628 build_dir=options.build_dir, |
392 | 629 keep_files=options.keep_files, |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
630 single_build=options.single_build, |
543
ce16a5dcf8b3
Pass `--no-loop` option to the `BuildSlave` (fixes #375, from the patch there - thanks!)
dfraser
parents:
542
diff
changeset
|
631 no_loop=options.no_loop, |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
632 poll_interval=options.interval, |
786 | 633 keepalive_interval=options.keepalive_interval, |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
634 username=options.username, password=options.password, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
635 dump_reports=options.dump_reports, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
636 form_auth=options.form_auth) |
392 | 637 try: |
490 | 638 exit_code = slave.run() |
392 | 639 except KeyboardInterrupt: |
640 slave.quit() | |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
641 except ConfigFileNotFound, e: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
642 log.error(e) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
643 exit_code = EX_IOERR |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
644 except MissingSectionHeaderError: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
645 log.error("Error parsing configuration file %r. Wrong format?" \ |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
646 % options.config) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
647 exit_code = EX_IOERR |
490 | 648 except ExitSlave, e: |
649 exit_code = e.exit_code | |
31 | 650 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
651 if slave and not (options.work_dir or options.keep_files): |
576
4b75e3f6f27c
0.6dev: Fixing #229 again - keeping `slave.work_dir` if either `--keep_files` or `--work_dir=` are part of options.
osimons
parents:
570
diff
changeset
|
652 log.debug('Removing working directory %s' % slave.work_dir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
653 _rmtree(slave.work_dir) |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
654 |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
655 log.info("Slave exited at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
656 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
657 |
490 | 658 return exit_code |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
659 |
31 | 660 if __name__ == '__main__': |
490 | 661 sys.exit(main()) |