Mercurial > bitten > bitten-test
annotate bitten/slave.py @ 913:5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
author | hodgestar |
---|---|
date | Thu, 19 May 2011 14:02:55 +0000 |
parents | 0ae349142bf2 |
children |
rev | line source |
---|---|
379 | 1 # -*- coding: utf-8 -*- |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
2 # |
832 | 3 # Copyright (C) 2007-2010 Edgewall Software |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
4 # Copyright (C) 2005-2007 Christopher Lenz <cmlenz@gmx.de> |
163 | 5 # All rights reserved. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
6 # |
163 | 7 # This software is licensed as described in the file COPYING, which |
8 # you should have received as part of this distribution. The terms | |
408
933105ab516b
Update file headers and other stuff pointing to the old home.
cmlenz
parents:
405
diff
changeset
|
9 # are also available at http://bitten.edgewall.org/wiki/License. |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
10 |
313 | 11 """Implementation of the build slave.""" |
12 | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
13 from datetime import datetime |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
14 import errno |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
15 import urllib |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
16 import urllib2 |
15
06207499c58c
* Use logging in the BEEP core as well as in the master and slave scripts. Closes #4.
cmlenz
parents:
14
diff
changeset
|
17 import logging |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
18 import os |
66
df40a4eb7553
Use the standard {{{platform}}} module for reporting of system info by slaves. Closes #6.
cmlenz
parents:
63
diff
changeset
|
19 import platform |
90
2c4e104afef8
The build master now transmits snapshot archives without blocking while reading the file and sending the BEEP frames. Closes #17.
cmlenz
parents:
88
diff
changeset
|
20 import shutil |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
21 import socket |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
22 import sys |
42
efa525876b1e
Basic infrastructure for transmission of snapshot archives to build slaves. See #8.
cmlenz
parents:
34
diff
changeset
|
23 import tempfile |
392 | 24 import time |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
25 import re |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
26 import cookielib |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
27 import threading |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
28 import mimetools |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
29 from ConfigParser import MissingSectionHeaderError |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
30 |
654
cc1fefd5d814
0.6dev: Moved a constant declaration so that master should still work with Python 2.4. Updated docs with regards to Python requirements + added a changelog message from cmlenz.
osimons
parents:
652
diff
changeset
|
31 from bitten import PROTOCOL_VERSION |
61
47ab019508dd
Moved {{{BuildError}}} class into package {{{bitten.build}}}.
cmlenz
parents:
60
diff
changeset
|
32 from bitten.build import BuildError |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
33 from bitten.build.config import Configuration, ConfigFileNotFound |
436
cfbc9ee622d5
Finish the move of build configuration management into the admin interface.
cmlenz
parents:
424
diff
changeset
|
34 from bitten.recipe import Recipe |
392 | 35 from bitten.util import xmlio |
895
7d93d6358fe0
Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents:
891
diff
changeset
|
36 from bitten.util.compat import HTTPBasicAuthHandler |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
37 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
38 EX_OK = getattr(os, "EX_OK", 0) |
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
39 EX_UNAVAILABLE = getattr(os, "EX_UNAVAILABLE", 69) |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
40 EX_IOERR = getattr(os, "EX_IOERR", 74) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
41 EX_PROTOCOL = getattr(os, "EX_PROTOCOL", 76) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
42 EX_NOPERM = getattr(os, "EX_NOPERM", 77) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
43 |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
44 FORM_TOKEN_RE = re.compile('__FORM_TOKEN\" value=\"(.+)\"') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
45 |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
46 __all__ = ['BuildSlave', 'ExitSlave'] |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
47 __docformat__ = 'restructuredtext en' |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
48 |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
49 log = logging.getLogger('bitten.slave') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
50 |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
51 # List of network errors which are usually temporary and non critical. |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
52 temp_net_errors = [errno.ENETUNREACH, errno.ENETDOWN, errno.ETIMEDOUT, |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
53 errno.ECONNREFUSED] |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
54 |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
55 def _rmtree(root): |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
56 """Catch shutil.rmtree failures on Windows when files are read-only, and only remove if root exists.""" |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
57 def _handle_error(fn, path, excinfo): |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
58 os.chmod(path, 0666) |
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
59 fn(path) |
549
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
60 if os.path.exists(root): |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
61 return shutil.rmtree(root, onerror=_handle_error) |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
62 else: |
cb137f2121b9
Only actually run `rmtree` if the root directory exists (fixes #381; patch is from there)
dfraser
parents:
545
diff
changeset
|
63 return False |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
64 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
65 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
66 class SaneHTTPRequest(urllib2.Request): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
67 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
68 def __init__(self, method, url, data=None, headers={}): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
69 urllib2.Request.__init__(self, url, data, headers) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
70 self.method = method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
71 |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
72 def get_method(self): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
73 if self.method is None: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
74 self.method = self.has_data() and 'POST' or 'GET' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
75 return self.method |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
76 |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
77 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
78 def encode_multipart_formdata(fields): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
79 """ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
80 Given a dictionary field parameters, returns the HTTP request body and the |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
81 content_type (which includes the boundary string), to be used with an |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
82 httplib-like call. |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
83 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
84 Normal key/value items are treated as regular parameters, but key/tuple |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
85 items are treated as files, where a value tuple is a (filename, data) tuple. |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
86 |
870 | 87 For example:: |
88 | |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
89 fields = { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
90 'foo': 'bar', |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
91 'foofile': ('foofile.txt', 'contents of foofile'), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
92 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
93 body, content_type = encode_multipart_formdata(fields) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
94 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
95 Note: Adapted from http://code.google.com/p/urllib3/ (MIT license) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
96 """ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
97 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
98 BOUNDARY = mimetools.choose_boundary() |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
99 ENCODE_TEMPLATE= "--%(boundary)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
100 "Content-Disposition: form-data; name=\"%(name)s\"\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
101 "\r\n%(value)s\r\n" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
102 ENCODE_TEMPLATE_FILE = "--%(boundary)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
103 "Content-Disposition: form-data; name=\"%(name)s\"; " \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
104 "filename=\"%(filename)s\"\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
105 "Content-Type: %(contenttype)s\r\n" \ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
106 "\r\n%(value)s\r\n" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
107 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
108 body = "" |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
109 for key, value in fields.iteritems(): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
110 if isinstance(value, tuple): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
111 filename, value = value |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
112 body += ENCODE_TEMPLATE_FILE % { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
113 'boundary': BOUNDARY, |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
114 'name': str(key), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
115 'value': str(value), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
116 'filename': str(filename), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
117 'contenttype': 'application/octet-stream' |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
118 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
119 else: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
120 body += ENCODE_TEMPLATE % { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
121 'boundary': BOUNDARY, |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
122 'name': str(key), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
123 'value': str(value) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
124 } |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
125 body += '--%s--\r\n' % BOUNDARY |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
126 content_type = 'multipart/form-data; boundary=%s' % BOUNDARY |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
127 return body, content_type |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
128 |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
129 |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
130 class KeepAliveThread(threading.Thread): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
131 "A thread to periodically send keep-alive messages to the master" |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
132 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
133 def __init__(self, opener, build_url, single_build, keepalive_interval): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
134 threading.Thread.__init__(self, None, None, "KeepaliveThread") |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
135 self.build_url = build_url |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
136 self.keepalive_interval = keepalive_interval |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
137 self.single_build = single_build |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
138 self.last_keepalive = int(time.time()) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
139 self.kill = False |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
140 self.opener = opener |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
141 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
142 def keepalive(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
143 log.debug('Sending keepalive') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
144 method = 'POST' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
145 url = self.build_url + '/keepalive/' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
146 body = None |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
147 shutdown = False |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
148 headers = { |
891
60d0807334ec
Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents:
889
diff
changeset
|
149 'Content-Type': 'application/x-bitten+xml', |
60d0807334ec
Set 'Content-Length' header on keep-alive requests. Fixes #667. Thanks eblot.
osimons
parents:
889
diff
changeset
|
150 'Content-Length': '0' |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
151 } |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
152 log.debug('Sending %s request to %r', method, url) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
153 req = SaneHTTPRequest(method, url, body, headers or {}) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
154 try: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
155 return self.opener.open(req) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
156 except urllib2.HTTPError, e: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
157 # a conflict error lets us know that we've been |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
158 # invalidated. Ideally, we'd engineer something to stop any |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
159 # running steps in progress, but killing threads is tricky |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
160 # stuff. For now, we'll wait for whatever's going |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
161 # on to stop, and the main thread'll figure out that we've |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
162 # been invalidated. |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
163 log.warning('Server returned keepalive error %d: %s', e.code, e.msg) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
164 except: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
165 log.warning('Server returned unknown keepalive error') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
166 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
167 def run(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
168 log.debug('Keepalive thread starting.') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
169 while (not self.kill): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
170 now = int(time.time()) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
171 if (self.last_keepalive + self.keepalive_interval) < now: |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
172 self.keepalive() |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
173 self.last_keepalive = now |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
174 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
175 time.sleep(1) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
176 log.debug('Keepalive thread exiting.') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
177 |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
178 def stop(self): |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
179 log.debug('Stopping keepalive thread') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
180 self.kill = True |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
181 self.join(30) |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
182 log.debug('Keepalive thread stopped') |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
183 |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
184 |
392 | 185 class BuildSlave(object): |
542 | 186 """HTTP client implementation for the build slave.""" |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
187 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
188 def __init__(self, urls, name=None, config=None, dry_run=False, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
189 work_dir=None, build_dir="build_${build}", |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
190 keep_files=False, single_build=False, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
191 poll_interval=300, keepalive_interval = 60, |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
192 username=None, password=None, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
193 dump_reports=False, no_loop=False, form_auth=False): |
313 | 194 """Create the build slave instance. |
195 | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
196 :param urls: a list of URLs of the build masters to connect to, or a |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
197 single-element list containing the path to a build recipe |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
198 file |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
199 :param name: the name with which this slave should identify itself |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
200 :param config: the path to the slave configuration file |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
201 :param dry_run: wether the build outcome should not be reported back |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
202 to the master |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
203 :param work_dir: the working directory to use for build execution |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
204 :param build_dir: the pattern to use for naming the build subdir |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
205 :param keep_files: whether files and directories created for build |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
206 execution should be kept when done |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
207 :param single_build: whether this slave should exit after completing a |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
208 single build, or continue processing builds forever |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
209 :param poll_interval: the time in seconds to wait between requesting |
451 | 210 builds from the build master (default is five |
211 minutes) | |
870 | 212 :param keepalive_interval: the time in seconds to wait between sending |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
213 keepalive heartbeats (default is 30 seconds) |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
214 :param username: the username to use when authentication against the |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
215 build master is requested |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
216 :param password: the password to use when authentication is needed |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
217 :param dump_reports: whether report data should be written to the |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
218 standard output, in addition to being transmitted |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
219 to the build master |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
220 :param no_loop: for this slave to just perform a single check, regardless |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
221 of whether a build is done or not |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
222 :param form_auth: login using AccountManager HTML form instead of |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
223 HTTP authentication for all urls |
313 | 224 """ |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
225 self.local = len(urls) == 1 and not urls[0].startswith('http://') \ |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
226 and not urls[0].startswith('https://') |
913
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
227 if self.local: |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
228 self.urls = urls |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
229 else: |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
230 self.urls = [ |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
231 not url.endswith('/builds') and url.rstrip('/') + '/builds' |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
232 or url for url in urls] |
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
233 |
392 | 234 if name is None: |
235 name = platform.node().split('.', 1)[0].lower() | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
236 self.name = name |
392 | 237 self.config = Configuration(config) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
238 self.dry_run = dry_run |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
239 if not work_dir: |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
240 work_dir = tempfile.mkdtemp(prefix='bitten') |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
241 elif not os.path.exists(work_dir): |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
242 os.makedirs(work_dir) |
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
243 self.work_dir = work_dir |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
244 self.build_dir = build_dir |
241
d3c8a74922cd
* Add an option to `bitten-slave` that tells it to keep any files in the working directory.
cmlenz
parents:
233
diff
changeset
|
245 self.keep_files = keep_files |
377 | 246 self.single_build = single_build |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
247 self.no_loop = no_loop |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
248 self.poll_interval = poll_interval |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
249 self.keepalive_interval = keepalive_interval |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
250 self.dump_reports = dump_reports |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
251 self.cookiejar = cookielib.CookieJar() |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
252 self.username = username \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
253 or self.config['authentication.username'] or '' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
254 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
255 if not self.local: |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
256 self.password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
257 if self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
258 log.debug('Enabling authentication with username %r', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
259 self.username) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
260 self.form_auth = form_auth |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
261 password = password \ |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
262 or self.config['authentication.password'] or '' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
263 self.config.packages.pop('authentication', None) |
913
5bfdd7348d6f
Allow passing the build master's URL without the /builds suffix when calling bitten-slave by appending /builds to URLs that don't have it. This also prevents bitten-slave from contacting the wrong domain if a bad URL is passed.
hodgestar
parents:
905
diff
changeset
|
264 urls = [url[:-len('/builds')] for url in self.urls] |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
265 self.password_mgr.add_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
266 None, urls, self.username, password) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
267 self.auth_map = dict(map(lambda x: (x, False), urls)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
268 |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
269 def _get_opener(self): |
652
de04ce69da53
0.6dev: Removing code and updated docs related to Trac < 0.11 and Python < 2.4 (base requirements for Bitten 0.6).
osimons
parents:
649
diff
changeset
|
270 opener = urllib2.build_opener(urllib2.HTTPErrorProcessor()) |
895
7d93d6358fe0
Use our own HTTPBasicAuthHandler under Python 2.6 to avoid issue http://bugs.python.org/issue8797. Fixes #658.
hodgestar
parents:
891
diff
changeset
|
271 opener.add_handler(HTTPBasicAuthHandler(self.password_mgr)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
272 opener.add_handler(urllib2.HTTPDigestAuthHandler(self.password_mgr)) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
273 opener.add_handler(urllib2.HTTPCookieProcessor(self.cookiejar)) |
569
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
274 return opener |
74bc339ce7d5
0.6dev: Fixing digest authentication in #330. Instead of having a single `urllib2` opener for the life of the slave, a new opener object is created for each request.
osimons
parents:
560
diff
changeset
|
275 opener = property(_get_opener) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
276 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
277 def request(self, method, url, body=None, headers=None): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
278 log.debug('Sending %s request to %r', method, url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
279 req = SaneHTTPRequest(method, url, body, headers or {}) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
280 try: |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
281 resp = self.opener.open(req) |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
282 if not hasattr(resp, 'code'): |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
283 resp.code = 200 |
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
284 return resp |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
285 except urllib2.HTTPError, e: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
286 if e.code >= 300: |
647
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
287 if hasattr(e, 'headers') and \ |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
288 e.headers.getheader('Content-Type', '' |
6fe6fc8ee36c
0.6dev: Follow-up to [719]. Seems some http errors don't populate `headers` (at least apache digest auth challenge didn't).
osimons
parents:
645
diff
changeset
|
289 ).startswith('text/plain'): |
645
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
290 content = e.read() |
8c824b14e1c5
0.6dev: Switching `master.py` to use new `self._send_response()` and `self._send_error() methods. Simplifies code, but most importantly for errors it allows a consistent method for transmitting plain-text error messages to the slave (that the slave will now output as part of debug logging).
osimons
parents:
628
diff
changeset
|
291 else: |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
292 content = 'no message available' |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
293 log.debug('Server returned error %d: %s (%s)', |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
294 e.code, e.msg, content) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
295 raise |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
296 return e |
29 | 297 |
392 | 298 def run(self): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
299 if self.local: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
300 fileobj = open(self.urls[0]) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
301 try: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
302 self._execute_build(None, fileobj) |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
303 finally: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
304 fileobj.close() |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
305 return EX_OK |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
306 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
307 urls = [] |
392 | 308 while True: |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
309 if not urls: |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
310 urls[:] = self.urls |
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
311 url = urls.pop(0) |
392 | 312 try: |
313 try: | |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
314 if self.username and not self.auth_map.get(url): |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
315 login_url = '%s/login?referer=%s' % (url[:-7], |
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
316 urllib.quote_plus(url)) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
317 # First request to url, authentication needed |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
318 if self.form_auth: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
319 log.debug('Performing http form authentication') |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
320 resp = self.request('POST', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
321 match = FORM_TOKEN_RE.search(resp.read()) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
322 if not match: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
323 log.error("Project %s does not support form " |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
324 "authentication" % url[:-7]) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
325 raise ExitSlave(EX_NOPERM) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
326 values = {'user': self.username, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
327 'password': |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
328 self.password_mgr.find_user_password( |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
329 None, url)[1], |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
330 'referer': '', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
331 '__FORM_TOKEN': match.group(1)} |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
332 self.request('POST', login_url, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
333 body=urllib.urlencode(values)) |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
334 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
335 log.debug('Performing basic/digest authentication') |
715
cd4405466a77
Add a `referer=` argument when logging in, and redirect to a Bitten URL so that slave doesn't need non-build permissions. Closes #459.
osimons
parents:
675
diff
changeset
|
336 self.request('HEAD', login_url) |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
337 self.auth_map[url] = True |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
338 elif self.username: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
339 log.debug('Reusing authentication information.') |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
340 else: |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
341 log.debug('Authentication not provided. Attempting to ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
342 'execute build anonymously.') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
343 job_done = self._create_build(url) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
344 if job_done: |
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
345 continue |
465
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
346 except urllib2.HTTPError, e: |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
347 # HTTPError doesn't have the "reason" attribute of URLError |
04205ba5dc98
handle HTTPErrors separately to work around them not having a "reason" attribute like URLError
mgood
parents:
461
diff
changeset
|
348 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
349 raise ExitSlave(EX_UNAVAILABLE) |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
350 except urllib2.URLError, e: |
457
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
351 # Is this a temporary network glitch or something a bit |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
352 # more severe? |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
353 if isinstance(e.reason, socket.error) and \ |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
354 e.reason.args[0] in temp_net_errors: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
355 log.warning(e) |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
356 else: |
b267ad4b93e9
Make the slave differentiate between benign and fatal network errors and
jonas
parents:
456
diff
changeset
|
357 log.error(e) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
358 raise ExitSlave(EX_UNAVAILABLE) |
490 | 359 except ExitSlave, e: |
360 return e.exit_code | |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
361 if self.no_loop: |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
362 break |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
363 time.sleep(self.poll_interval) |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
364 |
392 | 365 def quit(self): |
366 log.info('Shutting down') | |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
367 raise ExitSlave(EX_OK) |
392 | 368 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
369 def _create_build(self, url): |
649
eed0149c302a
0.6dev: Switching to use the new cookie-support, and using trac auth/session ID as unique identification for linking builds with slaves. This overcomes problems with IP address not being unique behind NAT, and also where IP address may change during a build. Closes #421.
osimons
parents:
648
diff
changeset
|
370 xml = xmlio.Element('slave', name=self.name, version=PROTOCOL_VERSION)[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
371 xmlio.Element('platform', processor=self.config['processor'])[ |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
372 self.config['machine'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
373 ], |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
374 xmlio.Element('os', family=self.config['family'], |
244
1aa624af9ebb
* Allowing specifying the main entry point of a module in `<python:exec>`. This can be used to execute Python scripts in modules that don't map to files on the file system. See #49.
cmlenz
parents:
242
diff
changeset
|
375 version=self.config['version'])[ |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
376 self.config['os'] |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
377 ], |
29 | 378 ] |
392 | 379 |
380 log.debug('Configured packages: %s', self.config.packages) | |
233
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
381 for package, properties in self.config.packages.items(): |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
382 xml.append(xmlio.Element('package', name=package, **properties)) |
8f816147620f
* Moved SlaveConfiguration logic into new module ([source:/trunk/bitten/build/config.py bitten.build.config]).
cmlenz
parents:
221
diff
changeset
|
383 |
392 | 384 body = str(xml) |
385 log.debug('Sending slave configuration: %s', body) | |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
386 resp = self.request('POST', url, body, { |
559
bf3e0c2f6b80
Apply patch to allow Python 2.3 compatibility (remy blank) - fixes #163
dfraser
parents:
549
diff
changeset
|
387 'Content-Length': str(len(body)), |
392 | 388 'Content-Type': 'application/x-bitten+xml' |
389 }) | |
277 | 390 |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
391 if resp.code == 201: |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
392 self._initiate_build(resp.info().get('location')) |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
393 return True |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
394 elif resp.code == 204: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
395 log.info('No pending builds') |
480
5181353ccb41
Slaves should continue to run builds without sleeping as long as new builds are available. Applied patch from Thomas Mueller. Closes #181
wbell
parents:
474
diff
changeset
|
396 return False |
392 | 397 else: |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
398 log.error('Unexpected response (%d %s)', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
399 raise ExitSlave(EX_PROTOCOL) |
56 | 400 |
392 | 401 def _initiate_build(self, build_url): |
418 | 402 log.info('Build pending at %s', build_url) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
403 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
404 resp = self.request('GET', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
405 if resp.code == 200: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
406 self._execute_build(build_url, resp) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
407 else: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
408 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
409 self._cancel_build(build_url, exit_code=EX_PROTOCOL) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
410 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
411 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
412 self._cancel_build(build_url) |
392 | 413 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
414 def _execute_build(self, build_url, fileobj): |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
415 build_id = build_url and int(build_url.split('/')[-1]) or 0 |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
416 xml = xmlio.parse(fileobj) |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
417 basedir = '' |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
418 try: |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
419 if not self.local: |
796
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
420 keepalive_thread = KeepAliveThread(self.opener, build_url, |
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
421 self.single_build, self.keepalive_interval) |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
422 keepalive_thread.start() |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
423 recipe = Recipe(xml, os.path.join(self.work_dir, self.build_dir), |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
424 self.config) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
425 basedir = recipe.ctxt.basedir |
490 | 426 log.debug('Running build in directory %s' % basedir) |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
427 if not os.path.exists(basedir): |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
428 os.mkdir(basedir) |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
429 |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
430 for step in recipe: |
754
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
431 try: |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
432 log.info('Executing build step %r, onerror = %s', step.id, step.onerror) |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
433 if not self._execute_step(build_url, recipe, step): |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
434 log.warning('Stopping build due to failure') |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
435 break |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
436 except Exception, e: |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
437 log.error('Exception raised processing step %s. Reraising %s', step.id, e) |
545be0c8f405
Adding the ability to modify the default ''onerror'' property in the ''<build>'' element. If not specified, the behavior is unchanged; by default any step failure will result in the build failing and stopping.
wbell
parents:
719
diff
changeset
|
438 raise |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
439 else: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
440 log.info('Build completed') |
461
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
441 if self.dry_run: |
f487963dd886
When the slave is run in `--dry-run` mode, it now also sends a build cancellation message to the master in case of failed builds. Closes #187.
cmlenz
parents:
457
diff
changeset
|
442 self._cancel_build(build_url) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
443 finally: |
796
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
444 if not self.local: |
031ce03d2baf
Don't initialize the keep-alive thread when running a local build (it will fail trying to get the opener).
osimons
parents:
785
diff
changeset
|
445 keepalive_thread.stop() |
580
c66190747a9c
0.6dev: In the event that loading Recipe fails, `basedir` would not yet have been defined, and removing it would fail.
osimons
parents:
576
diff
changeset
|
446 if not self.keep_files and os.path.isdir(basedir): |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
447 log.debug('Removing build directory %s' % basedir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
448 _rmtree(basedir) |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
449 if self.single_build: |
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
450 log.info('Exiting after single build completed.') |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
451 raise ExitSlave(EX_OK) |
392 | 452 |
453 def _execute_step(self, build_url, recipe, step): | |
454 failed = False | |
758
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
455 started = int(time.time()) |
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
456 xml = xmlio.Element('result', step=step.id) |
392 | 457 try: |
458 for type, category, generator, output in \ | |
459 step.execute(recipe.ctxt): | |
460 if type == Recipe.ERROR: | |
82
01200c88ddb0
Include timing information in the build messages transmitted from slave to master for better accuracy.
cmlenz
parents:
80
diff
changeset
|
461 failed = True |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
462 if type == Recipe.REPORT and self.dump_reports: |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
463 print output |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
464 if type == Recipe.ATTACH: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
465 # Attachments are added out-of-band due to major |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
466 # performance issues with inlined base64 xml content |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
467 self._attach_file(build_url, recipe, output) |
392 | 468 xml.append(xmlio.Element(type, category=category, |
469 generator=generator)[ | |
470 output | |
471 ]) | |
414
aa34d82b2c9a
The build slave can now run locally against a recipe file, which is useful for testing recipes. Simply pass the path to the recipe instead of the URL of the build master to the script.
cmlenz
parents:
411
diff
changeset
|
472 except KeyboardInterrupt: |
417
ab2557ff0d97
Change the default log verbosity of the build slave, and remove the `--debug` option.
cmlenz
parents:
414
diff
changeset
|
473 log.warning('Build interrupted') |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
474 self._cancel_build(build_url) |
392 | 475 except BuildError, e: |
628
05686657e989
0.6dev: Implement a `subprocess` implementation of `execute()` that is now the default for all slaves where this is available - essentially all slaves with Python 2.4 and higher, or where module is installed separately. This fixes:
osimons
parents:
610
diff
changeset
|
476 log.error('Build step %r failed', step.id) |
392 | 477 failed = True |
478 except Exception, e: | |
479 log.error('Internal error in build step %r', step.id, exc_info=True) | |
480 failed = True | |
758
d11ef8024d7c
Make all times be generated by the server. This eliminates a number of inconsistencies you'll see when the clocks on the slaves are skewed from the master.
wbell
parents:
754
diff
changeset
|
481 xml.attr['duration'] = (time.time() - started) |
392 | 482 if failed: |
483 xml.attr['status'] = 'failure' | |
484 else: | |
485 xml.attr['status'] = 'success' | |
486 log.info('Build step %s completed successfully', step.id) | |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
487 |
424
bc81d7b069ff
Rudimentary implementation of ``dry-run`` option in build slave. Builds are still put in in progress mode, but no results are transmitted back, and the build is automatically invalidated after the slave is done.
cmlenz
parents:
420
diff
changeset
|
488 if not self.local and not self.dry_run: |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
489 try: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
490 resp = self.request('POST', build_url + '/steps/', str(xml), { |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
491 'Content-Type': 'application/x-bitten+xml' |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
492 }) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
493 if resp.code != 201: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
494 log.error('Unexpected response (%d): %s', resp.code, |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
495 resp.msg) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
496 except KeyboardInterrupt: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
497 log.warning('Build interrupted') |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
498 self._cancel_build(build_url) |
392 | 499 return not failed or step.onerror != 'fail' |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
500 |
495
7e9a60417cb3
`os.EX_OK` etc are only defined on UNIX and Mac. Replace with fallback constants. Fixes #298 so Windows slaves now run successfully
dfraser
parents:
494
diff
changeset
|
501 def _cancel_build(self, build_url, exit_code=EX_OK): |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
502 log.info('Cancelling build at %s', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
503 if not self.local: |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
504 resp = self.request('DELETE', build_url) |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
505 if resp.code not in (200, 204): |
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
506 log.error('Unexpected response (%d): %s', resp.code, resp.msg) |
490 | 507 raise ExitSlave(exit_code) |
420
23de253435b8
Slaves now attempt to explicitly cancel builds when they are interrupted.
cmlenz
parents:
418
diff
changeset
|
508 |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
509 def _attach_file(self, build_url, recipe, attachment): |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
510 form_token = recipe._root.attr.get('form_token', '') |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
511 if self.local or self.dry_run or not form_token: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
512 log.info('Attachment %s not sent due to current slave options', |
905
0ae349142bf2
Oops. Typo in `local` and `dry_run` slave attachment info logging. Fixes #670. Thanks.
osimons
parents:
897
diff
changeset
|
513 attachment.attr['filename']) |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
514 return |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
515 resource_type = attachment.attr['resource'] |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
516 url = str(build_url + '/attach/' + resource_type) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
517 path = recipe.ctxt.resolve(attachment.attr['filename']) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
518 filename = os.path.basename(path) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
519 log.debug('Attaching file %s to %s...', attachment.attr['filename'], |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
520 resource_type) |
889
9ac91d38198c
Make sure slave reads attachments as binary files. Fixes #657.
osimons
parents:
870
diff
changeset
|
521 f = open(path, 'rb') |
835
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
522 try: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
523 data, content_type = encode_multipart_formdata({ |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
524 'file': (filename, f.read()), |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
525 'description': attachment.attr['description'], |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
526 '__FORM_TOKEN': form_token}) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
527 finally: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
528 f.close() |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
529 resp = self.request('POST', url , data, { |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
530 'Content-Type': content_type}) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
531 if not resp.code == 201: |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
532 msg = 'Error attaching %s to %s' |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
533 log.error(msg, filename, resource_type) |
59acaa8b52c0
Slave attachment support via `<attach />` is totally redone to use multi-part form post instead of inlining it in the XML (ie. like a web file upload form). For larger binaries the previous inlining would effectively be an internal denial-of-service attack...
osimons
parents:
832
diff
changeset
|
534 raise BuildError(msg, filename, resource_type) |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
535 |
392 | 536 class ExitSlave(Exception): |
537 """Exception used internally by the slave to signal that the slave process | |
411
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
538 should be stopped. |
a169d2e96463
Use reStructuredText as the API documentation syntax.
cmlenz
parents:
408
diff
changeset
|
539 """ |
490 | 540 def __init__(self, exit_code): |
541 self.exit_code = exit_code | |
542 Exception.__init__(self) | |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
543 |
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
544 |
31 | 545 def main(): |
313 | 546 """Main entry point for running the build slave.""" |
56 | 547 from bitten import __version__ as VERSION |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
548 from optparse import OptionParser |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
549 |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
550 parser = OptionParser(usage='usage: %prog [options] url1 [url2] ...', |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
551 version='%%prog %s' % VERSION) |
185
2c24d9a950ed
Add a `--dry-run` option to the build slave. This will result in the slave being registered and executing builds, but without submitting the progress and results of the build back to the server. Useful for getting the configuration of new slaves right without polluting the database with invalid builds.
cmlenz
parents:
163
diff
changeset
|
552 parser.add_option('--name', action='store', dest='name', |
63
2332aedba328
* Allow specifying a different name for a build slave (default is the host name).
cmlenz
parents:
61
diff
changeset
|
553 help='name of this slave (defaults to host name)') |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
554 parser.add_option('-f', '--config', action='store', dest='config', |
127
a9443f673344
Add option for specifying a [wiki:SlaveConfiguration configuration file] for the build slave. Closes #29.
cmlenz
parents:
119
diff
changeset
|
555 metavar='FILE', help='path to configuration file') |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
556 parser.add_option('-u', '--user', dest='username', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
557 help='the username to use for authentication') |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
558 parser.add_option('-p', '--password', dest='password', |
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
559 help='the password to use when authenticating') |
587
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
560 def _ask_password(option, opt_str, value, parser): |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
561 from getpass import getpass |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
562 parser.values.password = getpass('Passsword: ') |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
563 parser.add_option('-P', '--ask-password', action='callback', |
0ac21e3343a4
0.6dev: Adding `-P` option to bitten-slave to prompt for password as alternative to command-line option or configuration file. Closes #402.
osimons
parents:
580
diff
changeset
|
564 callback=_ask_password, help='Prompt for password') |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
565 parser.add_option('--form-auth', action='store_true', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
566 dest='form_auth', |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
567 help='login using AccountManager HTML form instead of ' |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
568 'HTTP authentication for all urls') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
569 |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
570 group = parser.add_option_group('building') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
571 group.add_option('-d', '--work-dir', action='store', dest='work_dir', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
572 metavar='DIR', help='working directory for builds') |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
573 group.add_option('--build-dir', action='store', dest='build_dir', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
574 default = 'build_${config}_${build}', |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
575 help='name pattern for the build dir to use inside the ' |
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
576 'working dir ["%default"]') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
577 group.add_option('-k', '--keep-files', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
578 dest='keep_files', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
579 help='don\'t delete files after builds') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
580 group.add_option('-s', '--single', action='store_true', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
581 dest='single_build', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
582 help='exit after completing a single build') |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
583 group.add_option('', '--no-loop', action='store_true', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
584 dest='no_loop', |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
585 help='exit after completing a single check and running ' |
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
586 'the required builds') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
587 group.add_option('-n', '--dry-run', action='store_true', dest='dry_run', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
588 help='don\'t report results back to master') |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
589 group.add_option('-i', '--interval', dest='interval', metavar='SECONDS', |
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
590 type='int', help='time to wait between requesting builds') |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
591 group.add_option('-b', '--keepalive_interval', dest='keepalive_interval', metavar='SECONDS', type='int', help='time to wait between keepalive heartbeats') |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
592 group = parser.add_option_group('logging') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
593 group.add_option('-l', '--log', dest='logfile', metavar='FILENAME', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
594 help='write log messages to FILENAME') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
595 group.add_option('-v', '--verbose', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
596 const=logging.DEBUG, help='print as much as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
597 group.add_option('-q', '--quiet', action='store_const', dest='loglevel', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
598 const=logging.WARN, help='print as little as possible') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
599 group.add_option('--dump-reports', action='store_true', dest='dump_reports', |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
600 help='whether report data should be printed') |
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
601 |
242
372d1de2e3ec
* Fixes to the `<c:configure>` command added in [247]: Set current directory when invoking the script, and correctly pass `CFLAGS` and `CXXFLAGS`.
cmlenz
parents:
241
diff
changeset
|
602 parser.set_defaults(dry_run=False, keep_files=False, |
525
bb448ad9bf35
Added option for not looping after check. Fixes #371
dfraser
parents:
515
diff
changeset
|
603 loglevel=logging.INFO, single_build=False, no_loop=False, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
604 dump_reports=False, interval=300, keepalive_interval=60, |
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
605 form_auth=False) |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
606 options, args = parser.parse_args() |
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
607 |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
608 if len(args) < 1: |
19
9db5f8eddb0d
Proper {{{optparse}}}-based command-line interface for master and slave.
cmlenz
parents:
18
diff
changeset
|
609 parser.error('incorrect number of arguments') |
494
9c9bf7a69bf0
Apply patch by Emmanuel Blot to allow a single slave to connect to multiple masters. Closes #271.
cmlenz
parents:
490
diff
changeset
|
610 urls = args |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
611 |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
612 logger = logging.getLogger('bitten') |
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
613 logger.setLevel(options.loglevel) |
93
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
614 handler = logging.StreamHandler() |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
615 handler.setLevel(options.loglevel) |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
616 formatter = logging.Formatter('[%(levelname)-8s] %(message)s') |
b289e572bc7e
Improved logging; the build master can now optionally log to a file. Closes #13.
cmlenz
parents:
90
diff
changeset
|
617 handler.setFormatter(formatter) |
157
2efdc69e63c3
Some style/documentation improvements to make Pylint happier.
cmlenz
parents:
149
diff
changeset
|
618 logger.addHandler(handler) |
339
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
619 if options.logfile: |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
620 handler = logging.FileHandler(options.logfile) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
621 handler.setLevel(options.loglevel) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
622 formatter = logging.Formatter('%(asctime)s [%(name)s] %(levelname)s: ' |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
623 '%(message)s') |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
624 handler.setFormatter(formatter) |
6469c692c603
Add a logfile option to the slave. Thanks to Walter Bell for the patch.
cmlenz
parents:
318
diff
changeset
|
625 logger.addHandler(handler) |
13
21aa17f97522
Initial code for build master and slave... these don't do a lot yet.
cmlenz
parents:
diff
changeset
|
626 |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
627 log.info("Slave launched at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
628 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
629 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
630 slave = None |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
631 try: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
632 slave = BuildSlave(urls, name=options.name, config=options.config, |
392 | 633 dry_run=options.dry_run, work_dir=options.work_dir, |
466
79be3c00ae69
Applied patch to #188 for stable/configurable names of build directories. Thanks to Allen Bierbaum for the patch.
cmlenz
parents:
465
diff
changeset
|
634 build_dir=options.build_dir, |
392 | 635 keep_files=options.keep_files, |
402
08801667f00f
Switch to urllib2 in slave so that we can support basic and digest authentication.
cmlenz
parents:
401
diff
changeset
|
636 single_build=options.single_build, |
543
ce16a5dcf8b3
Pass `--no-loop` option to the `BuildSlave` (fixes #375, from the patch there - thanks!)
dfraser
parents:
542
diff
changeset
|
637 no_loop=options.no_loop, |
449
01503ed994e7
Make the interval the slave sleeps between requesting builds configurable from the commandline.
cmlenz
parents:
442
diff
changeset
|
638 poll_interval=options.interval, |
785
8e76b8f6310a
Adding keepalives to the bitten client/server protocol.
wbell
parents:
758
diff
changeset
|
639 keepalive_interval=options.keepalive_interval, |
442
a8787de4fbc3
Improve the still experimental support for using `coverage.py`.
cmlenz
parents:
436
diff
changeset
|
640 username=options.username, password=options.password, |
648
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
641 dump_reports=options.dump_reports, |
a04e46a0bce3
0.6dev: Adding real authentication by making a request to `/login` and storing and reusing cookies provided by Trac (#208). Additionally, support for account-manager form authentication is added to the slave (#219).
osimons
parents:
647
diff
changeset
|
642 form_auth=options.form_auth) |
392 | 643 try: |
490 | 644 exit_code = slave.run() |
392 | 645 except KeyboardInterrupt: |
646 slave.quit() | |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
647 except ConfigFileNotFound, e: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
648 log.error(e) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
649 exit_code = EX_IOERR |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
650 except MissingSectionHeaderError: |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
651 log.error("Error parsing configuration file %r. Wrong format?" \ |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
652 % options.config) |
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
653 exit_code = EX_IOERR |
490 | 654 except ExitSlave, e: |
655 exit_code = e.exit_code | |
31 | 656 |
675
251be647314c
0.6dev: Adding some error-handling to bitten-slave for config files. It now reports errors and exists gracefully if a) config file isn't found, or b) problem parsing content (wrong format).
osimons
parents:
654
diff
changeset
|
657 if slave and not (options.work_dir or options.keep_files): |
576
4b75e3f6f27c
0.6dev: Fixing #229 again - keeping `slave.work_dir` if either `--keep_files` or `--work_dir=` are part of options.
osimons
parents:
570
diff
changeset
|
658 log.debug('Removing working directory %s' % slave.work_dir) |
474
8c3dfe0efe78
Committing rmtree patch for windows submitted by Doug Patterson. Thanks! Closes #211, #183, #236.
wbell
parents:
466
diff
changeset
|
659 _rmtree(slave.work_dir) |
610
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
660 |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
661 log.info("Slave exited at %s" % \ |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
662 datetime.now().strftime('%Y-%m-%d %H:%M:%S')) |
610a97f7e1ba
0.6dev: Logging start + stop time for the slave. Closes #268.
osimons
parents:
587
diff
changeset
|
663 |
490 | 664 return exit_code |
208
f39136272683
* Slave now removes build working directories when done.
cmlenz
parents:
203
diff
changeset
|
665 |
31 | 666 if __name__ == '__main__': |
490 | 667 sys.exit(main()) |